From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by gabe.freedesktop.org (Postfix) with ESMTPS id D872810FC7B for ; Thu, 14 Apr 2022 12:25:11 +0000 (UTC) From: Mauro Carvalho Chehab To: igt-dev@lists.freedesktop.org, Petri Latvala Date: Thu, 14 Apr 2022 14:24:57 +0200 Message-Id: <3f620c9eb05bfacdee38f1975fb9e03ee1ff62f7.1649939026.git.mchehab@kernel.org> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [igt-dev] [PATCH v3 07/12] docs/code_coverage.md: document the code coverage filter script List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ch Sai Gowtham , Andrzej Hajda Errors-To: igt-dev-bounces@lists.freedesktop.org Sender: "igt-dev" List-ID: From: Mauro Carvalho Chehab Add documentation bits for the code_coverage_parse_info. While here, also drop the extensions from the scripts that were renamed. Reviewed-by: Ch Sai Gowtham Reviewed-by: Andrzej Hajda Signed-off-by: Mauro Carvalho Chehab --- To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover. See [PATCH v3 00/12] at: https://lore.kernel.org/all/cover.1649939026.git.mchehab@kernel.org/ docs/code_coverage.md | 238 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 224 insertions(+), 14 deletions(-) diff --git a/docs/code_coverage.md b/docs/code_coverage.md index 34d56d283a75..915e800ed68b 100644 --- a/docs/code_coverage.md +++ b/docs/code_coverage.md @@ -132,15 +132,15 @@ Kernel that was built at the same machine, at the directory `~/linux`, and wants to capture one file per test, it would use: ``` -./scripts/run-tests.sh -T my.testlist -k ~/linux -c scripts/code_cov_capture.sh -P +./scripts/run-tests.sh -T my.testlist -k ~/linux -c code_cov_capture -P ``` ### Code Coverage Collect script While any script could in thesis be used, currently, there are two ones -under `scripts/`: +under the IGT's `scripts/` source directory: -- `scripts/code_cov_capture.sh`: +- `code_cov_capture`: Assumes that the Kernel was built at the same machine, and uses the lcov tool to generate GCC-independent code coverage data, @@ -150,7 +150,7 @@ under `scripts/`: Such script requires `lcov` tool to be installed at the test machine. -- `scripts/code_cov_gather_on_test.py`: +- `code_cov_gather_on_test`: Generates a gzipped tarbal with the code coverage counters in binary format. Such kind of output should then be parsed at @@ -162,12 +162,12 @@ For each script, the igt_runner passes just one parameter: the results directory + the test name. For instance, if it is needed to run a test called -`debugfs_test (read_all_entries)` using `scripts/code_cov_capture.sh` +`debugfs_test (read_all_entries)` using `code_cov_capture` parameter, e. g.: ``` $ echo "igt@debugfs_test@read_all_entries" > my.testlist -$ ./scripts/run-tests.sh -T my.testlist -k ~/linux -c scripts/code_cov_capture.sh -P +$ ./scripts/run-tests.sh -T my.testlist -k ~/linux -c code_cov_capture -P Found test list: "/basedir/igt/build/tests/test-list.txt" [31410.499969] [1/1] debugfs_test (read_all_entries) [31411.060446] Storing code coverage results... @@ -178,7 +178,7 @@ Done. The script will be called as: ``` -scripts/code_cov_capture.sh /basedir/igt/results/code_cov/debugfs_test_read_all_entries +code_cov_capture results/code_cov/debugfs_test_read_all_entries ``` Please notice that any character that it is not a number nor a letter at the @@ -190,6 +190,73 @@ as titles at the lcov files. If any extra global parameters are needed by the script, those can be sent via shell's environment var. +## Parsing data from code coverage *.info files + +The `*.info` files generated by `lcov` are plain text files that list the +tests that were executed in runtime. + +The `code_cov_parse_info` script has some logic on it that allows +printing the called functions stored inside the `*.info` file. It can also +optionally apply the following filters. Its main options are: + +- `--stat` or `--statistics` + + Prints code coverage statistics. + + It displays function, line, branch and file coverage percentage. + + The statistics report is affected by the applied filters. + +- `--print-coverage`, `--print` or `-p` + + + Prints the functions that were executed in runtime and how many times + they were reached. + + The function coverage report is affected by the applied filters. + +- `--print-unused` or `-u` + + Prints the functions that were never reached. + + The function coverage report is affected by the applied filters. + +- `--show-lines` or `--show_lines` + + When printing per-function code coverage data, always output the source + file and the line number where the function is defined. + +- `--output` *output file* or `-o` *output file* + + Produces an output file merging all input files. + + The generated output file is affected by the applied filters. + +- `--show-files` or `--show_files` + + Shows the list of files that were useed to produce the code coverage + results. + +- It also has a set of parameters that filters the code coverage results: + `--only-drm`, `--only-i915`, `--func-filters`, `--source-filters`, + `--ignore-unused`. + When used, all coverage displayed reports, and the stored output file + will be affected by such filters. + +More details can be seen by calling: + +``` +code_cov_parse_info --help + +``` + +or: + +``` +code_cov_parse_info --man + +``` + ### The `*.info` file format The `*.info` files contain several fields on it, grouped into records. @@ -224,7 +291,7 @@ So, the above example means that, inside ## Generating code coverage documentation The `lcov` package contains the needed tools to parse and generate code -coverage documentation. It is used by `scripts/code_cov_capture.sh` script +coverage documentation. It is used by `code_cov_capture` script to convery from compiler-dependent `*.gcno` counters into a compiler-independent format (`*.info`). @@ -243,11 +310,16 @@ source files. Some optional arguments can be used at the command line, or can be stored at `/etc/lcovrc` or `~/.lcovrc` files. As generating the documentation depends wheather the results were generated -as with a single or multiple `*.info` files by `scripts/code_cov_capture.sh` +as with a single or multiple `*.info` files by `code_cov_capture` or stored in raw formats inside `*.tar.gz` file(s) by -`scripts/code_cov_gather_on_test.py`, there's a script that does all the +`code_cov_gather_on_test`, there's a script that does all the required steps to build the code coverage html reports: -`scripts/code_cov_gen_report.sh`. +`code_cov_gen_report`. Besides its own command line arguments, it +also accepts arguments to be passed to `code_cov_parse_info`. + +If a `code_cov_parse_info` command line parameter is passed, it will +also call the script, in order to use a filtered `*.info` file to be +used when generating the HTML reports. It requires the following arguments: @@ -276,18 +348,156 @@ It requires the following arguments: The files specified by `--read` are gzipped tarballs containing all `*.gcno` files and all `*.gcda` softlinks from the `/sys/kernel/debug/gcov/` - directory at the test machine, created by - `scripts/code_cov_gather_on_test.py` script. + directory at the test machine, created by `code_cov_gather_on_test` script. - `--force-override` Allow using a non-empty directory for `--output-dir`. +- It also accepts `--print`, `--only-drm`, `--only-i915` and `--ignore-unused` + options from `code_cov_parse_info`. + `--info` and `--tar` are mutually exclusive and at least one of them should be specified. +## Code coverage capture script example + +### Capture, parse and generate code coverage html data + +The script below provides a simple yet powerful script using code +coverage capture on a test machine that also contains the Linux +Kernel source and objects. It assumes that LGT was installed. + +``` +#/bin/bash -e + +TESTLIST="my_tests.testlist" +OUT_DIR="${HOME}/results" + +mkdir -p $OUT_DIR/html + +echo "igt@debugfs_test@read_all_entries" > $TESTLIST +echo "igt@core_auth@basic-auth" >> $TESTLIST +echo "igt@gem_exec_basic@basic" >> $TESTLIST + +sudo IGT_KERNEL_TREE="${HOME}/linux" igt_runner -s -o --coverage-per-test \ + --collect-script code_cov_capture --test-list $TESTLIST \ + /usr/local/libexec/igt-gpu-tools $OUT_DIR/ | sed s,$HOME/,, + +sudo chown -R $(id -u):$(id -g) $OUT_DIR/ + +for i in $OUT_DIR/code_cov/*.info; do + echo -e "\n$(basename $i):" + code_cov_parse_info --only-drm --ignore-unused --stat $i +done +echo -e "\nTOTAL:" +code_cov_parse_info --only-drm --stat --output $OUT_DIR/results.info \ + $OUT_DIR/code_cov/*.info + +cd $OUT_DIR/html +genhtml -q -s --legend --branch-coverage $OUT_DIR/results.info +``` + +Running such script produces the following output: + +``` +[3622.993304] [1/3] debugfs_test (read_all_entries) +[3631.95] Code coverage wrote to results/code_cov/debugfs_test_read_all_entries.info +[3626.217016] Storing code coverage results... +[3631.957998] [2/3] core_auth (basic-auth) +[3638.03] Code coverage wrote to results/code_cov/core_auth_basic_auth.info +[3632.116024] Storing code coverage results... +[3638.070869] [3/3] gem_exec_basic (basic) +[3644.24] Code coverage wrote to results/code_cov/gem_exec_basic_basic.info +[3638.366790] Storing code coverage results... +Done. + +core_auth_basic_auth.info: + lines......: 11.7% (8217 of 70257 lines) + functions..: 7.1% (776 of 10971 functions) + branches...: 7.0% (3596 of 51041 branches) +Ignored......: non-drm headers and source files where none of its code ran. +Source files.: 23.27% (165 of 709 total), 29.57% (165 of 558 filtered) + +debugfs_test_read_all_entries.info: + lines......: 19.3% (20266 of 104802 lines) + functions..: 17.5% (1922 of 10971 functions) + branches...: 12.7% (9462 of 74555 branches) +Ignored......: non-drm headers and source files where none of its code ran. +Source files.: 34.70% (246 of 709 total), 44.09% (246 of 558 filtered) + +gem_exec_basic_basic.info: + lines......: 17.1% (14964 of 87503 lines) + functions..: 13.0% (1422 of 10971 functions) + branches...: 10.1% (6446 of 63758 branches) +Ignored......: non-drm headers and source files where none of its code ran. +Source files.: 30.89% (219 of 709 total), 39.25% (219 of 558 filtered) + +TOTAL: + lines......: 15.5% (25821 of 166849 lines) + functions..: 22.1% (2429 of 10971 functions) + branches...: 10.5% (11869 of 112665 branches) +Ignored......: non-drm headers. +Source files.: 78.70% (558 of 709 total) +``` + +### Reporting detailed function coverage stored on *.info files + +The `code_cov_parse_info` script can be used alone in order to provide +a text file output containing code coverage data obtained from a *.info +file. For example, listing code coverage usage for all functions whose name +contains "edid_" can be done with: + +``` +$ echo edid_ >filter.txt +$ code_cov_parse_info --func-filters filter.txt results/results.info -p -u --stat +TEST: Code_coverage_tests +__drm_get_edid_firmware_path(): unused +__drm_set_edid_firmware_path(): unused +displayid_iter_edid_begin(): executed 10 times +drm_add_edid_modes(): executed 2 times +drm_add_override_edid_modes(): unused +drm_connector_attach_edid_property(): unused +drm_connector_update_edid_property(): executed 8 times +drm_dp_send_real_edid_checksum(): unused +drm_edid_are_equal(): executed 4 times +drm_edid_block_valid(): executed 8 times +drm_edid_duplicate(): unused +drm_edid_get_monitor_name(): unused +drm_edid_header_is_valid(): executed 4 times +drm_edid_is_valid(): executed 2 times +drm_edid_to_eld(): executed 2 times +drm_edid_to_sad(): unused +drm_edid_to_speaker_allocation(): unused +drm_find_edid_extension(): executed 22 times +drm_get_edid_switcheroo(): unused +drm_load_edid_firmware(): executed 2 times +edid_firmware_get(): unused +edid_firmware_set(): unused +edid_fixup_preferred(): unused +edid_get_quirks(): executed 6 times +edid_load(): unused +edid_open(): executed 4 times +edid_show() from linux/drivers/gpu/drm/drm_debugfs.c: executed 4 times +edid_show() from linux/drivers/gpu/drm/drm_sysfs.c: unused +edid_vendor(): executed 348 times +edid_write(): unused +intel_panel_edid_downclock_mode(): unused +intel_panel_edid_fixed_mode(): unused +is_edid_digital_input_dp(): unused + lines......: 5.5% (5 of 91 lines) + functions..: 42.4% (14 of 33 functions) + branches...: 1.9% (1 of 52 branches) +Ignored......: unmatched functions m/(?^:edid_)/ and source files where none of its code ran. +Source files.: 0.90% (5 of 558 total), 55.56% (5 of 9 filtered) +``` + +When the function is unique, it will just display the function name and how +many times the IGT test(s) executed it. When the same function name exists +on multiple files (like the `edid_show()` on the above example), it will +display multiple lines, one for each different function/file combination. + ## References More information is available at Kernel gcov documentation: [Using gcov with the Linux kernel](https://www.kernel.org/doc/html/latest/dev-tools/gcov.html). - -- 2.35.1