* [PATCH 0/4] Static analyser finding deviation
@ 2022-11-28 14:10 Luca Fancellu
2022-11-28 14:10 ` [PATCH 1/4] xen/scripts: add xen-analysis.py for coverity and eclair analysis Luca Fancellu
` (4 more replies)
0 siblings, 5 replies; 31+ messages in thread
From: Luca Fancellu @ 2022-11-28 14:10 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, wei.chen, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu
This serie introduces a way to suppress a static analyser finding providing a
proper justification for it.
The process is explained in the docs/misra/documenting-violations.rst document
that this serie will provide.
The tools currently supported are eclair, coverity and cppcheck, but the design
is open to support many other static analysis tool.
The changes are split between the first two patches to reduce the review effort,
the first patch is introducing the deviation process for the eclair and coverity
tools, this is because their analysis system is similar.
The second patch is introducing the same deviation process for cppcheck,
modifying the current way it is called from the makefile and improving its
analysis.
The third patch is a fix for a tool used for cppcheck and the fourth patch
is an example of how a deviation can be applied for some MISRA findings.
---
This serie was pushed as RFC and collected many feedbacks, thank you for the
review.
In this serie to analyse the codebase, a script is used instead of integrating
the process into the makefile.
---
Luca Fancellu (4):
xen/scripts: add xen-analysis.py for coverity and eclair analysis
xen/scripts: add cppcheck tool to the xen-analysis.py script
tools/misra: fix skipped rule numbers
xen: Justify linker script defined symbols in include/xen/kernel.h
.gitignore | 9 +-
docs/misra/cppcheck.txt | 27 +-
docs/misra/documenting-violations.rst | 192 +++++++++++++
docs/misra/false-positive-coverity.json | 12 +
docs/misra/false-positive-cppcheck.json | 12 +
docs/misra/false-positive-eclair.json | 12 +
docs/misra/safe.json | 20 ++
docs/misra/xen-static-analysis.rst | 90 ++++++
xen/Makefile | 116 +-------
xen/include/hypercall-defs.c | 9 +
xen/include/xen/kernel.h | 4 +
xen/scripts/xen-analysis.py | 45 +++
xen/scripts/xen_analysis/__init__.py | 0
xen/scripts/xen_analysis/cppcheck_analysis.py | 272 ++++++++++++++++++
.../xen_analysis/cppcheck_report_utils.py | 130 +++++++++
xen/scripts/xen_analysis/generic_analysis.py | 88 ++++++
xen/scripts/xen_analysis/settings.py | 152 ++++++++++
xen/scripts/xen_analysis/tag_database.py | 109 +++++++
xen/scripts/xen_analysis/utils.py | 56 ++++
xen/tools/convert_misra_doc.py | 32 ++-
xen/tools/cppcheck-cc.sh | 223 ++++++++++++++
xen/tools/cppcheck-plat/arm32-wchar_t4.xml | 17 ++
xen/tools/cppcheck-plat/arm64-wchar_t2.xml | 17 ++
xen/tools/cppcheck-plat/arm64-wchar_t4.xml | 17 ++
xen/tools/cppcheck-plat/x86_64-wchar_t2.xml | 17 ++
xen/tools/cppcheck-plat/x86_64-wchar_t4.xml | 17 ++
xen/tools/merge_cppcheck_reports.py | 86 ------
27 files changed, 1557 insertions(+), 224 deletions(-)
create mode 100644 docs/misra/documenting-violations.rst
create mode 100644 docs/misra/false-positive-coverity.json
create mode 100644 docs/misra/false-positive-cppcheck.json
create mode 100644 docs/misra/false-positive-eclair.json
create mode 100644 docs/misra/safe.json
create mode 100644 docs/misra/xen-static-analysis.rst
create mode 100755 xen/scripts/xen-analysis.py
create mode 100644 xen/scripts/xen_analysis/__init__.py
create mode 100644 xen/scripts/xen_analysis/cppcheck_analysis.py
create mode 100644 xen/scripts/xen_analysis/cppcheck_report_utils.py
create mode 100644 xen/scripts/xen_analysis/generic_analysis.py
create mode 100644 xen/scripts/xen_analysis/settings.py
create mode 100644 xen/scripts/xen_analysis/tag_database.py
create mode 100644 xen/scripts/xen_analysis/utils.py
create mode 100755 xen/tools/cppcheck-cc.sh
create mode 100644 xen/tools/cppcheck-plat/arm32-wchar_t4.xml
create mode 100644 xen/tools/cppcheck-plat/arm64-wchar_t2.xml
create mode 100644 xen/tools/cppcheck-plat/arm64-wchar_t4.xml
create mode 100644 xen/tools/cppcheck-plat/x86_64-wchar_t2.xml
create mode 100644 xen/tools/cppcheck-plat/x86_64-wchar_t4.xml
delete mode 100755 xen/tools/merge_cppcheck_reports.py
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 1/4] xen/scripts: add xen-analysis.py for coverity and eclair analysis
2022-11-28 14:10 [PATCH 0/4] Static analyser finding deviation Luca Fancellu
@ 2022-11-28 14:10 ` Luca Fancellu
2022-11-29 1:39 ` Stefano Stabellini
2022-11-28 14:10 ` [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script Luca Fancellu
` (3 subsequent siblings)
4 siblings, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-11-28 14:10 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, wei.chen, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu
Add new script for coverity/eclair analysis tool that will enable
the procedure to suppress findings when these tool are used.
The procedure is documented in docs/misra/documenting-violations.rst
and the script is documented in docs/misra/xen-static-analysis.rst.
Add in docs/misra/ the files safe.json and
false-positive-{coverity,eclair}.json that are JSON files containing
the data structures for the justifications, they are used by the
analysis script to link the Xen tags to the proprietary tool comment.
Add docs/misra/documenting-violations.rst to explain how to add
justifications.
Add docs/misra/xen-static-analysis.rst to explain how to use the
script to analyse Xen.
Add analysis artifacts files to .gitignore.
Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
---
.gitignore | 1 +
docs/misra/documenting-violations.rst | 191 +++++++++++++++++++
docs/misra/false-positive-coverity.json | 12 ++
docs/misra/false-positive-eclair.json | 12 ++
docs/misra/safe.json | 11 ++
docs/misra/xen-static-analysis.rst | 54 ++++++
xen/scripts/xen-analysis.py | 31 +++
xen/scripts/xen_analysis/__init__.py | 0
xen/scripts/xen_analysis/generic_analysis.py | 93 +++++++++
xen/scripts/xen_analysis/settings.py | 97 ++++++++++
xen/scripts/xen_analysis/tag_database.py | 109 +++++++++++
xen/scripts/xen_analysis/utils.py | 37 ++++
12 files changed, 648 insertions(+)
create mode 100644 docs/misra/documenting-violations.rst
create mode 100644 docs/misra/false-positive-coverity.json
create mode 100644 docs/misra/false-positive-eclair.json
create mode 100644 docs/misra/safe.json
create mode 100644 docs/misra/xen-static-analysis.rst
create mode 100755 xen/scripts/xen-analysis.py
create mode 100644 xen/scripts/xen_analysis/__init__.py
create mode 100644 xen/scripts/xen_analysis/generic_analysis.py
create mode 100644 xen/scripts/xen_analysis/settings.py
create mode 100644 xen/scripts/xen_analysis/tag_database.py
create mode 100644 xen/scripts/xen_analysis/utils.py
diff --git a/.gitignore b/.gitignore
index ea3243af9dde..f5a66f6194dd 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,6 +10,7 @@
*.c.cppcheck
*.opic
*.a
+*.safparse
*.so
*.so.[0-9]*
*.bin
diff --git a/docs/misra/documenting-violations.rst b/docs/misra/documenting-violations.rst
new file mode 100644
index 000000000000..1d23447556d2
--- /dev/null
+++ b/docs/misra/documenting-violations.rst
@@ -0,0 +1,191 @@
+.. SPDX-License-Identifier: CC-BY-4.0
+
+Documenting violations
+======================
+
+Static analysers are used on the Xen codebase for both static analysis and MISRA
+compliance.
+There might be the need to suppress some findings instead of fixing them and
+many tools permit the usage of in-code comments that suppress findings so that
+they are not shown in the final report.
+
+Xen includes a tool capable of translating a specific comment used in its
+codebase to the right proprietary in-code comment understandable by the selected
+analyser that suppress its finding.
+
+In the Xen codebase, these tags will be used to document and suppress findings:
+
+ - SAF-X-safe: This tag means that the next line of code contains a finding, but
+ the non compliance to the checker is analysed and demonstrated to be safe.
+ - SAF-X-false-positive-<tool>: This tag means that the next line of code
+ contains a finding, but the finding is a bug of the tool.
+
+SAF stands for Static Analyser Finding, the X is a placeholder for a positive
+number that starts from zero, the number after SAF- shall be incremental and
+unique, base ten notation and without leading zeros.
+
+Entries in the database shall never be removed, even if they are not used
+anymore in the code (if a patch is removing or modifying the faulty line).
+This is to make sure that numbers are not reused which could lead to conflicts
+with old branches or misleading justifications.
+
+An entry can be reused in multiple places in the code to suppress a finding if
+and only if the justification holds for the same non-compliance to the coding
+standard.
+
+An orphan entry, that is an entry who was justifying a finding in the code, but
+later that code was removed and there is no other use of that entry in the code,
+can be reused as long as the justification for the finding holds. This is done
+to avoid the allocation of a new entry with exactly the same justification, that
+would lead to waste of space and maintenance issues of the database.
+
+The files where to store all the justifications are in xen/docs/misra/ and are
+named as safe.json and false-positive-<tool>.json, they have JSON format, each
+one has a different justification schema which shares some fields.
+
+Here is an example to add a new justification in safe.json::
+
+|{
+| "version": "1.0",
+| "content": [
+| {
+| "id": "SAF-0-safe",
+| "analyser": {
+| "coverity": "misra_c_2012_rule_20_7_violation",
+| "eclair": "MC3R1.R20.7"
+| },
+| "name": "R20.7 C macro parameters not used as expression",
+| "text": "The macro parameters used in this [...]"
+| },
+| {
+| "id": "SAF-1-safe",
+| "analyser": {},
+| "name": "Sentinel",
+| "text": "Next ID to be used"
+| }
+| ]
+|}
+
+To document a finding in safe.json, just add another block {[...]} before the
+sentinel block, using the id contained in the sentinel block and increment by
+one the number contained in the id of the sentinel block.
+
+Here is an explanation of the fields inside an object of the "content" array:
+ - id: it is a unique string that is used to refer to the finding, many finding
+ can be tagged with the same id, if the justification holds for any applied
+ case.
+ It tells the tool to substitute a Xen in-code comment having this structure:
+ /* SAF-0-safe [...] \*/
+ - analyser: it is an object containing pair of key-value strings, the key is
+ the analyser, so it can be coverity or eclair, the value is the proprietary
+ id corresponding on the finding, for example when coverity is used as
+ analyser, the tool will translate the Xen in-code coment in this way:
+ /* SAF-0-safe [...] \*/ -> /* coverity[misra_c_2012_rule_20_7_violation] \*/
+ if the object doesn't have a key-value, then the corresponding in-code
+ comment won't be translated.
+ - name: a simple name for the finding
+ - text: a proper justification to turn off the finding.
+
+
+Here is an example to add a new justification in false-positive-<tool>.json::
+
+|{
+| "version": "1.0",
+| "content": [
+| {
+| "id": "SAF-0-false-positive-<tool>",
+| "violation-id": "<proprietary-id>",
+| "tool-version": "<version>",
+| "name": "R20.7 [...]",
+| "text": "[...]"
+| },
+| {
+| "id": "SAF-1-false-positive-<tool>",
+| "violation-id": "",
+| "tool-version": "",
+| "name": "Sentinel",
+| "text": "Next ID to be used"
+| }
+| ]
+|}
+
+To document a finding in false-positive-<tool>.json, just add another block
+{[...]} before the sentinel block, using the id contained in the sentinel block
+and increment by one the number contained in the id of the sentinel block.
+
+Here is an explanation of the fields inside an object of the "content" array:
+ - id: it has the same meaning as in the "safe" justification schema.
+ It tells the tool to substitute a Xen in-code comment having this structure:
+ /* SAF-0-false-positive-<tool> [...] \*/
+ - violation-id: its value is a string containing the proprietary id
+ corresponding to the finding, for example when <tool> is coverity, the Xen
+ tool will translate the Xen in-code coment in this way:
+ /* SAF-0-false-positive-coverity [...] \*/ -> /* coverity[misra_c_2012_rule_20_7_violation] \*/
+ if the object doesn't have a value, then the corresponding in-code comment
+ won't be translated.
+ - tool-version: the version of the tool affected by the false positive, if it
+ is discovered in more than one version, this string can be a range
+ (eg. 2.7 - 3.0)
+ - name, text: they have the same meaning as in the "safe" justification schema.
+
+
+Justification example
+---------------------
+
+Here an example of the usage of the in-code comment tags to suppress a finding
+for the Rule 8.6:
+
+Eclair reports it in its web report, file xen/include/xen/kernel.h, line 68:
+
+| MC3R1.R8.6 for program 'xen/xen-syms', variable '_start' has no definition
+
+Also coverity reports it, here is an extract of the finding:
+
+| xen/include/xen/kernel.h:68:
+| 1. misra_c_2012_rule_8_6_violation: Function "_start" is declared but never
+ defined.
+
+The analysers are complaining because we have this in xen/include/xen/kernel.h
+at line 68::
+
+| extern char _start[], _end[], start[];
+
+Those are symbols exported by the linker, hence we will need to have a proper
+deviation for this finding.
+
+We will prepare our entry in the safe.json database::
+
+|{
+| "version": "1.0",
+| "content": [
+| {
+| [...]
+| },
+| {
+| "id": "SAF-1-safe",
+| "analyser": {
+| "eclair": "MC3R1.R8.6",
+| "coverity": "misra_c_2012_rule_8_6_violation"
+| },
+| "name": "Rule 8.6: linker script defined symbols",
+| "text": "It is safe to declare this symbol because it is defined in the linker script."
+| },
+| {
+| "id": "SAF-2-safe",
+| "analyser": {},
+| "name": "Sentinel",
+| "text": "Next ID to be used"
+| }
+| ]
+|}
+
+And we will use the proper tag above the violation line::
+
+| /* SAF-1-safe R8.6 linker defined symbols */
+| extern char _start[], _end[], start[];
+
+This entry will fix also the violation on _end and start, because they are on
+the same line and the same "violation ID".
+
+Also, the same tag can be used on other symbols from the linker that are
+declared in the codebase, because the justification holds for them too.
diff --git a/docs/misra/false-positive-coverity.json b/docs/misra/false-positive-coverity.json
new file mode 100644
index 000000000000..462448414f80
--- /dev/null
+++ b/docs/misra/false-positive-coverity.json
@@ -0,0 +1,12 @@
+{
+ "version": "1.0",
+ "content": [
+ {
+ "id": "SAF-0-false-positive-coverity",
+ "violation-id": "",
+ "tool-version": "",
+ "name": "Sentinel",
+ "text": "Next ID to be used"
+ }
+ ]
+}
diff --git a/docs/misra/false-positive-eclair.json b/docs/misra/false-positive-eclair.json
new file mode 100644
index 000000000000..1d6ea5d7f045
--- /dev/null
+++ b/docs/misra/false-positive-eclair.json
@@ -0,0 +1,12 @@
+{
+ "version": "1.0",
+ "content": [
+ {
+ "id": "SAF-0-false-positive-eclair",
+ "violation-id": "",
+ "tool-version": "",
+ "name": "Sentinel",
+ "text": "Next ID to be used"
+ }
+ ]
+}
diff --git a/docs/misra/safe.json b/docs/misra/safe.json
new file mode 100644
index 000000000000..e079d3038120
--- /dev/null
+++ b/docs/misra/safe.json
@@ -0,0 +1,11 @@
+{
+ "version": "1.0",
+ "content": [
+ {
+ "id": "SAF-0-safe",
+ "analyser": {},
+ "name": "Sentinel",
+ "text": "Next ID to be used"
+ }
+ ]
+}
diff --git a/docs/misra/xen-static-analysis.rst b/docs/misra/xen-static-analysis.rst
new file mode 100644
index 000000000000..5b886474d4a0
--- /dev/null
+++ b/docs/misra/xen-static-analysis.rst
@@ -0,0 +1,54 @@
+.. SPDX-License-Identifier: CC-BY-4.0
+
+Xen static analysis
+===================
+
+The Xen codebase integrates some scripts and tools that helps the developer to
+perform static analysis of the code, currently Xen supports three analysis tool
+that are eclair, coverity and cppcheck.
+The Xen tree has a script (xen-analysis.py) available to ease the analysis
+process and it integrates a way to suppress findings on these tools (only Eclair
+and Coverity are currently supported by the script), please check the
+documenting-violation.rst document to know more about it.
+
+Analyse Xen with Coverity or Eclair
+-----------------------------------
+
+The xen-analysis.py script has two arguments to select which tool is used for
+the analysis:
+
+ - xen-analysis.py --run-coverity -- [optional make arguments]
+ - xen-analysis.py --run-eclair -- [optional make arguments]
+
+For example when using Coverity to analyse a Xen build obtained by passing these
+arguments to the make system: XEN_TARGET_ARCH=arm64
+CROSS_COMPILE=aarch64-linux-gnu-, the optional make arguments passed to
+xen-analysis.py must be the same and the command below should be passed to
+Coverity in its build phase:
+
+ - xen-analysis.py --run-coverity -- XEN_TARGET_ARCH=arm64
+ CROSS_COMPILE=aarch64-linux-gnu-
+
+Which tells to the script to prepare the codebase for an analysis by Coverity
+and forwards the make arguments to the make build invocation.
+
+When invoking the script, the procedure below will be followed:
+
+ 1. Find which files among \*.c and \*.h has any in-code comment as
+ /* SAF-X-[...] \*/, the meaning of these comments is explained in
+ documenting-violation.rst.
+ Save the files obtained as <file>.safparse and generate <file> files where
+ the special in-code comments above are substituted with the proprietary
+ in-code comment used by the selected analysis tool. The safe.json and
+ false-positive-<tool>.json text file database are used to link each Xen tag
+ to the right proprietary in-code comment.
+ 2. Now Xen compilation starts using every <additional make parameters> supplied
+ at the script invocation. Coverity and Eclair are capable of intercepting
+ the compiler running from make to perform their analysis without
+ instrumenting the makefile.
+ 3. As final step every <file>.safparse file are reverted back as <file> and
+ every artifact related to the analysis will be cleaned.
+ This step is performed even in case any of the previous step fail, to skip
+ this step, call the script adding the --no-clean argument, but before
+ running again the script, call it with the --clean-only argument, that will
+ execute only this cleaning step.
diff --git a/xen/scripts/xen-analysis.py b/xen/scripts/xen-analysis.py
new file mode 100755
index 000000000000..b5d9ef1862c9
--- /dev/null
+++ b/xen/scripts/xen-analysis.py
@@ -0,0 +1,31 @@
+#!/usr/bin/env python3
+
+import sys
+from xen_analysis import settings, generic_analysis
+from xen_analysis.generic_analysis import *
+
+
+def main(argv):
+ ret_code = 0
+ settings.parse_commandline(argv)
+ try:
+ if settings.step_parse_tags:
+ generic_analysis.parse_xen_tags()
+ if settings.step_build_xen:
+ generic_analysis.build_xen()
+ except (ParseTagPhaseError, BuildPhaseError) as e:
+ print("ERROR: {}".format(e))
+ if hasattr(e, "errorcode"):
+ ret_code = e.errorcode
+ finally:
+ if settings.step_clean_analysis:
+ e = generic_analysis.clean_analysis_artifacts()
+ if e:
+ print("ERROR: {}".format(e))
+ ret_code = 1
+
+ sys.exit(ret_code)
+
+
+if __name__ == "__main__":
+ main(sys.argv[1:])
diff --git a/xen/scripts/xen_analysis/__init__.py b/xen/scripts/xen_analysis/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/xen/scripts/xen_analysis/generic_analysis.py b/xen/scripts/xen_analysis/generic_analysis.py
new file mode 100644
index 000000000000..0b470c4ecf7d
--- /dev/null
+++ b/xen/scripts/xen_analysis/generic_analysis.py
@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+
+import os, subprocess
+from . import settings, utils, tag_database
+
+class ParseTagPhaseError(Exception):
+ pass
+
+class BuildPhaseError(Exception):
+ pass
+
+class CleanPhaseError(Exception):
+ pass
+
+
+def parse_xen_tags():
+ # Load the database for the Xen tags
+ subs_list = tag_database.load_tag_database(
+ settings.analysis_tool,
+ [settings.repo_dir + "/docs/misra/safe.json"]
+ )
+ subs_list = tag_database.load_tag_database(
+ settings.analysis_tool,
+ [settings.repo_dir + "/docs/misra/false-positive-{}.json"
+ .format(settings.analysis_tool)],
+ subs_list,
+ "false-positive"
+ )
+
+ # Create outdir if it doesn't exists
+ os.makedirs(settings.outdir, exist_ok=True)
+
+ # The following lambda function will return a file if it contains lines with
+ # a comment containing "SAF-<number>-{safe|false-positive-<tool>}" on a
+ # single line.
+ grep_action = lambda x: utils.grep(x,
+ tag_database.get_xen_tag_comment_regex(
+ settings.analysis_tool)
+ )
+ # Look for a list of .h/.c files that matches the condition above
+ parse_file_list = utils.recursive_find_file(settings.xen_dir, r'.*\.[ch]$',
+ grep_action)
+
+ for entry in parse_file_list:
+ file = entry["file"]
+ bkp_file = file + ".safparse"
+ if os.path.isfile(bkp_file):
+ raise ParseTagPhaseError(
+ "Found {}, please check the integrity of {}"
+ .format(bkp_file,file)
+ )
+ os.rename(file, bkp_file)
+ time_bkp_file = os.stat(bkp_file)
+ # Create <file> from <file>.safparse but with the Xen tag parsed
+ tag_database.substitute_tags(settings.analysis_tool, bkp_file, entry,
+ subs_list)
+ # Set timestamp for file equal to bkp_file, so that if the file is
+ # modified during the process by the user, we can catch it
+ os.utime(file, (time_bkp_file.st_atime, time_bkp_file.st_mtime))
+
+
+def build_xen():
+ try:
+ subprocess.run(
+ "make -C {} {} build"
+ .format(settings.xen_dir, settings.make_forward_args),
+ shell=True, check=True
+ )
+ except (subprocess.CalledProcessError, subprocess.SubprocessError) as e:
+ excp = BuildPhaseError(
+ "Build error occured when running:\n{}".format(e.cmd)
+ )
+ excp.errorcode = e.returncode if hasattr(e, 'returncode') else 1
+ raise excp
+
+
+def clean_analysis_artifacts():
+ safparse_files = utils.recursive_find_file(settings.xen_dir,
+ r'.*.safparse$')
+ for original_file in safparse_files:
+ # This commands strips the .safparse extension, leaving <file>
+ parsed_file_path = os.path.splitext(original_file)[0]
+ mtime_original_file = os.stat(original_file).st_mtime
+ mtime_parsed_file = os.stat(parsed_file_path).st_mtime
+ if mtime_original_file != mtime_parsed_file:
+ return CleanPhaseError(
+ "The file {} was modified during the analysis "
+ "procedure, it is impossible now to restore from the "
+ "content of {}, please handle it manually"
+ .format(parsed_file_path, original_file)
+ )
+ # Replace <file>.safparse to <file>
+ os.replace(original_file, parsed_file_path)
diff --git a/xen/scripts/xen_analysis/settings.py b/xen/scripts/xen_analysis/settings.py
new file mode 100644
index 000000000000..947dfa2d50af
--- /dev/null
+++ b/xen/scripts/xen_analysis/settings.py
@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+
+import sys, re, os
+
+module_dir = os.path.dirname(os.path.realpath(__file__))
+xen_dir = os.path.realpath(module_dir + "/../..")
+repo_dir = os.path.realpath(xen_dir + "/..")
+tools_dir = os.path.realpath(xen_dir + "/tools")
+
+step_parse_tags = True
+step_build_xen = True
+step_clean_analysis = True
+
+target_build = False
+target_clean = False
+
+analysis_tool = ""
+make_forward_args = ""
+outdir = xen_dir
+
+
+def help():
+ msg="""
+Usage: {} [OPTION] ... [-- [make arguments]]
+
+This script runs the analysis on the Xen codebase.
+
+Options:
+ --build-only Run only the commands to build Xen with the optional make
+ arguments passed to the script
+ --clean-only Run only the commands to clean the analysis artifacts
+ -h, --help Print this help
+ --no-build Skip the build Xen phase
+ --no-clean Don\'t clean the analysis artifacts on exit
+ --run-coverity Run the analysis for the Coverity tool
+ --run-eclair Run the analysis for the Eclair tool
+"""
+ print(msg.format(sys.argv[0]))
+
+
+def parse_commandline(argv):
+ global analysis_tool
+ global make_forward_args
+ global outdir
+ global step_parse_tags
+ global step_build_xen
+ global step_clean_analysis
+ global target_build
+ global target_clean
+ forward_to_make = False
+ for option in argv:
+ if forward_to_make:
+ # Intercept outdir
+ outdir_regex = re.match("^O=(.*)$", option)
+ if outdir_regex:
+ outdir = outdir_regex.group(1)
+ # Forward any make arguments
+ make_forward_args = make_forward_args + " " + option
+ elif option == "--build-only":
+ target_build = True
+ elif option == "--clean-only":
+ target_clean = True
+ elif (option == "--help") or (option == "-h"):
+ help()
+ sys.exit(0)
+ elif option == "--no-build":
+ step_build_xen = False
+ elif option == "--no-clean":
+ step_clean_analysis = False
+ elif (option == "--run-coverity") or (option == "--run-eclair"):
+ analysis_tool = option[6:]
+ elif option == "--":
+ forward_to_make = True
+ else:
+ print("Invalid option: {}".format(option))
+ help()
+ sys.exit(1)
+
+ if target_build and target_clean:
+ print("--build-only is not compatible with --clean-only argument.")
+ sys.exit(1)
+
+ if target_clean:
+ step_parse_tags = False
+ step_build_xen = False
+ step_clean_analysis = True
+ return
+
+ if analysis_tool == "":
+ print("Please specify one analysis tool.")
+ help()
+ sys.exit(1)
+
+ if target_build:
+ step_parse_tags = False
+ step_build_xen = True
+ step_clean_analysis = False
diff --git a/xen/scripts/xen_analysis/tag_database.py b/xen/scripts/xen_analysis/tag_database.py
new file mode 100644
index 000000000000..ca374bbb62dd
--- /dev/null
+++ b/xen/scripts/xen_analysis/tag_database.py
@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+
+import re, json
+
+class TagDatabaseError(Exception):
+ pass
+
+# This is the dictionary for the rules that translates to proprietary comments:
+# - cppcheck: /* cppcheck-suppress[id] */
+# - coverity: /* coverity[id] */
+# - eclair: /* -E> hide id 1 "" */
+# Add entries to support more analyzers
+tool_syntax = {
+ "cppcheck":"cppcheck-suppress[VID]",
+ "coverity":"coverity[VID]",
+ "eclair":"-E> hide VID 1 \"\""
+}
+
+
+def get_xen_tag_index_type_regex(tool):
+ return r'^SAF-(\d+)-(safe|false-positive-' + tool + ')$'
+
+
+def get_xen_tag_comment_regex(tool):
+ return r'^[ \t]*/\* +(SAF-\d+-(?:safe|false-positive-' + tool + ')).*\*/$'
+
+
+# Returns a data structure containing dictionaries for safe and false-positive-*
+# Xen tags, the key is the unique index of the tag and the content is the
+# proprietary in-code comment to be used when the tag is found in the codebase
+def load_tag_database(tool, input_files, data_struct = None, schema = "safe"):
+ ret = data_struct if data_struct is not None else {
+ "safe": {},
+ "false-positive-" + tool: {}
+ }
+ database = []
+
+ # Open all input files
+ for file in input_files:
+ try:
+ with open(file, "rt") as handle:
+ content = json.load(handle)
+ database = database + content['content']
+ except json.JSONDecodeError as e:
+ raise TagDatabaseError("JSON decoding error in file {}: {}"
+ .format(file, e))
+ except Exception as e:
+ raise TagDatabaseError("Can't open file {}: {}"
+ .format(file, e))
+
+ for entry in database:
+ # If the false-positive schema is used, check the proprietary id in the
+ # 'violation-id' field, otherwise rely on the "safe" schema.
+ if schema == "false-positive":
+ proprietary_id = entry['violation-id']
+ elif tool in entry['analyser']:
+ proprietary_id = entry['analyser'][tool]
+ else:
+ proprietary_id = ""
+ if proprietary_id != "":
+ comment=tool_syntax[tool].replace("VID",proprietary_id)
+ # Regex to capture the index of the Xen tag and the schema
+ xen_tag = re.compile(get_xen_tag_index_type_regex(tool))\
+ .match(entry["id"])
+ if xen_tag and xen_tag.group(1) and xen_tag.group(2):
+ # Save in safe or false-positive-* the key {#id: "comment"}
+ id_number = int(xen_tag.group(1))
+ key = xen_tag.group(2)
+ ret[key][id_number] = "/* {} */\n".format(comment)
+ else:
+ raise TagDatabaseError(
+ "Error in database file, entry {} has unexpected "
+ "format.".format(entry["id"])
+ )
+
+ return ret
+
+
+def substitute_tags(tool, input_file, grep_struct, subs_rules):
+ try:
+ with open(grep_struct["file"], "wt") as outfile:
+
+ try:
+ with open(input_file, "rt") as infile:
+ parsed_content = infile.readlines()
+ except Exception as e:
+ raise TagDatabaseError("Issue with reading file {}: {}"
+ .format(input_file, e))
+
+ # grep_struct contains the line number where the comments are, the
+ # line number starts from 1 but in the array the first line is zero.
+ # For every line where there is a Xen tag comment, get the Xen tag
+ # that is in the capture group zero, extract from the Xen tag the
+ # unique index and the type (safe, false-positive-*) and with those
+ # information access the subs_rules dictionary to see if there is
+ # a match
+ for line_number in grep_struct["matches"]:
+ xen_tag = grep_struct["matches"][line_number][0]
+ xen_tag_regex_obj = re.compile(
+ get_xen_tag_index_type_regex(tool)).match(xen_tag)
+ id_number = int(xen_tag_regex_obj.group(1))
+ key = xen_tag_regex_obj.group(2)
+ if id_number in subs_rules[key]:
+ parsed_content[line_number-1] = subs_rules[key][id_number]
+
+ outfile.writelines(parsed_content)
+ except Exception as e:
+ raise TagDatabaseError("Issue with writing file {}: {}"
+ .format(grep_struct["file"], e))
diff --git a/xen/scripts/xen_analysis/utils.py b/xen/scripts/xen_analysis/utils.py
new file mode 100644
index 000000000000..a912d812c3df
--- /dev/null
+++ b/xen/scripts/xen_analysis/utils.py
@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+
+import os, re
+
+
+def grep(filepath, regex):
+ regObj = re.compile(regex)
+ res = { "file": filepath, "matches": {} }
+ try:
+ with open(filepath, "rt") as f:
+ line_number = 1
+ for line in f:
+ match = regObj.match(line)
+ if match:
+ res["matches"][line_number] = match.groups()
+ line_number = line_number + 1
+ except Exception as e:
+ print("WARNING: Can't open {}: {}".format(filepath, e))
+
+ # Return filename and line matches if there are
+ return res if res["matches"] else {}
+
+
+def recursive_find_file(path, filename_regex, action = None):
+ filename_reg_obj = re.compile(filename_regex)
+ res = []
+ for root, dirs, fnames in os.walk(path):
+ for fname in fnames:
+ if filename_reg_obj.match(fname):
+ if action is None:
+ res.append(os.path.join(root, fname))
+ else:
+ out = action(os.path.join(root, fname))
+ if out:
+ res.append(out)
+
+ return res
--
2.17.1
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-28 14:10 [PATCH 0/4] Static analyser finding deviation Luca Fancellu
2022-11-28 14:10 ` [PATCH 1/4] xen/scripts: add xen-analysis.py for coverity and eclair analysis Luca Fancellu
@ 2022-11-28 14:10 ` Luca Fancellu
2022-11-28 15:19 ` Jan Beulich
2022-11-30 1:05 ` Stefano Stabellini
2022-11-28 14:10 ` [PATCH 3/4] tools/misra: fix skipped rule numbers Luca Fancellu
` (2 subsequent siblings)
4 siblings, 2 replies; 31+ messages in thread
From: Luca Fancellu @ 2022-11-28 14:10 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, wei.chen, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu
Change cppcheck invocation method by using the xen-analysis.py
script using the arguments --run-cppcheck.
Now cppcheck analysis will build Xen while the analysis is performed
on the source files, it will produce a text report and an additional
html output when the script is called with --cppcheck-html.
With this patch cppcheck will benefit of platform configuration files
that will help it to understand the target of the compilation and
improve the analysis.
To do so:
- remove cppcheck rules from Makefile and move them to the script.
- Update xen-analysis.py with the code to integrate cppcheck.
- merge the script merge_cppcheck_reports.py into the xen-analysis
script package and rework the code to integrate it.
- add platform configuration files for cppcheck..
- add cppcheck-cc.sh script that is a wrapper for cppcheck and it's
used as Xen compiler, it will intercept all flags given from the
make build system and it will execute cppcheck on the compiled
file together with the file compilation.
- guarded hypercall-defs.c with CPPCHECK define because cppcheck
gets confused as the file does not contain c code.
- add false-positive-cppcheck.json file
- update documentation.
- update .gitignore
Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
---
.gitignore | 8 +-
docs/misra/cppcheck.txt | 27 +-
docs/misra/documenting-violations.rst | 7 +-
docs/misra/false-positive-cppcheck.json | 12 +
docs/misra/xen-static-analysis.rst | 42 ++-
xen/Makefile | 116 +-------
xen/include/hypercall-defs.c | 9 +
xen/scripts/xen-analysis.py | 18 +-
xen/scripts/xen_analysis/cppcheck_analysis.py | 272 ++++++++++++++++++
.../xen_analysis/cppcheck_report_utils.py | 130 +++++++++
xen/scripts/xen_analysis/generic_analysis.py | 21 +-
xen/scripts/xen_analysis/settings.py | 77 ++++-
xen/scripts/xen_analysis/utils.py | 21 +-
xen/tools/cppcheck-cc.sh | 223 ++++++++++++++
xen/tools/cppcheck-plat/arm32-wchar_t4.xml | 17 ++
xen/tools/cppcheck-plat/arm64-wchar_t2.xml | 17 ++
xen/tools/cppcheck-plat/arm64-wchar_t4.xml | 17 ++
xen/tools/cppcheck-plat/x86_64-wchar_t2.xml | 17 ++
xen/tools/cppcheck-plat/x86_64-wchar_t4.xml | 17 ++
xen/tools/merge_cppcheck_reports.py | 86 ------
20 files changed, 899 insertions(+), 255 deletions(-)
create mode 100644 docs/misra/false-positive-cppcheck.json
create mode 100644 xen/scripts/xen_analysis/cppcheck_analysis.py
create mode 100644 xen/scripts/xen_analysis/cppcheck_report_utils.py
create mode 100755 xen/tools/cppcheck-cc.sh
create mode 100644 xen/tools/cppcheck-plat/arm32-wchar_t4.xml
create mode 100644 xen/tools/cppcheck-plat/arm64-wchar_t2.xml
create mode 100644 xen/tools/cppcheck-plat/arm64-wchar_t4.xml
create mode 100644 xen/tools/cppcheck-plat/x86_64-wchar_t2.xml
create mode 100644 xen/tools/cppcheck-plat/x86_64-wchar_t4.xml
delete mode 100755 xen/tools/merge_cppcheck_reports.py
diff --git a/.gitignore b/.gitignore
index f5a66f6194dd..68566d0c2587 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,9 +7,11 @@
*.o
*.d
*.d2
-*.c.cppcheck
+*.cppcheck.txt
+*.cppcheck.xml
*.opic
*.a
+*.c.json
*.safparse
*.so
*.so.[0-9]*
@@ -282,9 +284,11 @@ xen/arch/*/efi/efi.h
xen/arch/*/efi/pe.c
xen/arch/*/efi/runtime.c
xen/arch/*/include/asm/asm-offsets.h
+xen/build-dir-cppcheck/
xen/common/config_data.S
xen/common/config.gz
xen/cppcheck-htmlreport/
+xen/cppcheck-report/
xen/cppcheck-misra.*
xen/include/headers*.chk
xen/include/compat/*
@@ -315,7 +319,7 @@ xen/xsm/flask/xenpolicy-*
tools/flask/policy/policy.conf
tools/flask/policy/xenpolicy-*
xen/xen
-xen/xen-cppcheck.xml
+xen/suppression-list.txt
xen/xen-syms
xen/xen-syms.map
xen/xen.*
diff --git a/docs/misra/cppcheck.txt b/docs/misra/cppcheck.txt
index 25d8c3050b72..f7b9f678b4d5 100644
--- a/docs/misra/cppcheck.txt
+++ b/docs/misra/cppcheck.txt
@@ -3,8 +3,7 @@ Cppcheck for Xen static and MISRA analysis
Xen can be analysed for both static analysis problems and MISRA violation using
cppcheck, the open source tool allows the creation of a report with all the
-findings. Xen has introduced the support in the Makefile so it's very easy to
-use and in this document we can see how.
+findings.
The minimum version required for cppcheck is 2.7. Note that at the time of
writing (June 2022), the version 2.8 is known to be broken [1].
@@ -38,27 +37,7 @@ Dependencies are listed in the readme.md of the project repository.
Use cppcheck to analyse Xen
===========================
-Using cppcheck integration is very simple, it requires few steps:
-
- 1) Compile Xen
- 2) call the cppcheck make target to generate a report in xml format:
- make CPPCHECK_MISRA=y cppcheck
- 3) call the cppcheck-html make target to generate a report in xml and html
- format:
- make CPPCHECK_MISRA=y cppcheck-html
-
- In case the cppcheck binaries are not in the PATH, CPPCHECK and
- CPPCHECK_HTMLREPORT variables can be overridden with the full path to the
- binaries:
-
- make -C xen \
- CPPCHECK=/path/to/cppcheck \
- CPPCHECK_HTMLREPORT=/path/to/cppcheck-htmlreport \
- CPPCHECK_MISRA=y \
- cppcheck-html
-
-The output is by default in a folder named cppcheck-htmlreport, but the name
-can be changed by passing it in the CPPCHECK_HTMLREPORT_OUTDIR variable.
-
+To analyse Xen using cppcheck, please refer to the document
+xen-static-analysis.rst, section "Analyse Xen with Cppcheck".
[1] https://sourceforge.net/p/cppcheck/discussion/general/thread/bfc3ab6c41/?limit=25
diff --git a/docs/misra/documenting-violations.rst b/docs/misra/documenting-violations.rst
index 1d23447556d2..31dafd5d4ece 100644
--- a/docs/misra/documenting-violations.rst
+++ b/docs/misra/documenting-violations.rst
@@ -51,6 +51,7 @@ Here is an example to add a new justification in safe.json::
| {
| "id": "SAF-0-safe",
| "analyser": {
+| "cppcheck": "misra-c2012-20.7",
| "coverity": "misra_c_2012_rule_20_7_violation",
| "eclair": "MC3R1.R20.7"
| },
@@ -77,9 +78,9 @@ Here is an explanation of the fields inside an object of the "content" array:
It tells the tool to substitute a Xen in-code comment having this structure:
/* SAF-0-safe [...] \*/
- analyser: it is an object containing pair of key-value strings, the key is
- the analyser, so it can be coverity or eclair, the value is the proprietary
- id corresponding on the finding, for example when coverity is used as
- analyser, the tool will translate the Xen in-code coment in this way:
+ the analyser, so it can be cppcheck, coverity or eclair, the value is the
+ proprietary id corresponding on the finding, for example when coverity is
+ used as analyser, the tool will translate the Xen in-code coment in this way:
/* SAF-0-safe [...] \*/ -> /* coverity[misra_c_2012_rule_20_7_violation] \*/
if the object doesn't have a key-value, then the corresponding in-code
comment won't be translated.
diff --git a/docs/misra/false-positive-cppcheck.json b/docs/misra/false-positive-cppcheck.json
new file mode 100644
index 000000000000..5d4da2ce6170
--- /dev/null
+++ b/docs/misra/false-positive-cppcheck.json
@@ -0,0 +1,12 @@
+{
+ "version": "1.0",
+ "content": [
+ {
+ "id": "SAF-0-false-positive-cppcheck",
+ "violation-id": "",
+ "tool-version": "",
+ "name": "Sentinel",
+ "text": "Next ID to be used"
+ }
+ ]
+}
diff --git a/docs/misra/xen-static-analysis.rst b/docs/misra/xen-static-analysis.rst
index 5b886474d4a0..2712255db1b0 100644
--- a/docs/misra/xen-static-analysis.rst
+++ b/docs/misra/xen-static-analysis.rst
@@ -7,9 +7,8 @@ The Xen codebase integrates some scripts and tools that helps the developer to
perform static analysis of the code, currently Xen supports three analysis tool
that are eclair, coverity and cppcheck.
The Xen tree has a script (xen-analysis.py) available to ease the analysis
-process and it integrates a way to suppress findings on these tools (only Eclair
-and Coverity are currently supported by the script), please check the
-documenting-violation.rst document to know more about it.
+process and it integrates a way to suppress findings on these tools, please
+check the documenting-violation.rst document to know more about it.
Analyse Xen with Coverity or Eclair
-----------------------------------
@@ -52,3 +51,40 @@ When invoking the script, the procedure below will be followed:
this step, call the script adding the --no-clean argument, but before
running again the script, call it with the --clean-only argument, that will
execute only this cleaning step.
+
+
+Analyse Xen with Cppcheck
+-------------------------
+
+Cppcheck tool is integrated in xen-analysis.py script, when using the script,
+the tool will be called on every source file compiled by the make build system.
+Here how to start the analysis with Cppcheck:
+
+ - xen-analysis.py --run-cppcheck [--cppcheck-misra] [--cppcheck-html] --
+ [optional make arguments]
+
+The command above tells the script to prepare the codebase and use Cppcheck tool
+for the analysis.
+The optional argument --cppcheck-misra activates the analysis also for MISRA
+compliance.
+The optional argument --cppcheck-html instruct cppcheck to produce an additional
+HTML report.
+
+When invoking the script for Cppcheck analysis, the followed procedure is
+similar to the one above for Coverity or Eclair, but it has some additional
+steps:
+
+ 1. This step is the same as step 1 for Coverity/Eclair.
+ 2. The cppcheck dependency are created, build directory for cppcheck analysis
+ and an header file containing internal compiler macro
+ (include/generated/compiler-def.h) are generated
+ 3. Xen compilation starts using every <additional make parameters> supplied
+ at the script invocation, but because cppcheck is not able to intercept the
+ compiled files and flags on compiler invocation, a script (cppcheck-cc.sh)
+ is passed as CC to the make system, it is a wrapper for the compiler that
+ will also execute cppcheck on every compiled file.
+ 4. After the compilation and analysis, the cppcheck report will be created
+ putting together all the cppcheck report fragments for every analysed file.
+ Cppcheck will produce a text fragment and an additional XML report fragment
+ if the script is configured to produce the HTML output.
+ 5. This step is the same as step 3 for Coverity/Eclair.
diff --git a/xen/Makefile b/xen/Makefile
index 9d0df5e2c543..77926724bcd7 100644
--- a/xen/Makefile
+++ b/xen/Makefile
@@ -457,7 +457,7 @@ endif # need-config
__all: build
-main-targets := build install uninstall clean distclean MAP cppcheck cppcheck-html
+main-targets := build install uninstall clean distclean MAP
.PHONY: $(main-targets)
ifneq ($(XEN_TARGET_ARCH),x86_32)
$(main-targets): %: _% ;
@@ -566,18 +566,16 @@ _clean:
$(Q)$(MAKE) $(clean)=tools/kconfig
find . \( -name "*.o" -o -name ".*.d" -o -name ".*.d2" \
-o -name ".*.o.tmp" -o -name "*~" -o -name "core" \
- -o -name '*.lex.c' -o -name '*.tab.[ch]' -o -name '*.c.cppcheck' \
- -o -name "*.gcno" -o -name ".*.cmd" -o -name "lib.a" \) -exec rm -f {} \;
+ -o -name '*.lex.c' -o -name '*.tab.[ch]' -o -name "*.gcno" \
+ -o -name ".*.cmd" -o -name "lib.a" \) -exec rm -f {} \;
rm -f include/asm $(TARGET) $(TARGET).gz $(TARGET)-syms $(TARGET)-syms.map
rm -f $(TARGET).efi $(TARGET).efi.map $(TARGET).efi.stripped
rm -f asm-offsets.s arch/*/include/asm/asm-offsets.h
rm -f .banner .allconfig.tmp include/xen/compile.h
- rm -f cppcheck-misra.* xen-cppcheck.xml
.PHONY: _distclean
_distclean: clean
rm -f tags TAGS cscope.files cscope.in.out cscope.out cscope.po.out GTAGS GPATH GRTAGS GSYMS .config source
- rm -rf $(CPPCHECK_HTMLREPORT_OUTDIR)
$(TARGET).gz: $(TARGET)
gzip -n -f -9 < $< > $@.new
@@ -651,111 +649,9 @@ cloc:
done; \
done | cloc --list-file=-
-# What cppcheck command to use.
-# To get proper results, it is recommended to build cppcheck manually from the
-# latest source and use CPPCHECK to give the full path to the built version.
-CPPCHECK ?= cppcheck
-
-# What cppcheck-htmlreport to use.
-# If you give the full path to a self compiled cppcheck, this should be set
-# to the full path to cppcheck-html in the htmlreport directory of cppcheck.
-# On recent distribution, this is available in the standard path.
-CPPCHECK_HTMLREPORT ?= cppcheck-htmlreport
-
-# By default we generate the report in cppcheck-htmlreport directory in the
-# build directory. This can be changed by giving a directory in this variable.
-CPPCHECK_HTMLREPORT_OUTDIR ?= cppcheck-htmlreport
-
-# By default we do not check misra rules, to enable pass "CPPCHECK_MISRA=y" to
-# make command line.
-CPPCHECK_MISRA ?= n
-
-# Compile flags to pass to cppcheck:
-# - include directories and defines Xen Makefile is passing (from CFLAGS)
-# - include config.h as this is passed directly to the compiler.
-# - define CPPCHECK as we use to disable or enable some specific part of the
-# code to solve some cppcheck issues.
-# - explicitely enable some cppcheck checks as we do not want to use "all"
-# which includes unusedFunction which gives wrong positives as we check file
-# per file.
-#
-# Compiler defines are in compiler-def.h which is included in config.h
-#
-CPPCHECKFLAGS := -DCPPCHECK --max-ctu-depth=10 \
- --enable=style,information,missingInclude \
- --include=$(srctree)/include/xen/config.h \
- -I $(srctree)/xsm/flask/include \
- -I $(srctree)/include/xen/libfdt \
- $(filter -D% -I%,$(CFLAGS))
-
-# We need to find all C files (as we are not checking assembly files) so
-# we find all generated .o files which have a .c corresponding file.
-CPPCHECKFILES := $(wildcard $(patsubst $(objtree)/%.o,$(srctree)/%.c, \
- $(filter-out $(objtree)/tools/%, \
- $(shell find $(objtree) -name "*.o"))))
-
-# Headers and files required to run cppcheck on a file
-CPPCHECKDEPS := $(objtree)/include/generated/autoconf.h \
- $(objtree)/include/generated/compiler-def.h
-
-ifeq ($(CPPCHECK_MISRA),y)
- CPPCHECKFLAGS += --addon=cppcheck-misra.json
- CPPCHECKDEPS += cppcheck-misra.json
-endif
-
-quiet_cmd_cppcheck_xml = CPPCHECK $(patsubst $(srctree)/%,%,$<)
-cmd_cppcheck_xml = $(CPPCHECK) -v -q --xml $(CPPCHECKFLAGS) \
- --output-file=$@ $<
-
-quiet_cmd_merge_cppcheck_reports = CPPCHECK-MERGE $@
-cmd_merge_cppcheck_reports = $(PYTHON) $(srctree)/tools/merge_cppcheck_reports.py $^ $@
-
-quiet_cmd_cppcheck_html = CPPCHECK-HTML $<
-cmd_cppcheck_html = $(CPPCHECK_HTMLREPORT) --file=$< --source-dir=$(srctree) \
- --report-dir=$(CPPCHECK_HTMLREPORT_OUTDIR) --title=Xen
-
-PHONY += _cppcheck _cppcheck-html cppcheck-version
-
-_cppcheck-html: xen-cppcheck.xml
- $(call if_changed,cppcheck_html)
-
-_cppcheck: xen-cppcheck.xml
-
-xen-cppcheck.xml: $(patsubst $(srctree)/%.c,$(objtree)/%.c.cppcheck,$(CPPCHECKFILES))
-ifeq ($(CPPCHECKFILES),)
- $(error Please build Xen before running cppcheck)
-endif
- $(call if_changed,merge_cppcheck_reports)
-
-$(objtree)/%.c.cppcheck: $(srctree)/%.c $(CPPCHECKDEPS) | cppcheck-version
- $(call if_changed,cppcheck_xml)
-
-cppcheck-version:
- $(Q)if ! which $(CPPCHECK) > /dev/null 2>&1; then \
- echo "Cannot find cppcheck executable: $(CPPCHECK)"; \
- exit 1; \
- fi
- $(Q)if [ "$$($(CPPCHECK) --version | awk '{print ($$2 < 2.7)}')" -eq 1 ]; then \
- echo "Please upgrade your cppcheck to version 2.7 or greater"; \
- exit 1; \
- fi
-
-# List of Misra rules to respect is written inside a doc.
-# In order to have some helpful text in the cppcheck output, generate a text
-# file containing the rules identifier, classification and text from the Xen
-# documentation file. Also generate a json file with the right arguments for
-# cppcheck in json format including the list of rules to ignore.
-#
-# convert_misra_doc.py, producing both targets at the same time, should be
-# executed only once. Utilize a pattern rule to achieve this effect, with the
-# stem kind of arbitrarily chosen to be "cppcheck".
-.PRECIOUS: %-misra.json
-%-misra.txt %-misra.json: $(XEN_ROOT)/docs/misra/rules.rst $(srctree)/tools/convert_misra_doc.py
- $(Q)$(PYTHON) $(srctree)/tools/convert_misra_doc.py -i $< -o $*-misra.txt -j $*-misra.json
-
-# Put this in generated headers this way it is cleaned by include/Makefile
-$(objtree)/include/generated/compiler-def.h:
- $(Q)$(CC) -dM -E -o $@ - < /dev/null
+# Target used by xen-analysis.sh script to retrieve Xen build system variables
+export-variable-%:
+ $(info $*=$($*))
endif #config-build
endif # need-sub-make
diff --git a/xen/include/hypercall-defs.c b/xen/include/hypercall-defs.c
index 45b6f969d2ab..3d1eb7f04a73 100644
--- a/xen/include/hypercall-defs.c
+++ b/xen/include/hypercall-defs.c
@@ -60,6 +60,13 @@
* are possible.
*/
+/*
+ * Cppcheck thinks this file needs to be analysed because it is preprocessed by
+ * the compiler, but it gets confused because this file does not contains C
+ * code. Hence protect the code when CPPCHECK is used.
+ */
+#ifndef CPPCHECK
+
#ifdef CONFIG_HVM
#define PREFIX_hvm hvm
#else
@@ -286,3 +293,5 @@ mca do do - - -
#ifndef CONFIG_PV_SHIM_EXCLUSIVE
paging_domctl_cont do do do do -
#endif
+
+#endif /* !CPPCHECK */
diff --git a/xen/scripts/xen-analysis.py b/xen/scripts/xen-analysis.py
index b5d9ef1862c9..8e50c27cd898 100755
--- a/xen/scripts/xen-analysis.py
+++ b/xen/scripts/xen-analysis.py
@@ -1,28 +1,42 @@
#!/usr/bin/env python3
import sys
-from xen_analysis import settings, generic_analysis
+from xen_analysis import settings, generic_analysis, cppcheck_analysis
from xen_analysis.generic_analysis import *
+from xen_analysis.cppcheck_analysis import *
+
+PhaseExceptions = (GetMakeVarsPhaseError, ParseTagPhaseError,
+ CppcheckDepsPhaseError, BuildPhaseError,
+ CppcheckReportPhaseError)
def main(argv):
ret_code = 0
settings.parse_commandline(argv)
try:
+ if settings.step_get_make_vars:
+ cppcheck_analysis.get_make_vars()
if settings.step_parse_tags:
generic_analysis.parse_xen_tags()
+ if settings.step_cppcheck_deps:
+ cppcheck_analysis.generate_cppcheck_deps()
if settings.step_build_xen:
generic_analysis.build_xen()
- except (ParseTagPhaseError, BuildPhaseError) as e:
+ if settings.step_cppcheck_report:
+ cppcheck_analysis.generate_cppcheck_report()
+ except PhaseExceptions as e:
print("ERROR: {}".format(e))
if hasattr(e, "errorcode"):
ret_code = e.errorcode
finally:
if settings.step_clean_analysis:
+ cppcheck_analysis.clean_analysis_artifacts()
e = generic_analysis.clean_analysis_artifacts()
if e:
print("ERROR: {}".format(e))
ret_code = 1
+ if settings.step_distclean_analysis:
+ cppcheck_analysis.clean_reports()
sys.exit(ret_code)
diff --git a/xen/scripts/xen_analysis/cppcheck_analysis.py b/xen/scripts/xen_analysis/cppcheck_analysis.py
new file mode 100644
index 000000000000..e5c2f3be3e85
--- /dev/null
+++ b/xen/scripts/xen_analysis/cppcheck_analysis.py
@@ -0,0 +1,272 @@
+#!/usr/bin/env python3
+
+import os, re, shutil
+from . import settings, utils, cppcheck_report_utils
+
+class GetMakeVarsPhaseError(Exception):
+ pass
+
+class CppcheckDepsPhaseError(Exception):
+ pass
+
+class CppcheckReportPhaseError(Exception):
+ pass
+
+CPPCHECK_BUILD_DIR = "build-dir-cppcheck"
+CPPCHECK_HTMLREPORT_OUTDIR = "cppcheck-htmlreport"
+CPPCHECK_REPORT_OUTDIR = "cppcheck-report"
+cppcheck_extra_make_args = ""
+xen_cc = ""
+
+def get_make_vars():
+ global xen_cc
+ invoke_make = utils.invoke_command(
+ "make -C {} {} export-variable-CC"
+ .format(settings.xen_dir, settings.make_forward_args),
+ True, GetMakeVarsPhaseError,
+ "Error occured retrieving make vars:\n{}"
+ )
+
+ cc_var_regex = re.search('^CC=(.*)$', invoke_make, flags=re.M)
+ if cc_var_regex:
+ xen_cc = cc_var_regex.group(1)
+
+ if xen_cc == "":
+ raise GetMakeVarsPhaseError("CC variable not found in Xen make output")
+
+
+def __generate_suppression_list(out_file):
+ # The following lambda function will return a file if it contains lines with
+ # a comment containing "cppcheck-suppress[*]" on a single line.
+ grep_action = lambda x: utils.grep(x,
+ r'^[ \t]*/\* cppcheck-suppress\[(.*)\] \*/$')
+ # Look for a list of .h files that matches the condition above
+ headers = utils.recursive_find_file(settings.xen_dir, r'.*\.h$',
+ grep_action)
+
+ try:
+ with open(out_file, "wt") as supplist_file:
+ # Add this rule to skip every finding in the autogenerated
+ # header for cppcheck
+ supplist_file.write("*:*generated/compiler-def.h\n")
+
+ for entry in headers:
+ filename = entry["file"]
+ try:
+ with open(filename, "rt") as infile:
+ header_content = infile.readlines()
+ except OSError as e:
+ raise CppcheckDepsPhaseError(
+ "Issue with reading file {}: {}"
+ .format(filename, e)
+ )
+ header_lines_len = len(header_content)
+ # line_num in entry will be header_content[line_num-1], here we
+ # are going to search the first line after line_num that have
+ # anything different from comments or empty line, because the
+ # in-code comment suppression is related to that line then.
+ for line_num in entry["matches"]:
+ cppcheck_violation_id = ""
+ tmp_line = line_num
+ # look up to which line is referring the comment at
+ # line_num (which would be header_content[tmp_line-1])
+ comment_section = False
+ while tmp_line < header_lines_len:
+ line = header_content[tmp_line]
+ # Matches a line with just optional spaces/tabs and the
+ # start of a comment '/*'
+ comment_line_starts = re.match('^[ \t]*/\*.*$', line)
+ # Matches a line with text and the end of a comment '*/'
+ comment_line_stops = re.match('^.*\*/$', line)
+ if (not comment_section) and comment_line_starts:
+ comment_section = True
+ if (len(line.strip()) != 0) and (not comment_section):
+ cppcheck_violation_id = entry["matches"][line_num][0]
+ break
+ if comment_section and comment_line_stops:
+ comment_section = False
+ tmp_line = tmp_line + 1
+
+ if cppcheck_violation_id == "":
+ raise CppcheckDepsPhaseError(
+ "Error matching cppcheck comment in {} at line {}."
+ .format(filename, line_num)
+ )
+ # Write [error id]:[filename]:[line]
+ # tmp_line refers to the array index, so translated to the
+ # file line (that begins with 1) it is tmp_line+1
+ supplist_file.write(
+ "{}:{}:{}\n".format(cppcheck_violation_id, filename,
+ (tmp_line + 1))
+ )
+ except OSError as e:
+ raise CppcheckDepsPhaseError("Issue with writing file {}: {}"
+ .format(out_file, e))
+
+
+def generate_cppcheck_deps():
+ global cppcheck_extra_make_args
+
+ # Compile flags to pass to cppcheck:
+ # - include config.h as this is passed directly to the compiler.
+ # - define CPPCHECK as we use it to disable or enable some specific part of
+ # the code to solve some cppcheck issues.
+ # - explicitely enable some cppcheck checks as we do not want to use "all"
+ # which includes unusedFunction which gives wrong positives as we check
+ # file per file.
+ # - Explicitly suppress warnings on compiler-def.h because cppcheck throws
+ # an unmatchedSuppression due to the rule we put in suppression-list.txt
+ # to skip every finding in the file.
+ #
+ # Compiler defines are in compiler-def.h which is included in config.h
+ #
+ cppcheck_flags="""
+--cppcheck-build-dir={}/{}
+ --max-ctu-depth=10
+ --enable=style,information,missingInclude
+ --template=\'{{file}}({{line}},{{column}}):{{id}}:{{severity}}:{{message}}\'
+ --relative-paths={}
+ --inline-suppr
+ --suppressions-list={}/suppression-list.txt
+ --suppress='unmatchedSuppression:*generated/compiler-def.h'
+ --include={}/include/xen/config.h
+ -DCPPCHECK
+""".format(settings.outdir, CPPCHECK_BUILD_DIR, settings.xen_dir,
+ settings.outdir, settings.xen_dir)
+
+ invoke_cppcheck = utils.invoke_command(
+ "{} --version".format(settings.cppcheck_binpath),
+ True, CppcheckDepsPhaseError,
+ "Error occured retrieving cppcheck version:\n{}\n\n{}"
+ )
+
+ version_regex = re.search('^Cppcheck (.*)$', invoke_cppcheck, flags=re.M)
+ # Currently, only cppcheck version >= 2.7 is supported, but version 2.8 is
+ # known to be broken, please refer to docs/misra/cppcheck.txt
+ if (not version_regex) or (version_regex.group(1) != "2.7"):
+ raise CppcheckDepsPhaseError(
+ "Can't find cppcheck version or version is not 2.7"
+ )
+
+ # If misra option is selected, append misra addon and generate cppcheck
+ # files for misra analysis
+ if settings.cppcheck_misra:
+ cppcheck_flags = cppcheck_flags + " --addon=cppcheck-misra.json"
+
+ utils.invoke_command(
+ "{}/convert_misra_doc.py -i {}/docs/misra/rules.rst"
+ " -o {}/cppcheck-misra.txt -j {}/cppcheck-misra.json"
+ .format(settings.tools_dir, settings.repo_dir,
+ settings.outdir, settings.outdir),
+ False, CppcheckDepsPhaseError,
+ "An error occured when running:\n{}"
+ )
+
+ # Generate compiler macros
+ os.makedirs("{}/include/generated".format(settings.outdir), exist_ok=True)
+ utils.invoke_command(
+ "{} -dM -E -o \"{}/include/generated/compiler-def.h\" - < /dev/null"
+ .format(xen_cc, settings.outdir),
+ False, CppcheckDepsPhaseError,
+ "An error occured when running:\n{}"
+ )
+
+ # Generate cppcheck suppression list
+ __generate_suppression_list(
+ "{}/suppression-list.txt".format(settings.outdir))
+
+ # Generate cppcheck build folder
+ os.makedirs("{}/{}".format(settings.outdir, CPPCHECK_BUILD_DIR),
+ exist_ok=True)
+
+ cppcheck_cc_flags = """--compiler={} --cppcheck-cmd={} {}
+ --cppcheck-plat={}/cppcheck-plat --ignore-path=tools/
+""".format(xen_cc, settings.cppcheck_binpath, cppcheck_flags,
+ settings.tools_dir)
+
+ if settings.cppcheck_html:
+ cppcheck_cc_flags = cppcheck_cc_flags + " --cppcheck-html"
+
+ # Generate the extra make argument to pass the cppcheck-cc.sh wrapper as CC
+ cppcheck_extra_make_args = "CC=\"{}/cppcheck-cc.sh {} --\"".format(
+ settings.tools_dir,
+ cppcheck_cc_flags
+ ).replace("\n", "")
+
+
+def generate_cppcheck_report():
+ # Prepare text report
+ # Look for a list of .cppcheck.txt files, those are the txt report
+ # fragments
+ fragments = utils.recursive_find_file(settings.outdir, r'.*\.cppcheck.txt$')
+ text_report_dir = "{}/{}".format(settings.outdir,
+ CPPCHECK_REPORT_OUTDIR)
+ report_filename = "{}/xen-cppcheck.txt".format(text_report_dir)
+ os.makedirs(text_report_dir, exist_ok=True)
+ try:
+ cppcheck_report_utils.cppcheck_merge_txt_fragments(fragments,
+ report_filename,
+ [settings.xen_dir])
+ except cppcheck_report_utils.CppcheckTXTReportError as e:
+ raise CppcheckReportPhaseError(e)
+
+ # If HTML output is requested
+ if settings.cppcheck_html:
+ # Look for a list of .cppcheck.xml files, those are the XML report
+ # fragments
+ fragments = utils.recursive_find_file(settings.outdir,
+ r'.*\.cppcheck.xml$')
+ html_report_dir = "{}/{}".format(settings.outdir,
+ CPPCHECK_HTMLREPORT_OUTDIR)
+ xml_filename = "{}/xen-cppcheck.xml".format(html_report_dir)
+ os.makedirs(html_report_dir, exist_ok=True)
+ try:
+ cppcheck_report_utils.cppcheck_merge_xml_fragments(fragments,
+ xml_filename,
+ settings.xen_dir,
+ settings.outdir)
+ except cppcheck_report_utils.CppcheckHTMLReportError as e:
+ raise CppcheckReportPhaseError(e)
+ # Call cppcheck-htmlreport utility to generate the HTML output
+ utils.invoke_command(
+ "{} --file={} --source-dir={} --report-dir={}/html --title=Xen"
+ .format(settings.cppcheck_htmlreport_binpath, xml_filename,
+ settings.xen_dir, html_report_dir),
+ False, CppcheckReportPhaseError,
+ "Error occured generating Cppcheck HTML report:\n{}"
+ )
+ # Strip src and obj path from *.html files
+ html_files = utils.recursive_find_file(html_report_dir, r'.*\.html$')
+ try:
+ cppcheck_report_utils.cppcheck_strip_path_html(html_files,
+ (settings.xen_dir,
+ settings.outdir))
+ except cppcheck_report_utils.CppcheckHTMLReportError as e:
+ raise CppcheckReportPhaseError(e)
+
+
+def clean_analysis_artifacts():
+ clean_files = ("suppression-list.txt", "cppcheck-misra.txt",
+ "cppcheck-misra.json")
+ cppcheck_build_dir = "{}/{}".format(settings.outdir, CPPCHECK_BUILD_DIR)
+ if os.path.isdir(cppcheck_build_dir):
+ shutil.rmtree(cppcheck_build_dir)
+ artifact_files = utils.recursive_find_file(settings.outdir,
+ r'.*\.(?:c\.json|cppcheck\.txt|cppcheck\.xml)$')
+ for file in clean_files:
+ file = "{}/{}".format(settings.outdir, file)
+ if os.path.isfile(file):
+ artifact_files.append(file)
+ for delfile in artifact_files:
+ os.remove(delfile)
+
+
+def clean_reports():
+ text_report_dir = "{}/{}".format(settings.outdir,
+ CPPCHECK_REPORT_OUTDIR)
+ html_report_dir = "{}/{}".format(settings.outdir,
+ CPPCHECK_HTMLREPORT_OUTDIR)
+ if os.path.isdir(text_report_dir):
+ shutil.rmtree(text_report_dir)
+ if os.path.isdir(html_report_dir):
+ shutil.rmtree(html_report_dir)
diff --git a/xen/scripts/xen_analysis/cppcheck_report_utils.py b/xen/scripts/xen_analysis/cppcheck_report_utils.py
new file mode 100644
index 000000000000..02440aefdfec
--- /dev/null
+++ b/xen/scripts/xen_analysis/cppcheck_report_utils.py
@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+
+import os
+from xml.etree import ElementTree
+
+class CppcheckHTMLReportError(Exception):
+ pass
+
+class CppcheckTXTReportError(Exception):
+ pass
+
+
+def __elements_equal(el1, el2):
+ if type(el1) != type(el2): return False
+
+ if el1.find('location') is None: return False
+ if el2.find('location') is None: return False
+
+ el1_location = str(el1.find('location').attrib)
+ el2_location = str(el2.find('location').attrib)
+
+ if el1_location != el2_location: return False
+
+ return True
+
+
+def __contain_element(new, lst):
+ for elem in lst:
+ if __elements_equal(new, elem):
+ return True
+ return False
+
+
+def __get_xml_root_file(filename):
+ try:
+ result_xml_root = ElementTree.parse(filename).getroot()
+ except ElementTree.ParseError as e:
+ raise CppcheckHTMLReportError(
+ "XML parsing error in {}: {}".format(filename, e)
+ )
+ return result_xml_root
+
+
+def __sanitize_cppcheck_xml_path(xml_tree, src_path, obj_path):
+ # Some path are relative to the source tree but some others are generated
+ # in the obj tree, for cppcheck when using cppcheck-htmlreport we can pass
+ # only one source tree where the files will be fetched if relative path are
+ # found. So for every path that does not exists in src tree, we guess it
+ # comes from obj tree and we put explicit absolute path to it
+ error_item_root = xml_tree.findall("errors")[0]
+ for error_item in error_item_root:
+ for location_item in error_item.findall("location"):
+ path = location_item.attrib["file"]
+ new_obj_path = obj_path + "/" + path
+ new_src_path = src_path + "/" + path
+ if (path[0] != "/") and (not os.path.isfile(new_src_path)) \
+ and os.path.isfile(new_obj_path):
+ location_item.attrib["file"] = new_obj_path
+
+
+def cppcheck_merge_xml_fragments(fragments_list, out_xml_file, src_path,
+ obj_path):
+
+ result_xml = __get_xml_root_file(fragments_list[0])
+ insert_point = result_xml.findall("errors")[0]
+ for xml_file in fragments_list[1:]:
+ xml_root = __get_xml_root_file(xml_file)
+ curr_elem_list = list(insert_point)
+ new_elem_list = list(xml_root.findall("errors")[0])
+ for xml_error_elem in new_elem_list:
+ if not __contain_element(xml_error_elem, curr_elem_list):
+ insert_point.insert(1, xml_error_elem)
+
+ if result_xml is None:
+ return False
+
+ __sanitize_cppcheck_xml_path(result_xml, src_path, obj_path)
+
+ ElementTree.ElementTree(result_xml).write(out_xml_file)
+
+ return True
+
+
+def cppcheck_merge_txt_fragments(fragments_list, out_txt_file, strip_paths):
+ try:
+ with open(out_txt_file, "wt") as outfile:
+ # Using a set will remove automatically the duplicate lines
+ text_report_content = set()
+ for file in fragments_list:
+ try:
+ with open(file, "rt") as infile:
+ frag_lines = infile.readlines()
+ except OSError as e:
+ raise CppcheckTXTReportError(
+ "Issue with reading file {}: {}"
+ .format(file, e)
+ )
+ text_report_content.update(frag_lines)
+
+ # Back to modifiable list
+ text_report_content = list(text_report_content)
+ # Strip path from report lines
+ for i in list(range(0, len(text_report_content))):
+ for path in strip_paths:
+ text_report_content[i] = text_report_content[i].replace(
+ path + "/", "")
+ # Write the final text report
+ outfile.writelines(text_report_content)
+ except OSError as e:
+ raise CppcheckTXTReportError("Issue with writing file {}: {}"
+ .format(out_txt_file, e))
+
+
+def cppcheck_strip_path_html(html_files, strip_paths):
+ for file in html_files:
+ try:
+ with open(file, "rt") as infile:
+ html_lines = infile.readlines()
+ except OSError as e:
+ raise CppcheckHTMLReportError("Issue with reading file {}: {}"
+ .format(file, e))
+ for i in list(range(0, len(html_lines))):
+ for path in strip_paths:
+ html_lines[i] = html_lines[i].replace(path + "/", "")
+ try:
+ with open(file, "wt") as outfile:
+ outfile.writelines(html_lines)
+ except OSError as e:
+ raise CppcheckHTMLReportError("Issue with writing file {}: {}"
+ .format(file, e))
diff --git a/xen/scripts/xen_analysis/generic_analysis.py b/xen/scripts/xen_analysis/generic_analysis.py
index 0b470c4ecf7d..94122aebace0 100644
--- a/xen/scripts/xen_analysis/generic_analysis.py
+++ b/xen/scripts/xen_analysis/generic_analysis.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
-import os, subprocess
-from . import settings, utils, tag_database
+import os
+from . import settings, utils, tag_database, cppcheck_analysis
class ParseTagPhaseError(Exception):
pass
@@ -60,18 +60,13 @@ def parse_xen_tags():
def build_xen():
- try:
- subprocess.run(
- "make -C {} {} build"
- .format(settings.xen_dir, settings.make_forward_args),
- shell=True, check=True
+ utils.invoke_command(
+ "make -C {} {} {} build"
+ .format(settings.xen_dir, settings.make_forward_args,
+ cppcheck_analysis.cppcheck_extra_make_args),
+ False, BuildPhaseError,
+ "Build error occured when running:\n{}"
)
- except (subprocess.CalledProcessError, subprocess.SubprocessError) as e:
- excp = BuildPhaseError(
- "Build error occured when running:\n{}".format(e.cmd)
- )
- excp.errorcode = e.returncode if hasattr(e, 'returncode') else 1
- raise excp
def clean_analysis_artifacts():
diff --git a/xen/scripts/xen_analysis/settings.py b/xen/scripts/xen_analysis/settings.py
index 947dfa2d50af..bd1faafe79a3 100644
--- a/xen/scripts/xen_analysis/settings.py
+++ b/xen/scripts/xen_analysis/settings.py
@@ -7,14 +7,23 @@ xen_dir = os.path.realpath(module_dir + "/../..")
repo_dir = os.path.realpath(xen_dir + "/..")
tools_dir = os.path.realpath(xen_dir + "/tools")
+step_get_make_vars = False
step_parse_tags = True
+step_cppcheck_deps = False
step_build_xen = True
+step_cppcheck_report = False
step_clean_analysis = True
+step_distclean_analysis = False
target_build = False
target_clean = False
+target_distclean = False
analysis_tool = ""
+cppcheck_binpath = "cppcheck"
+cppcheck_html = False
+cppcheck_htmlreport_binpath = "cppcheck-htmlreport"
+cppcheck_misra = False
make_forward_args = ""
outdir = xen_dir
@@ -26,29 +35,47 @@ Usage: {} [OPTION] ... [-- [make arguments]]
This script runs the analysis on the Xen codebase.
Options:
- --build-only Run only the commands to build Xen with the optional make
- arguments passed to the script
- --clean-only Run only the commands to clean the analysis artifacts
- -h, --help Print this help
- --no-build Skip the build Xen phase
- --no-clean Don\'t clean the analysis artifacts on exit
- --run-coverity Run the analysis for the Coverity tool
- --run-eclair Run the analysis for the Eclair tool
+ --build-only Run only the commands to build Xen with the optional
+ make arguments passed to the script
+ --clean-only Run only the commands to clean the analysis artifacts
+ --cppcheck-bin= Path to the cppcheck binary (Default: {})
+ --cppcheck-html Produce an additional HTML output report for Cppcheck
+ --cppcheck-html-bin= Path to the cppcheck-html binary (Default: {})
+ --cppcheck-misra Activate the Cppcheck MISRA analysis
+ --distclean Clean analysis artifacts and reports
+ -h, --help Print this help
+ --no-build Skip the build Xen phase
+ --no-clean Don\'t clean the analysis artifacts on exit
+ --run-coverity Run the analysis for the Coverity tool
+ --run-cppcheck Run the Cppcheck analysis tool on Xen
+ --run-eclair Run the analysis for the Eclair tool
"""
- print(msg.format(sys.argv[0]))
+ print(msg.format(sys.argv[0], cppcheck_binpath,
+ cppcheck_htmlreport_binpath))
def parse_commandline(argv):
global analysis_tool
+ global cppcheck_binpath
+ global cppcheck_html
+ global cppcheck_htmlreport_binpath
+ global cppcheck_misra
global make_forward_args
global outdir
+ global step_get_make_vars
global step_parse_tags
+ global step_cppcheck_deps
global step_build_xen
+ global step_cppcheck_report
global step_clean_analysis
+ global step_distclean_analysis
global target_build
global target_clean
+ global target_distclean
forward_to_make = False
for option in argv:
+ args_with_content_regex = re.match(r'^(--[a-z]+[a-z-]*)=(.*)$', option)
+
if forward_to_make:
# Intercept outdir
outdir_regex = re.match("^O=(.*)$", option)
@@ -60,6 +87,18 @@ def parse_commandline(argv):
target_build = True
elif option == "--clean-only":
target_clean = True
+ elif args_with_content_regex and \
+ args_with_content_regex.group(1) == "--cppcheck-bin":
+ cppcheck_binpath = args_with_content_regex.group(2)
+ elif option == "--cppcheck-html":
+ cppcheck_html = True
+ elif args_with_content_regex and \
+ args_with_content_regex.group(1) == "--cppcheck-html-bin":
+ cppcheck_htmlreport_binpath = args_with_content_regex.group(2)
+ elif option == "--cppcheck-misra":
+ cppcheck_misra = True
+ elif option == "--distclean":
+ target_distclean = True
elif (option == "--help") or (option == "-h"):
help()
sys.exit(0)
@@ -69,6 +108,11 @@ def parse_commandline(argv):
step_clean_analysis = False
elif (option == "--run-coverity") or (option == "--run-eclair"):
analysis_tool = option[6:]
+ elif (option == "--run-cppcheck"):
+ analysis_tool = "cppcheck"
+ step_get_make_vars = True
+ step_cppcheck_deps = True
+ step_cppcheck_report = True
elif option == "--":
forward_to_make = True
else:
@@ -76,13 +120,23 @@ def parse_commandline(argv):
help()
sys.exit(1)
- if target_build and target_clean:
- print("--build-only is not compatible with --clean-only argument.")
+ if target_build and (target_clean or target_distclean):
+ print("--build-only is not compatible with --clean-only/--distclean "
+ "argument.")
sys.exit(1)
+ if target_distclean:
+ # Implicit activation of clean target
+ target_clean = True
+
+ step_distclean_analysis = True
+
if target_clean:
+ step_get_make_vars = False
step_parse_tags = False
+ step_cppcheck_deps = False
step_build_xen = False
+ step_cppcheck_report = False
step_clean_analysis = True
return
@@ -95,3 +149,4 @@ def parse_commandline(argv):
step_parse_tags = False
step_build_xen = True
step_clean_analysis = False
+ step_cppcheck_report = False
diff --git a/xen/scripts/xen_analysis/utils.py b/xen/scripts/xen_analysis/utils.py
index a912d812c3df..1193e3f4631e 100644
--- a/xen/scripts/xen_analysis/utils.py
+++ b/xen/scripts/xen_analysis/utils.py
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
-import os, re
+import os, re, subprocess
def grep(filepath, regex):
@@ -35,3 +35,22 @@ def recursive_find_file(path, filename_regex, action = None):
res.append(out)
return res
+
+
+def invoke_command(command, needs_output, exeption_type = Exception,
+ exeption_msg = ""):
+ try:
+ pipe_stdout = subprocess.PIPE if (needs_output == True) else None
+ output = subprocess.run(command, shell=True, check=True,
+ stdout=pipe_stdout, stderr=subprocess.STDOUT,
+ encoding='utf8')
+ except (subprocess.CalledProcessError, subprocess.SubprocessError) as e:
+ if needs_output == True:
+ exeption_msg = exeption_msg.format(e.cmd, output.stdout)
+ else:
+ exeption_msg = exeption_msg.format(e.cmd)
+ excp = exeption_type(exeption_msg)
+ excp.errorcode = e.returncode if hasattr(e, 'returncode') else 1
+ raise excp
+
+ return output.stdout
diff --git a/xen/tools/cppcheck-cc.sh b/xen/tools/cppcheck-cc.sh
new file mode 100755
index 000000000000..e682f6b9d79d
--- /dev/null
+++ b/xen/tools/cppcheck-cc.sh
@@ -0,0 +1,223 @@
+#!/usr/bin/env bash
+
+set -e
+
+function help() {
+ cat <<EOF
+Usage: ${0} [OPTION] ... -- <compiler arguments>
+
+This script is a wrapper for cppcheck that enables it to analyse the files that
+are the target for the build, it is used in place of a selected compiler and the
+make process will run it on every file that needs to be built.
+All the arguments passed to the original compiler are forwarded to it without
+modification, furthermore, they are used to improve the cppcheck analysis.
+
+Options:
+ --compiler= Use this compiler for the build
+ --cppcheck-cmd= Command line for the cppcheck analysis.
+ --cppcheck-html Prepare for cppcheck HTML output
+ --cppcheck-plat= Path to the cppcheck platform folder
+ --ignore-path= This script won't run cppcheck on the files having this
+ path, the compiler will run anyway on them. This argument
+ can be specified multiple times.
+ -h, --help Print this help
+EOF
+}
+
+CC_FILE=""
+COMPILER=""
+CPPCHECK_HTML="n"
+CPPCHECK_PLAT_PATH=""
+CPPCHECK_TOOL=""
+CPPCHECK_TOOL_ARGS=""
+FORWARD_FLAGS=""
+IGNORE_PATH="n"
+IGNORE_PATH_LIST=""
+JDB_FILE=""
+OBJTREE_PATH=""
+
+# Variable used for arg parsing
+forward_to_cc="n"
+sm_tool_args="n"
+obj_arg_content="n"
+
+for OPTION in "$@"
+do
+ if [ "${forward_to_cc}" = "y" ]; then
+ if [[ ${OPTION} == *.c ]]
+ then
+ CC_FILE="${OPTION}"
+ elif [ "${OPTION}" = "-o" ]
+ then
+ # After -o there is the path to the obj file, flag it
+ obj_arg_content="y"
+ elif [ "${obj_arg_content}" = "y" ]
+ then
+ # This must be the path to the obj file, turn off flag and save path
+ OBJTREE_PATH="$(dirname "${OPTION}")"
+ obj_arg_content="n"
+ fi
+ # Forward any argument to the compiler
+ FORWARD_FLAGS="${FORWARD_FLAGS} ${OPTION}"
+ continue
+ fi
+ case ${OPTION} in
+ -h|--help)
+ help
+ exit 0
+ ;;
+ --compiler=*)
+ COMPILER="$(eval echo "${OPTION#*=}")"
+ sm_tool_args="n"
+ ;;
+ --cppcheck-cmd=*)
+ CPPCHECK_TOOL="$(eval echo "${OPTION#*=}")"
+ sm_tool_args="y"
+ ;;
+ --cppcheck-html)
+ CPPCHECK_HTML="y"
+ sm_tool_args="n"
+ ;;
+ --cppcheck-plat=*)
+ CPPCHECK_PLAT_PATH="$(eval echo "${OPTION#*=}")"
+ sm_tool_args="n"
+ ;;
+ --ignore-path=*)
+ IGNORE_PATH_LIST="${IGNORE_PATH_LIST} $(eval echo "${OPTION#*=}")"
+ sm_tool_args="n"
+ ;;
+ --)
+ forward_to_cc="y"
+ sm_tool_args="n"
+ ;;
+ *)
+ if [ "${sm_tool_args}" = "y" ]; then
+ CPPCHECK_TOOL_ARGS="${CPPCHECK_TOOL_ARGS} ${OPTION}"
+ else
+ echo "Invalid option ${OPTION}"
+ exit 1
+ fi
+ ;;
+ esac
+done
+
+if [ "${COMPILER}" = "" ]
+then
+ echo "--compiler arg is mandatory."
+ exit 1
+fi
+
+function print_file() {
+ local text="${1}"
+ local init_file="${2}"
+
+ if [ "${init_file}" = "y" ]
+ then
+ echo -e -n "${text}" > "${JDB_FILE}"
+ else
+ echo -e -n "${text}" >> "${JDB_FILE}"
+ fi
+}
+
+function create_jcd() {
+ local line="${1}"
+ local arg_num=0
+ local same_line=0
+
+ print_file "[\n" "y"
+ print_file " {\n"
+ print_file " \"arguments\": [\n"
+
+ for arg in ${line}; do
+ # This code prevents to put comma in the last element of the list or on
+ # sequential lines that are going to be merged
+ [ "${arg_num}" -ne 0 ] && [ "${same_line}" -eq 0 ] && print_file ",\n"
+ if [ "${same_line}" -ne 0 ]
+ then
+ print_file "${arg}\""
+ same_line=0
+ elif [ "${arg}" = "-iquote" ] || [ "${arg}" = "-I" ]
+ then
+ # cppcheck doesn't understand -iquote, substitute with -I
+ print_file " \"-I"
+ same_line=1
+ else
+ print_file " \"${arg}\""
+ fi
+ arg_num=$(( arg_num + 1 ))
+ done
+ print_file "\n"
+ print_file " ],\n"
+ print_file " \"directory\": \"$(pwd -P)\",\n"
+ print_file " \"file\": \"${CC_FILE}\"\n"
+ print_file " }\n"
+ print_file "]\n"
+}
+
+
+# Execute compiler with forwarded flags
+# Shellcheck complains about missing quotes on FORWARD_FLAGS, but they can't be
+# used here
+# shellcheck disable=SC2086
+${COMPILER} ${FORWARD_FLAGS}
+
+if [ -n "${CC_FILE}" ];
+then
+ for path in ${IGNORE_PATH_LIST}
+ do
+ if [[ ${CC_FILE} == *${path}* ]]
+ then
+ IGNORE_PATH="y"
+ echo "${0}: ${CC_FILE} ignored by --ignore-path matching *${path}*"
+ fi
+ done
+ if [ "${IGNORE_PATH}" = "n" ]
+ then
+ JDB_FILE="${OBJTREE_PATH}/$(basename "${CC_FILE}".json)"
+
+ # Prepare the Json Compilation Database for the file
+ create_jcd "${COMPILER} ${FORWARD_FLAGS}"
+
+ out_file="${OBJTREE_PATH}/$(basename "${CC_FILE%.c}".cppcheck.txt)"
+
+ # Check wchar size
+ wchar_plat_suffix="t4"
+ # sed prints the last occurence of -f(no-)short-wchar which is the one
+ # applied to the file by the compiler
+ wchar_option=$(echo "${FORWARD_FLAGS}" | \
+ sed -nre 's,.*(-f(no-)?short-wchar).*,\1,p')
+ if [ "${wchar_option}" = "-fshort-wchar" ]
+ then
+ wchar_plat_suffix="t2"
+ fi
+
+ # Select the right target platform, ARCH is generated from Xen Makefile
+ platform="${CPPCHECK_PLAT_PATH}/${ARCH}-wchar_${wchar_plat_suffix}.xml"
+ if [ ! -f "${platform}" ]
+ then
+ echo "${platform} not found!"
+ exit 1
+ fi
+
+ # Shellcheck complains about missing quotes on CPPCHECK_TOOL_ARGS, but
+ # they can't be used here
+ # shellcheck disable=SC2086
+ ${CPPCHECK_TOOL} ${CPPCHECK_TOOL_ARGS} \
+ --project="${JDB_FILE}" \
+ --output-file="${out_file}" \
+ --platform=${platform}
+
+ if [ "${CPPCHECK_HTML}" = "y" ]
+ then
+ # Shellcheck complains about missing quotes on CPPCHECK_TOOL_ARGS,
+ # but they can't be used here
+ # shellcheck disable=SC2086
+ ${CPPCHECK_TOOL} ${CPPCHECK_TOOL_ARGS} \
+ --project="${JDB_FILE}" \
+ --output-file="${out_file%.txt}.xml" \
+ --platform=${platform} \
+ -q \
+ --xml
+ fi
+ fi
+fi
diff --git a/xen/tools/cppcheck-plat/arm32-wchar_t4.xml b/xen/tools/cppcheck-plat/arm32-wchar_t4.xml
new file mode 100644
index 000000000000..3aefa7ba5c98
--- /dev/null
+++ b/xen/tools/cppcheck-plat/arm32-wchar_t4.xml
@@ -0,0 +1,17 @@
+<?xml version="1.0"?>
+<platform>
+ <char_bit>8</char_bit>
+ <default-sign>unsigned</default-sign>
+ <sizeof>
+ <short>2</short>
+ <int>4</int>
+ <long>4</long>
+ <long-long>8</long-long>
+ <float>4</float>
+ <double>8</double>
+ <long-double>8</long-double>
+ <pointer>4</pointer>
+ <size_t>4</size_t>
+ <wchar_t>4</wchar_t>
+ </sizeof>
+</platform>
diff --git a/xen/tools/cppcheck-plat/arm64-wchar_t2.xml b/xen/tools/cppcheck-plat/arm64-wchar_t2.xml
new file mode 100644
index 000000000000..e345b934a986
--- /dev/null
+++ b/xen/tools/cppcheck-plat/arm64-wchar_t2.xml
@@ -0,0 +1,17 @@
+<?xml version="1.0"?>
+<platform>
+ <char_bit>8</char_bit>
+ <default-sign>unsigned</default-sign>
+ <sizeof>
+ <short>2</short>
+ <int>4</int>
+ <long>8</long>
+ <long-long>8</long-long>
+ <float>4</float>
+ <double>8</double>
+ <long-double>16</long-double>
+ <pointer>8</pointer>
+ <size_t>4</size_t>
+ <wchar_t>2</wchar_t>
+ </sizeof>
+</platform>
diff --git a/xen/tools/cppcheck-plat/arm64-wchar_t4.xml b/xen/tools/cppcheck-plat/arm64-wchar_t4.xml
new file mode 100644
index 000000000000..952b3640c91d
--- /dev/null
+++ b/xen/tools/cppcheck-plat/arm64-wchar_t4.xml
@@ -0,0 +1,17 @@
+<?xml version="1.0"?>
+<platform>
+ <char_bit>8</char_bit>
+ <default-sign>unsigned</default-sign>
+ <sizeof>
+ <short>2</short>
+ <int>4</int>
+ <long>8</long>
+ <long-long>8</long-long>
+ <float>4</float>
+ <double>8</double>
+ <long-double>16</long-double>
+ <pointer>8</pointer>
+ <size_t>4</size_t>
+ <wchar_t>4</wchar_t>
+ </sizeof>
+</platform>
diff --git a/xen/tools/cppcheck-plat/x86_64-wchar_t2.xml b/xen/tools/cppcheck-plat/x86_64-wchar_t2.xml
new file mode 100644
index 000000000000..b2dc2fb2cc50
--- /dev/null
+++ b/xen/tools/cppcheck-plat/x86_64-wchar_t2.xml
@@ -0,0 +1,17 @@
+<?xml version="1.0"?>
+<platform>
+ <char_bit>8</char_bit>
+ <default-sign>unsigned</default-sign>
+ <sizeof>
+ <short>2</short>
+ <int>4</int>
+ <long>8</long>
+ <long-long>8</long-long>
+ <float>4</float>
+ <double>8</double>
+ <long-double>16</long-double>
+ <pointer>8</pointer>
+ <size_t>8</size_t>
+ <wchar_t>2</wchar_t>
+ </sizeof>
+</platform>
diff --git a/xen/tools/cppcheck-plat/x86_64-wchar_t4.xml b/xen/tools/cppcheck-plat/x86_64-wchar_t4.xml
new file mode 100644
index 000000000000..21d97b611505
--- /dev/null
+++ b/xen/tools/cppcheck-plat/x86_64-wchar_t4.xml
@@ -0,0 +1,17 @@
+<?xml version="1.0"?>
+<platform>
+ <char_bit>8</char_bit>
+ <default-sign>unsigned</default-sign>
+ <sizeof>
+ <short>2</short>
+ <int>4</int>
+ <long>8</long>
+ <long-long>8</long-long>
+ <float>4</float>
+ <double>8</double>
+ <long-double>16</long-double>
+ <pointer>8</pointer>
+ <size_t>8</size_t>
+ <wchar_t>4</wchar_t>
+ </sizeof>
+</platform>
diff --git a/xen/tools/merge_cppcheck_reports.py b/xen/tools/merge_cppcheck_reports.py
deleted file mode 100755
index 1c1b63ba56b8..000000000000
--- a/xen/tools/merge_cppcheck_reports.py
+++ /dev/null
@@ -1,86 +0,0 @@
-#!/usr/bin/env python
-
-"""
-This script acts as a tool to merge XML files created by cppcheck.
-Usage:
- merge_cppcheck_reports.py [FILES] [OUTPUT]
-
- FILES - list of XML files with extension .cppcheck
- OUTPUT - file to store results (with .xml extension).
- If not specified, the script will print results to stdout.
-"""
-
-import sys
-from xml.etree import ElementTree
-
-def elements_equal(el1, el2):
- if type(el1) != type(el2): return False
-
- if el1.find('location') is None: return False
- if el2.find('location') is None: return False
-
- el1_location = str(el1.find('location').attrib)
- el2_location = str(el2.find('location').attrib)
-
- if el1_location != el2_location: return False
-
- return True
-
-def contain_element(new, lst):
- for elem in lst:
- if elements_equal(new, elem):
- return True
- return False
-
-def merge(files):
- try:
- result_xml_root = ElementTree.parse(files[0]).getroot()
- except:
- print("Xml parsing error in %s\n" % (files[0]))
- print("Please upgrade your cppcheck to version 2.7 or greater")
- sys.exit(1)
- insert_point = result_xml_root.findall("errors")[0]
- curr = 1
- total = len(files)
- numelem = len(insert_point)
- for xml_file in files[1:]:
- try:
- xml_root = ElementTree.parse(xml_file).getroot()
- except:
- print("Xml parsing error in %s\n" % (xml_file))
- print("Please upgrade your cppcheck to version 2.7 or greater")
- sys.exit(1)
- curr_elem_list = list(insert_point)
- new_elem_list = list(xml_root.findall("errors")[0])
- for xml_error_elem in new_elem_list:
- if not contain_element(xml_error_elem, curr_elem_list):
- insert_point.insert(1,xml_error_elem)
- numelem = numelem + 1
- curr = curr + 1
- sys.stdout.write('\r')
- sys.stdout.write(" %d / %d" % (curr,total))
- sys.stdout.flush()
-
- sys.stdout.write('\r\n')
- print("Done: %d elements" % (numelem))
- return result_xml_root
-
-def run():
- files = []
- output = None
- for i in sys.argv[1:]:
- output = i if '.xml' in i else None
- files.append(i) if '.cppcheck' in i else None
-
- result = merge(files)
-
- if result is None:
- return
-
- if output is not None:
- ElementTree.ElementTree(result).write(output)
- else:
- print(ElementTree.tostring(result).decode('utf-8'))
-
-if __name__ == '__main__':
- run()
--
2.17.1
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 3/4] tools/misra: fix skipped rule numbers
2022-11-28 14:10 [PATCH 0/4] Static analyser finding deviation Luca Fancellu
2022-11-28 14:10 ` [PATCH 1/4] xen/scripts: add xen-analysis.py for coverity and eclair analysis Luca Fancellu
2022-11-28 14:10 ` [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script Luca Fancellu
@ 2022-11-28 14:10 ` Luca Fancellu
2022-11-29 23:51 ` Stefano Stabellini
2022-11-28 14:10 ` [PATCH 4/4] xen: Justify linker script defined symbols in include/xen/kernel.h Luca Fancellu
2022-11-29 1:55 ` [PATCH 0/4] Static analyser finding deviation Stefano Stabellini
4 siblings, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-11-28 14:10 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, wei.chen, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu
Currently the script convert_misra_doc.py is using a loop through
range(1,22) to enumerate rules that needs to be skipped, however
range function does not include the stop counter in the enumeration
ending up into list rules until 21.21 instead of including rule 22.
Fix the issue using a dictionary that list the rules in misra c2012.
Fixes: 57caa5375321 ("xen: Add MISRA support to cppcheck make rule")
Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
---
xen/tools/convert_misra_doc.py | 32 ++++++++++++++++++++++++++++++--
1 file changed, 30 insertions(+), 2 deletions(-)
diff --git a/xen/tools/convert_misra_doc.py b/xen/tools/convert_misra_doc.py
index caa4487f645f..13074d8a2e91 100755
--- a/xen/tools/convert_misra_doc.py
+++ b/xen/tools/convert_misra_doc.py
@@ -14,6 +14,34 @@ Usage:
import sys, getopt, re
+# MISRA rule are identified by two numbers, e.g. Rule 1.2, the main rule number
+# and a sub-number. This dictionary contains the number of the MISRA rule as key
+# and the maximum sub-number for that rule as value.
+misra_c2012_rules = {
+ 1:4,
+ 2:7,
+ 3:2,
+ 4:2,
+ 5:9,
+ 6:2,
+ 7:4,
+ 8:14,
+ 9:5,
+ 10:8,
+ 11:9,
+ 12:5,
+ 13:6,
+ 14:4,
+ 15:7,
+ 16:7,
+ 17:8,
+ 18:8,
+ 19:2,
+ 20:14,
+ 21:21,
+ 22:10
+}
+
def main(argv):
infile = ''
outfile = ''
@@ -142,8 +170,8 @@ def main(argv):
skip_list = []
# Search for missing rules and add a dummy text with the rule number
- for i in list(range(1,22)):
- for j in list(range(1,22)):
+ for i in misra_c2012_rules:
+ for j in list(range(1,misra_c2012_rules[i]+1)):
if str(i) + '.' + str(j) not in rule_list:
outstr.write('Rule ' + str(i) + '.' + str(j) + '\n')
outstr.write('No description for rule ' + str(i) + '.' + str(j)
--
2.17.1
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [PATCH 4/4] xen: Justify linker script defined symbols in include/xen/kernel.h
2022-11-28 14:10 [PATCH 0/4] Static analyser finding deviation Luca Fancellu
` (2 preceding siblings ...)
2022-11-28 14:10 ` [PATCH 3/4] tools/misra: fix skipped rule numbers Luca Fancellu
@ 2022-11-28 14:10 ` Luca Fancellu
2022-11-28 15:19 ` Jan Beulich
2022-11-29 1:55 ` Stefano Stabellini
2022-11-29 1:55 ` [PATCH 0/4] Static analyser finding deviation Stefano Stabellini
4 siblings, 2 replies; 31+ messages in thread
From: Luca Fancellu @ 2022-11-28 14:10 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, wei.chen, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu
Eclair and Coverity found violation of the MISRA rule 8.6 for the
symbols _start, _end, start, _stext, _etext, _srodata, _erodata,
_sinittext, _einittext which are declared in
xen/include/xen/kernel.h.
All those symbols are defined by the liker script so we can deviate
from the rule 8.6 for these cases.
Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
---
docs/misra/safe.json | 9 +++++++++
xen/include/xen/kernel.h | 4 ++++
2 files changed, 13 insertions(+)
diff --git a/docs/misra/safe.json b/docs/misra/safe.json
index e079d3038120..e3c8a1d8eb36 100644
--- a/docs/misra/safe.json
+++ b/docs/misra/safe.json
@@ -3,6 +3,15 @@
"content": [
{
"id": "SAF-0-safe",
+ "analyser": {
+ "eclair": "MC3R1.R8.6",
+ "coverity": "misra_c_2012_rule_8_6_violation"
+ },
+ "name": "Rule 8.6: linker script defined symbols",
+ "text": "It is safe to declare this symbol because it is defined in the linker script."
+ },
+ {
+ "id": "SAF-1-safe",
"analyser": {},
"name": "Sentinel",
"text": "Next ID to be used"
diff --git a/xen/include/xen/kernel.h b/xen/include/xen/kernel.h
index 8cd142032d3b..f1a7713784fc 100644
--- a/xen/include/xen/kernel.h
+++ b/xen/include/xen/kernel.h
@@ -65,24 +65,28 @@
1; \
})
+/* SAF-0-safe */
extern char _start[], _end[], start[];
#define is_kernel(p) ({ \
char *__p = (char *)(unsigned long)(p); \
(__p >= _start) && (__p < _end); \
})
+/* SAF-0-safe */
extern char _stext[], _etext[];
#define is_kernel_text(p) ({ \
char *__p = (char *)(unsigned long)(p); \
(__p >= _stext) && (__p < _etext); \
})
+/* SAF-0-safe */
extern const char _srodata[], _erodata[];
#define is_kernel_rodata(p) ({ \
const char *__p = (const char *)(unsigned long)(p); \
(__p >= _srodata) && (__p < _erodata); \
})
+/* SAF-0-safe */
extern char _sinittext[], _einittext[];
#define is_kernel_inittext(p) ({ \
char *__p = (char *)(unsigned long)(p); \
--
2.17.1
^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-28 14:10 ` [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script Luca Fancellu
@ 2022-11-28 15:19 ` Jan Beulich
2022-11-28 15:37 ` Luca Fancellu
2022-11-30 1:05 ` Stefano Stabellini
1 sibling, 1 reply; 31+ messages in thread
From: Jan Beulich @ 2022-11-28 15:19 UTC (permalink / raw)
To: Luca Fancellu
Cc: bertrand.marquis, wei.chen, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Wei Liu, xen-devel
On 28.11.2022 15:10, Luca Fancellu wrote:
> Change cppcheck invocation method by using the xen-analysis.py
> script using the arguments --run-cppcheck.
>
> Now cppcheck analysis will build Xen while the analysis is performed
> on the source files, it will produce a text report and an additional
> html output when the script is called with --cppcheck-html.
>
> With this patch cppcheck will benefit of platform configuration files
> that will help it to understand the target of the compilation and
> improve the analysis.
>
> To do so:
> - remove cppcheck rules from Makefile and move them to the script.
> - Update xen-analysis.py with the code to integrate cppcheck.
> - merge the script merge_cppcheck_reports.py into the xen-analysis
> script package and rework the code to integrate it.
> - add platform configuration files for cppcheck..
> - add cppcheck-cc.sh script that is a wrapper for cppcheck and it's
> used as Xen compiler, it will intercept all flags given from the
> make build system and it will execute cppcheck on the compiled
> file together with the file compilation.
> - guarded hypercall-defs.c with CPPCHECK define because cppcheck
> gets confused as the file does not contain c code.
> - add false-positive-cppcheck.json file
> - update documentation.
> - update .gitignore
>
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
Just two and a half questions, not a full review:
> ---
> .gitignore | 8 +-
> docs/misra/cppcheck.txt | 27 +-
> docs/misra/documenting-violations.rst | 7 +-
> docs/misra/false-positive-cppcheck.json | 12 +
> docs/misra/xen-static-analysis.rst | 42 ++-
> xen/Makefile | 116 +-------
> xen/include/hypercall-defs.c | 9 +
> xen/scripts/xen-analysis.py | 18 +-
> xen/scripts/xen_analysis/cppcheck_analysis.py | 272 ++++++++++++++++++
> .../xen_analysis/cppcheck_report_utils.py | 130 +++++++++
> xen/scripts/xen_analysis/generic_analysis.py | 21 +-
> xen/scripts/xen_analysis/settings.py | 77 ++++-
> xen/scripts/xen_analysis/utils.py | 21 +-
> xen/tools/cppcheck-cc.sh | 223 ++++++++++++++
> xen/tools/cppcheck-plat/arm32-wchar_t4.xml | 17 ++
> xen/tools/cppcheck-plat/arm64-wchar_t2.xml | 17 ++
> xen/tools/cppcheck-plat/arm64-wchar_t4.xml | 17 ++
> xen/tools/cppcheck-plat/x86_64-wchar_t2.xml | 17 ++
> xen/tools/cppcheck-plat/x86_64-wchar_t4.xml | 17 ++
What are these last five files about? There's nothing about them in
the description afaics.
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/cppcheck_analysis.py
> @@ -0,0 +1,272 @@
> +#!/usr/bin/env python3
> +
> +import os, re, shutil
> +from . import settings, utils, cppcheck_report_utils
> +
> +class GetMakeVarsPhaseError(Exception):
> + pass
> +
> +class CppcheckDepsPhaseError(Exception):
> + pass
> +
> +class CppcheckReportPhaseError(Exception):
> + pass
> +
> +CPPCHECK_BUILD_DIR = "build-dir-cppcheck"
> +CPPCHECK_HTMLREPORT_OUTDIR = "cppcheck-htmlreport"
> +CPPCHECK_REPORT_OUTDIR = "cppcheck-report"
> +cppcheck_extra_make_args = ""
> +xen_cc = ""
> +
> +def get_make_vars():
> + global xen_cc
> + invoke_make = utils.invoke_command(
> + "make -C {} {} export-variable-CC"
> + .format(settings.xen_dir, settings.make_forward_args),
> + True, GetMakeVarsPhaseError,
> + "Error occured retrieving make vars:\n{}"
> + )
> +
> + cc_var_regex = re.search('^CC=(.*)$', invoke_make, flags=re.M)
> + if cc_var_regex:
> + xen_cc = cc_var_regex.group(1)
> +
> + if xen_cc == "":
> + raise GetMakeVarsPhaseError("CC variable not found in Xen make output")
What use is CC without CFLAGS? Once again the description could do
with containing some information on what's going on here, and why
you need to export any variables in the first place.
> +if [ -n "${CC_FILE}" ];
> +then
> + for path in ${IGNORE_PATH_LIST}
> + do
> + if [[ ${CC_FILE} == *${path}* ]]
> + then
> + IGNORE_PATH="y"
> + echo "${0}: ${CC_FILE} ignored by --ignore-path matching *${path}*"
> + fi
> + done
> + if [ "${IGNORE_PATH}" = "n" ]
> + then
> + JDB_FILE="${OBJTREE_PATH}/$(basename "${CC_FILE}".json)"
> +
> + # Prepare the Json Compilation Database for the file
> + create_jcd "${COMPILER} ${FORWARD_FLAGS}"
> +
> + out_file="${OBJTREE_PATH}/$(basename "${CC_FILE%.c}".cppcheck.txt)"
> +
> + # Check wchar size
> + wchar_plat_suffix="t4"
> + # sed prints the last occurence of -f(no-)short-wchar which is the one
> + # applied to the file by the compiler
> + wchar_option=$(echo "${FORWARD_FLAGS}" | \
> + sed -nre 's,.*(-f(no-)?short-wchar).*,\1,p')
> + if [ "${wchar_option}" = "-fshort-wchar" ]
> + then
> + wchar_plat_suffix="t2"
> + fi
> +
> + # Select the right target platform, ARCH is generated from Xen Makefile
> + platform="${CPPCHECK_PLAT_PATH}/${ARCH}-wchar_${wchar_plat_suffix}.xml"
Purely cosmetic, but still: Why is "t" part of the suffix rather than
being part of the expression here (allowing e.g. a grep for "wchar_t"
to hit here)?
Jan
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 4/4] xen: Justify linker script defined symbols in include/xen/kernel.h
2022-11-28 14:10 ` [PATCH 4/4] xen: Justify linker script defined symbols in include/xen/kernel.h Luca Fancellu
@ 2022-11-28 15:19 ` Jan Beulich
2022-11-29 1:55 ` Stefano Stabellini
1 sibling, 0 replies; 31+ messages in thread
From: Jan Beulich @ 2022-11-28 15:19 UTC (permalink / raw)
To: Luca Fancellu
Cc: bertrand.marquis, wei.chen, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Wei Liu, xen-devel
On 28.11.2022 15:10, Luca Fancellu wrote:
> Eclair and Coverity found violation of the MISRA rule 8.6 for the
> symbols _start, _end, start, _stext, _etext, _srodata, _erodata,
> _sinittext, _einittext which are declared in
> xen/include/xen/kernel.h.
> All those symbols are defined by the liker script so we can deviate
> from the rule 8.6 for these cases.
>
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-28 15:19 ` Jan Beulich
@ 2022-11-28 15:37 ` Luca Fancellu
2022-11-29 9:42 ` Jan Beulich
0 siblings, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-11-28 15:37 UTC (permalink / raw)
To: Jan Beulich
Cc: Bertrand Marquis, Wei Chen, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Wei Liu, xen-devel
> On 28 Nov 2022, at 15:19, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 28.11.2022 15:10, Luca Fancellu wrote:
>> Change cppcheck invocation method by using the xen-analysis.py
>> script using the arguments --run-cppcheck.
>>
>> Now cppcheck analysis will build Xen while the analysis is performed
>> on the source files, it will produce a text report and an additional
>> html output when the script is called with --cppcheck-html.
>>
>> With this patch cppcheck will benefit of platform configuration files
>> that will help it to understand the target of the compilation and
>> improve the analysis.
>>
>> To do so:
>> - remove cppcheck rules from Makefile and move them to the script.
>> - Update xen-analysis.py with the code to integrate cppcheck.
>> - merge the script merge_cppcheck_reports.py into the xen-analysis
>> script package and rework the code to integrate it.
>> - add platform configuration files for cppcheck..
>> - add cppcheck-cc.sh script that is a wrapper for cppcheck and it's
>> used as Xen compiler, it will intercept all flags given from the
>> make build system and it will execute cppcheck on the compiled
>> file together with the file compilation.
>> - guarded hypercall-defs.c with CPPCHECK define because cppcheck
>> gets confused as the file does not contain c code.
>> - add false-positive-cppcheck.json file
>> - update documentation.
>> - update .gitignore
>>
>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>
> Just two and a half questions, not a full review:
>
>> ---
>> .gitignore | 8 +-
>> docs/misra/cppcheck.txt | 27 +-
>> docs/misra/documenting-violations.rst | 7 +-
>> docs/misra/false-positive-cppcheck.json | 12 +
>> docs/misra/xen-static-analysis.rst | 42 ++-
>> xen/Makefile | 116 +-------
>> xen/include/hypercall-defs.c | 9 +
>> xen/scripts/xen-analysis.py | 18 +-
>> xen/scripts/xen_analysis/cppcheck_analysis.py | 272 ++++++++++++++++++
>> .../xen_analysis/cppcheck_report_utils.py | 130 +++++++++
>> xen/scripts/xen_analysis/generic_analysis.py | 21 +-
>> xen/scripts/xen_analysis/settings.py | 77 ++++-
>> xen/scripts/xen_analysis/utils.py | 21 +-
>> xen/tools/cppcheck-cc.sh | 223 ++++++++++++++
>> xen/tools/cppcheck-plat/arm32-wchar_t4.xml | 17 ++
>> xen/tools/cppcheck-plat/arm64-wchar_t2.xml | 17 ++
>> xen/tools/cppcheck-plat/arm64-wchar_t4.xml | 17 ++
>> xen/tools/cppcheck-plat/x86_64-wchar_t2.xml | 17 ++
>> xen/tools/cppcheck-plat/x86_64-wchar_t4.xml | 17 ++
>
> What are these last five files about? There's nothing about them in
> the description afaics.
They are cppcheck platform configuration files, they help cppcheck to understand
the size of the types depending on the target of the compilation.
This section in the commit message is to introduce them:
With this patch cppcheck will benefit of platform configuration files
that will help it to understand the target of the compilation and
improve the analysis.
Do you think I should say it differently? Or maybe say that they reside in xen/tools/cppcheck-plat/ ?
>
>> --- /dev/null
>> +++ b/xen/scripts/xen_analysis/cppcheck_analysis.py
>> @@ -0,0 +1,272 @@
>> +#!/usr/bin/env python3
>> +
>> +import os, re, shutil
>> +from . import settings, utils, cppcheck_report_utils
>> +
>> +class GetMakeVarsPhaseError(Exception):
>> + pass
>> +
>> +class CppcheckDepsPhaseError(Exception):
>> + pass
>> +
>> +class CppcheckReportPhaseError(Exception):
>> + pass
>> +
>> +CPPCHECK_BUILD_DIR = "build-dir-cppcheck"
>> +CPPCHECK_HTMLREPORT_OUTDIR = "cppcheck-htmlreport"
>> +CPPCHECK_REPORT_OUTDIR = "cppcheck-report"
>> +cppcheck_extra_make_args = ""
>> +xen_cc = ""
>> +
>> +def get_make_vars():
>> + global xen_cc
>> + invoke_make = utils.invoke_command(
>> + "make -C {} {} export-variable-CC"
>> + .format(settings.xen_dir, settings.make_forward_args),
>> + True, GetMakeVarsPhaseError,
>> + "Error occured retrieving make vars:\n{}"
>> + )
>> +
>> + cc_var_regex = re.search('^CC=(.*)$', invoke_make, flags=re.M)
>> + if cc_var_regex:
>> + xen_cc = cc_var_regex.group(1)
>> +
>> + if xen_cc == "":
>> + raise GetMakeVarsPhaseError("CC variable not found in Xen make output")
>
> What use is CC without CFLAGS? Once again the description could do
> with containing some information on what's going on here, and why
> you need to export any variables in the first place.
We don’t need CFLAGS here, we need only CC to generate include/generated/compiler-def.h and
to pass CC to the cppcheck-cc.sh --compiler argument.
Would a comment in the code be ok?
>
>> +if [ -n "${CC_FILE}" ];
>> +then
>> + for path in ${IGNORE_PATH_LIST}
>> + do
>> + if [[ ${CC_FILE} == *${path}* ]]
>> + then
>> + IGNORE_PATH="y"
>> + echo "${0}: ${CC_FILE} ignored by --ignore-path matching *${path}*"
>> + fi
>> + done
>> + if [ "${IGNORE_PATH}" = "n" ]
>> + then
>> + JDB_FILE="${OBJTREE_PATH}/$(basename "${CC_FILE}".json)"
>> +
>> + # Prepare the Json Compilation Database for the file
>> + create_jcd "${COMPILER} ${FORWARD_FLAGS}"
>> +
>> + out_file="${OBJTREE_PATH}/$(basename "${CC_FILE%.c}".cppcheck.txt)"
>> +
>> + # Check wchar size
>> + wchar_plat_suffix="t4"
>> + # sed prints the last occurence of -f(no-)short-wchar which is the one
>> + # applied to the file by the compiler
>> + wchar_option=$(echo "${FORWARD_FLAGS}" | \
>> + sed -nre 's,.*(-f(no-)?short-wchar).*,\1,p')
>> + if [ "${wchar_option}" = "-fshort-wchar" ]
>> + then
>> + wchar_plat_suffix="t2"
>> + fi
>> +
>> + # Select the right target platform, ARCH is generated from Xen Makefile
>> + platform="${CPPCHECK_PLAT_PATH}/${ARCH}-wchar_${wchar_plat_suffix}.xml"
>
> Purely cosmetic, but still: Why is "t" part of the suffix rather than
> being part of the expression here (allowing e.g. a grep for "wchar_t"
> to hit here)?
No reason, I don’t have a strong objection to change it
>
> Jan
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 1/4] xen/scripts: add xen-analysis.py for coverity and eclair analysis
2022-11-28 14:10 ` [PATCH 1/4] xen/scripts: add xen-analysis.py for coverity and eclair analysis Luca Fancellu
@ 2022-11-29 1:39 ` Stefano Stabellini
0 siblings, 0 replies; 31+ messages in thread
From: Stefano Stabellini @ 2022-11-29 1:39 UTC (permalink / raw)
To: Luca Fancellu
Cc: xen-devel, bertrand.marquis, wei.chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Stefano Stabellini,
Wei Liu
On Mon, 28 Nov 2022, Luca Fancellu wrote:
> Add new script for coverity/eclair analysis tool that will enable
> the procedure to suppress findings when these tool are used.
> The procedure is documented in docs/misra/documenting-violations.rst
> and the script is documented in docs/misra/xen-static-analysis.rst.
>
> Add in docs/misra/ the files safe.json and
> false-positive-{coverity,eclair}.json that are JSON files containing
> the data structures for the justifications, they are used by the
> analysis script to link the Xen tags to the proprietary tool comment.
>
> Add docs/misra/documenting-violations.rst to explain how to add
> justifications.
>
> Add docs/misra/xen-static-analysis.rst to explain how to use the
> script to analyse Xen.
>
> Add analysis artifacts files to .gitignore.
>
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
I would like to give my reviewed-by for all the docs and docs changes:
docs/misra/documenting-violations.rst
docs/misra/xen-static-analysis.rst
docs/misra/safe.json
docs/misra/false-positive-eclair.json
docs/misra/false-positive-coverity.json
I also checked that their conversion to html format works correctly.
I think the docs could go in as is if moved to a separate patch.
> ---
> .gitignore | 1 +
> docs/misra/documenting-violations.rst | 191 +++++++++++++++++++
> docs/misra/false-positive-coverity.json | 12 ++
> docs/misra/false-positive-eclair.json | 12 ++
> docs/misra/safe.json | 11 ++
> docs/misra/xen-static-analysis.rst | 54 ++++++
> xen/scripts/xen-analysis.py | 31 +++
> xen/scripts/xen_analysis/__init__.py | 0
> xen/scripts/xen_analysis/generic_analysis.py | 93 +++++++++
> xen/scripts/xen_analysis/settings.py | 97 ++++++++++
> xen/scripts/xen_analysis/tag_database.py | 109 +++++++++++
> xen/scripts/xen_analysis/utils.py | 37 ++++
> 12 files changed, 648 insertions(+)
> create mode 100644 docs/misra/documenting-violations.rst
> create mode 100644 docs/misra/false-positive-coverity.json
> create mode 100644 docs/misra/false-positive-eclair.json
> create mode 100644 docs/misra/safe.json
> create mode 100644 docs/misra/xen-static-analysis.rst
> create mode 100755 xen/scripts/xen-analysis.py
> create mode 100644 xen/scripts/xen_analysis/__init__.py
> create mode 100644 xen/scripts/xen_analysis/generic_analysis.py
> create mode 100644 xen/scripts/xen_analysis/settings.py
> create mode 100644 xen/scripts/xen_analysis/tag_database.py
> create mode 100644 xen/scripts/xen_analysis/utils.py
>
> diff --git a/.gitignore b/.gitignore
> index ea3243af9dde..f5a66f6194dd 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -10,6 +10,7 @@
> *.c.cppcheck
> *.opic
> *.a
> +*.safparse
> *.so
> *.so.[0-9]*
> *.bin
> diff --git a/docs/misra/documenting-violations.rst b/docs/misra/documenting-violations.rst
> new file mode 100644
> index 000000000000..1d23447556d2
> --- /dev/null
> +++ b/docs/misra/documenting-violations.rst
> @@ -0,0 +1,191 @@
> +.. SPDX-License-Identifier: CC-BY-4.0
> +
> +Documenting violations
> +======================
> +
> +Static analysers are used on the Xen codebase for both static analysis and MISRA
> +compliance.
> +There might be the need to suppress some findings instead of fixing them and
> +many tools permit the usage of in-code comments that suppress findings so that
> +they are not shown in the final report.
> +
> +Xen includes a tool capable of translating a specific comment used in its
> +codebase to the right proprietary in-code comment understandable by the selected
> +analyser that suppress its finding.
> +
> +In the Xen codebase, these tags will be used to document and suppress findings:
> +
> + - SAF-X-safe: This tag means that the next line of code contains a finding, but
> + the non compliance to the checker is analysed and demonstrated to be safe.
> + - SAF-X-false-positive-<tool>: This tag means that the next line of code
> + contains a finding, but the finding is a bug of the tool.
> +
> +SAF stands for Static Analyser Finding, the X is a placeholder for a positive
> +number that starts from zero, the number after SAF- shall be incremental and
> +unique, base ten notation and without leading zeros.
> +
> +Entries in the database shall never be removed, even if they are not used
> +anymore in the code (if a patch is removing or modifying the faulty line).
> +This is to make sure that numbers are not reused which could lead to conflicts
> +with old branches or misleading justifications.
> +
> +An entry can be reused in multiple places in the code to suppress a finding if
> +and only if the justification holds for the same non-compliance to the coding
> +standard.
> +
> +An orphan entry, that is an entry who was justifying a finding in the code, but
> +later that code was removed and there is no other use of that entry in the code,
> +can be reused as long as the justification for the finding holds. This is done
> +to avoid the allocation of a new entry with exactly the same justification, that
> +would lead to waste of space and maintenance issues of the database.
> +
> +The files where to store all the justifications are in xen/docs/misra/ and are
> +named as safe.json and false-positive-<tool>.json, they have JSON format, each
> +one has a different justification schema which shares some fields.
> +
> +Here is an example to add a new justification in safe.json::
> +
> +|{
> +| "version": "1.0",
> +| "content": [
> +| {
> +| "id": "SAF-0-safe",
> +| "analyser": {
> +| "coverity": "misra_c_2012_rule_20_7_violation",
> +| "eclair": "MC3R1.R20.7"
> +| },
> +| "name": "R20.7 C macro parameters not used as expression",
> +| "text": "The macro parameters used in this [...]"
> +| },
> +| {
> +| "id": "SAF-1-safe",
> +| "analyser": {},
> +| "name": "Sentinel",
> +| "text": "Next ID to be used"
> +| }
> +| ]
> +|}
> +
> +To document a finding in safe.json, just add another block {[...]} before the
> +sentinel block, using the id contained in the sentinel block and increment by
> +one the number contained in the id of the sentinel block.
> +
> +Here is an explanation of the fields inside an object of the "content" array:
> + - id: it is a unique string that is used to refer to the finding, many finding
> + can be tagged with the same id, if the justification holds for any applied
> + case.
> + It tells the tool to substitute a Xen in-code comment having this structure:
> + /* SAF-0-safe [...] \*/
> + - analyser: it is an object containing pair of key-value strings, the key is
> + the analyser, so it can be coverity or eclair, the value is the proprietary
> + id corresponding on the finding, for example when coverity is used as
> + analyser, the tool will translate the Xen in-code coment in this way:
> + /* SAF-0-safe [...] \*/ -> /* coverity[misra_c_2012_rule_20_7_violation] \*/
> + if the object doesn't have a key-value, then the corresponding in-code
> + comment won't be translated.
> + - name: a simple name for the finding
> + - text: a proper justification to turn off the finding.
> +
> +
> +Here is an example to add a new justification in false-positive-<tool>.json::
> +
> +|{
> +| "version": "1.0",
> +| "content": [
> +| {
> +| "id": "SAF-0-false-positive-<tool>",
> +| "violation-id": "<proprietary-id>",
> +| "tool-version": "<version>",
> +| "name": "R20.7 [...]",
> +| "text": "[...]"
> +| },
> +| {
> +| "id": "SAF-1-false-positive-<tool>",
> +| "violation-id": "",
> +| "tool-version": "",
> +| "name": "Sentinel",
> +| "text": "Next ID to be used"
> +| }
> +| ]
> +|}
> +
> +To document a finding in false-positive-<tool>.json, just add another block
> +{[...]} before the sentinel block, using the id contained in the sentinel block
> +and increment by one the number contained in the id of the sentinel block.
> +
> +Here is an explanation of the fields inside an object of the "content" array:
> + - id: it has the same meaning as in the "safe" justification schema.
> + It tells the tool to substitute a Xen in-code comment having this structure:
> + /* SAF-0-false-positive-<tool> [...] \*/
> + - violation-id: its value is a string containing the proprietary id
> + corresponding to the finding, for example when <tool> is coverity, the Xen
> + tool will translate the Xen in-code coment in this way:
> + /* SAF-0-false-positive-coverity [...] \*/ -> /* coverity[misra_c_2012_rule_20_7_violation] \*/
> + if the object doesn't have a value, then the corresponding in-code comment
> + won't be translated.
> + - tool-version: the version of the tool affected by the false positive, if it
> + is discovered in more than one version, this string can be a range
> + (eg. 2.7 - 3.0)
> + - name, text: they have the same meaning as in the "safe" justification schema.
> +
> +
> +Justification example
> +---------------------
> +
> +Here an example of the usage of the in-code comment tags to suppress a finding
> +for the Rule 8.6:
> +
> +Eclair reports it in its web report, file xen/include/xen/kernel.h, line 68:
> +
> +| MC3R1.R8.6 for program 'xen/xen-syms', variable '_start' has no definition
> +
> +Also coverity reports it, here is an extract of the finding:
> +
> +| xen/include/xen/kernel.h:68:
> +| 1. misra_c_2012_rule_8_6_violation: Function "_start" is declared but never
> + defined.
> +
> +The analysers are complaining because we have this in xen/include/xen/kernel.h
> +at line 68::
> +
> +| extern char _start[], _end[], start[];
> +
> +Those are symbols exported by the linker, hence we will need to have a proper
> +deviation for this finding.
> +
> +We will prepare our entry in the safe.json database::
> +
> +|{
> +| "version": "1.0",
> +| "content": [
> +| {
> +| [...]
> +| },
> +| {
> +| "id": "SAF-1-safe",
> +| "analyser": {
> +| "eclair": "MC3R1.R8.6",
> +| "coverity": "misra_c_2012_rule_8_6_violation"
> +| },
> +| "name": "Rule 8.6: linker script defined symbols",
> +| "text": "It is safe to declare this symbol because it is defined in the linker script."
> +| },
> +| {
> +| "id": "SAF-2-safe",
> +| "analyser": {},
> +| "name": "Sentinel",
> +| "text": "Next ID to be used"
> +| }
> +| ]
> +|}
> +
> +And we will use the proper tag above the violation line::
> +
> +| /* SAF-1-safe R8.6 linker defined symbols */
> +| extern char _start[], _end[], start[];
> +
> +This entry will fix also the violation on _end and start, because they are on
> +the same line and the same "violation ID".
> +
> +Also, the same tag can be used on other symbols from the linker that are
> +declared in the codebase, because the justification holds for them too.
> diff --git a/docs/misra/false-positive-coverity.json b/docs/misra/false-positive-coverity.json
> new file mode 100644
> index 000000000000..462448414f80
> --- /dev/null
> +++ b/docs/misra/false-positive-coverity.json
> @@ -0,0 +1,12 @@
> +{
> + "version": "1.0",
> + "content": [
> + {
> + "id": "SAF-0-false-positive-coverity",
> + "violation-id": "",
> + "tool-version": "",
> + "name": "Sentinel",
> + "text": "Next ID to be used"
> + }
> + ]
> +}
> diff --git a/docs/misra/false-positive-eclair.json b/docs/misra/false-positive-eclair.json
> new file mode 100644
> index 000000000000..1d6ea5d7f045
> --- /dev/null
> +++ b/docs/misra/false-positive-eclair.json
> @@ -0,0 +1,12 @@
> +{
> + "version": "1.0",
> + "content": [
> + {
> + "id": "SAF-0-false-positive-eclair",
> + "violation-id": "",
> + "tool-version": "",
> + "name": "Sentinel",
> + "text": "Next ID to be used"
> + }
> + ]
> +}
> diff --git a/docs/misra/safe.json b/docs/misra/safe.json
> new file mode 100644
> index 000000000000..e079d3038120
> --- /dev/null
> +++ b/docs/misra/safe.json
> @@ -0,0 +1,11 @@
> +{
> + "version": "1.0",
> + "content": [
> + {
> + "id": "SAF-0-safe",
> + "analyser": {},
> + "name": "Sentinel",
> + "text": "Next ID to be used"
> + }
> + ]
> +}
> diff --git a/docs/misra/xen-static-analysis.rst b/docs/misra/xen-static-analysis.rst
> new file mode 100644
> index 000000000000..5b886474d4a0
> --- /dev/null
> +++ b/docs/misra/xen-static-analysis.rst
> @@ -0,0 +1,54 @@
> +.. SPDX-License-Identifier: CC-BY-4.0
> +
> +Xen static analysis
> +===================
> +
> +The Xen codebase integrates some scripts and tools that helps the developer to
> +perform static analysis of the code, currently Xen supports three analysis tool
> +that are eclair, coverity and cppcheck.
> +The Xen tree has a script (xen-analysis.py) available to ease the analysis
> +process and it integrates a way to suppress findings on these tools (only Eclair
> +and Coverity are currently supported by the script), please check the
> +documenting-violation.rst document to know more about it.
> +
> +Analyse Xen with Coverity or Eclair
> +-----------------------------------
> +
> +The xen-analysis.py script has two arguments to select which tool is used for
> +the analysis:
> +
> + - xen-analysis.py --run-coverity -- [optional make arguments]
> + - xen-analysis.py --run-eclair -- [optional make arguments]
> +
> +For example when using Coverity to analyse a Xen build obtained by passing these
> +arguments to the make system: XEN_TARGET_ARCH=arm64
> +CROSS_COMPILE=aarch64-linux-gnu-, the optional make arguments passed to
> +xen-analysis.py must be the same and the command below should be passed to
> +Coverity in its build phase:
> +
> + - xen-analysis.py --run-coverity -- XEN_TARGET_ARCH=arm64
> + CROSS_COMPILE=aarch64-linux-gnu-
> +
> +Which tells to the script to prepare the codebase for an analysis by Coverity
> +and forwards the make arguments to the make build invocation.
> +
> +When invoking the script, the procedure below will be followed:
> +
> + 1. Find which files among \*.c and \*.h has any in-code comment as
> + /* SAF-X-[...] \*/, the meaning of these comments is explained in
> + documenting-violation.rst.
> + Save the files obtained as <file>.safparse and generate <file> files where
> + the special in-code comments above are substituted with the proprietary
> + in-code comment used by the selected analysis tool. The safe.json and
> + false-positive-<tool>.json text file database are used to link each Xen tag
> + to the right proprietary in-code comment.
> + 2. Now Xen compilation starts using every <additional make parameters> supplied
> + at the script invocation. Coverity and Eclair are capable of intercepting
> + the compiler running from make to perform their analysis without
> + instrumenting the makefile.
> + 3. As final step every <file>.safparse file are reverted back as <file> and
> + every artifact related to the analysis will be cleaned.
> + This step is performed even in case any of the previous step fail, to skip
> + this step, call the script adding the --no-clean argument, but before
> + running again the script, call it with the --clean-only argument, that will
> + execute only this cleaning step.
> diff --git a/xen/scripts/xen-analysis.py b/xen/scripts/xen-analysis.py
> new file mode 100755
> index 000000000000..b5d9ef1862c9
> --- /dev/null
> +++ b/xen/scripts/xen-analysis.py
> @@ -0,0 +1,31 @@
> +#!/usr/bin/env python3
> +
> +import sys
> +from xen_analysis import settings, generic_analysis
> +from xen_analysis.generic_analysis import *
> +
> +
> +def main(argv):
> + ret_code = 0
> + settings.parse_commandline(argv)
> + try:
> + if settings.step_parse_tags:
> + generic_analysis.parse_xen_tags()
> + if settings.step_build_xen:
> + generic_analysis.build_xen()
> + except (ParseTagPhaseError, BuildPhaseError) as e:
> + print("ERROR: {}".format(e))
> + if hasattr(e, "errorcode"):
> + ret_code = e.errorcode
> + finally:
> + if settings.step_clean_analysis:
> + e = generic_analysis.clean_analysis_artifacts()
> + if e:
> + print("ERROR: {}".format(e))
> + ret_code = 1
> +
> + sys.exit(ret_code)
> +
> +
> +if __name__ == "__main__":
> + main(sys.argv[1:])
> diff --git a/xen/scripts/xen_analysis/__init__.py b/xen/scripts/xen_analysis/__init__.py
> new file mode 100644
> index 000000000000..e69de29bb2d1
> diff --git a/xen/scripts/xen_analysis/generic_analysis.py b/xen/scripts/xen_analysis/generic_analysis.py
> new file mode 100644
> index 000000000000..0b470c4ecf7d
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/generic_analysis.py
> @@ -0,0 +1,93 @@
> +#!/usr/bin/env python3
> +
> +import os, subprocess
> +from . import settings, utils, tag_database
> +
> +class ParseTagPhaseError(Exception):
> + pass
> +
> +class BuildPhaseError(Exception):
> + pass
> +
> +class CleanPhaseError(Exception):
> + pass
> +
> +
> +def parse_xen_tags():
> + # Load the database for the Xen tags
> + subs_list = tag_database.load_tag_database(
> + settings.analysis_tool,
> + [settings.repo_dir + "/docs/misra/safe.json"]
> + )
> + subs_list = tag_database.load_tag_database(
> + settings.analysis_tool,
> + [settings.repo_dir + "/docs/misra/false-positive-{}.json"
> + .format(settings.analysis_tool)],
> + subs_list,
> + "false-positive"
> + )
> +
> + # Create outdir if it doesn't exists
> + os.makedirs(settings.outdir, exist_ok=True)
> +
> + # The following lambda function will return a file if it contains lines with
> + # a comment containing "SAF-<number>-{safe|false-positive-<tool>}" on a
> + # single line.
> + grep_action = lambda x: utils.grep(x,
> + tag_database.get_xen_tag_comment_regex(
> + settings.analysis_tool)
> + )
> + # Look for a list of .h/.c files that matches the condition above
> + parse_file_list = utils.recursive_find_file(settings.xen_dir, r'.*\.[ch]$',
> + grep_action)
> +
> + for entry in parse_file_list:
> + file = entry["file"]
> + bkp_file = file + ".safparse"
> + if os.path.isfile(bkp_file):
> + raise ParseTagPhaseError(
> + "Found {}, please check the integrity of {}"
> + .format(bkp_file,file)
> + )
> + os.rename(file, bkp_file)
> + time_bkp_file = os.stat(bkp_file)
> + # Create <file> from <file>.safparse but with the Xen tag parsed
> + tag_database.substitute_tags(settings.analysis_tool, bkp_file, entry,
> + subs_list)
> + # Set timestamp for file equal to bkp_file, so that if the file is
> + # modified during the process by the user, we can catch it
> + os.utime(file, (time_bkp_file.st_atime, time_bkp_file.st_mtime))
> +
> +
> +def build_xen():
> + try:
> + subprocess.run(
> + "make -C {} {} build"
> + .format(settings.xen_dir, settings.make_forward_args),
> + shell=True, check=True
> + )
> + except (subprocess.CalledProcessError, subprocess.SubprocessError) as e:
> + excp = BuildPhaseError(
> + "Build error occured when running:\n{}".format(e.cmd)
> + )
> + excp.errorcode = e.returncode if hasattr(e, 'returncode') else 1
> + raise excp
> +
> +
> +def clean_analysis_artifacts():
> + safparse_files = utils.recursive_find_file(settings.xen_dir,
> + r'.*.safparse$')
> + for original_file in safparse_files:
> + # This commands strips the .safparse extension, leaving <file>
> + parsed_file_path = os.path.splitext(original_file)[0]
> + mtime_original_file = os.stat(original_file).st_mtime
> + mtime_parsed_file = os.stat(parsed_file_path).st_mtime
> + if mtime_original_file != mtime_parsed_file:
> + return CleanPhaseError(
> + "The file {} was modified during the analysis "
> + "procedure, it is impossible now to restore from the "
> + "content of {}, please handle it manually"
> + .format(parsed_file_path, original_file)
> + )
> + # Replace <file>.safparse to <file>
> + os.replace(original_file, parsed_file_path)
> diff --git a/xen/scripts/xen_analysis/settings.py b/xen/scripts/xen_analysis/settings.py
> new file mode 100644
> index 000000000000..947dfa2d50af
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/settings.py
> @@ -0,0 +1,97 @@
> +#!/usr/bin/env python3
> +
> +import sys, re, os
> +
> +module_dir = os.path.dirname(os.path.realpath(__file__))
> +xen_dir = os.path.realpath(module_dir + "/../..")
> +repo_dir = os.path.realpath(xen_dir + "/..")
> +tools_dir = os.path.realpath(xen_dir + "/tools")
> +
> +step_parse_tags = True
> +step_build_xen = True
> +step_clean_analysis = True
> +
> +target_build = False
> +target_clean = False
> +
> +analysis_tool = ""
> +make_forward_args = ""
> +outdir = xen_dir
> +
> +
> +def help():
> + msg="""
> +Usage: {} [OPTION] ... [-- [make arguments]]
> +
> +This script runs the analysis on the Xen codebase.
> +
> +Options:
> + --build-only Run only the commands to build Xen with the optional make
> + arguments passed to the script
> + --clean-only Run only the commands to clean the analysis artifacts
> + -h, --help Print this help
> + --no-build Skip the build Xen phase
> + --no-clean Don\'t clean the analysis artifacts on exit
> + --run-coverity Run the analysis for the Coverity tool
> + --run-eclair Run the analysis for the Eclair tool
> +"""
> + print(msg.format(sys.argv[0]))
> +
> +
> +def parse_commandline(argv):
> + global analysis_tool
> + global make_forward_args
> + global outdir
> + global step_parse_tags
> + global step_build_xen
> + global step_clean_analysis
> + global target_build
> + global target_clean
> + forward_to_make = False
> + for option in argv:
> + if forward_to_make:
> + # Intercept outdir
> + outdir_regex = re.match("^O=(.*)$", option)
> + if outdir_regex:
> + outdir = outdir_regex.group(1)
> + # Forward any make arguments
> + make_forward_args = make_forward_args + " " + option
> + elif option == "--build-only":
> + target_build = True
> + elif option == "--clean-only":
> + target_clean = True
> + elif (option == "--help") or (option == "-h"):
> + help()
> + sys.exit(0)
> + elif option == "--no-build":
> + step_build_xen = False
> + elif option == "--no-clean":
> + step_clean_analysis = False
> + elif (option == "--run-coverity") or (option == "--run-eclair"):
> + analysis_tool = option[6:]
> + elif option == "--":
> + forward_to_make = True
> + else:
> + print("Invalid option: {}".format(option))
> + help()
> + sys.exit(1)
> +
> + if target_build and target_clean:
> + print("--build-only is not compatible with --clean-only argument.")
> + sys.exit(1)
> +
> + if target_clean:
> + step_parse_tags = False
> + step_build_xen = False
> + step_clean_analysis = True
> + return
> +
> + if analysis_tool == "":
> + print("Please specify one analysis tool.")
> + help()
> + sys.exit(1)
> +
> + if target_build:
> + step_parse_tags = False
> + step_build_xen = True
> + step_clean_analysis = False
> diff --git a/xen/scripts/xen_analysis/tag_database.py b/xen/scripts/xen_analysis/tag_database.py
> new file mode 100644
> index 000000000000..ca374bbb62dd
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/tag_database.py
> @@ -0,0 +1,109 @@
> +#!/usr/bin/env python3
> +
> +import re, json
> +
> +class TagDatabaseError(Exception):
> + pass
> +
> +# This is the dictionary for the rules that translates to proprietary comments:
> +# - cppcheck: /* cppcheck-suppress[id] */
> +# - coverity: /* coverity[id] */
> +# - eclair: /* -E> hide id 1 "" */
> +# Add entries to support more analyzers
> +tool_syntax = {
> + "cppcheck":"cppcheck-suppress[VID]",
> + "coverity":"coverity[VID]",
> + "eclair":"-E> hide VID 1 \"\""
> +}
> +
> +
> +def get_xen_tag_index_type_regex(tool):
> + return r'^SAF-(\d+)-(safe|false-positive-' + tool + ')$'
> +
> +
> +def get_xen_tag_comment_regex(tool):
> + return r'^[ \t]*/\* +(SAF-\d+-(?:safe|false-positive-' + tool + ')).*\*/$'
> +
> +
> +# Returns a data structure containing dictionaries for safe and false-positive-*
> +# Xen tags, the key is the unique index of the tag and the content is the
> +# proprietary in-code comment to be used when the tag is found in the codebase
> +def load_tag_database(tool, input_files, data_struct = None, schema = "safe"):
> + ret = data_struct if data_struct is not None else {
> + "safe": {},
> + "false-positive-" + tool: {}
> + }
> + database = []
> +
> + # Open all input files
> + for file in input_files:
> + try:
> + with open(file, "rt") as handle:
> + content = json.load(handle)
> + database = database + content['content']
> + except json.JSONDecodeError as e:
> + raise TagDatabaseError("JSON decoding error in file {}: {}"
> + .format(file, e))
> + except Exception as e:
> + raise TagDatabaseError("Can't open file {}: {}"
> + .format(file, e))
> +
> + for entry in database:
> + # If the false-positive schema is used, check the proprietary id in the
> + # 'violation-id' field, otherwise rely on the "safe" schema.
> + if schema == "false-positive":
> + proprietary_id = entry['violation-id']
> + elif tool in entry['analyser']:
> + proprietary_id = entry['analyser'][tool]
> + else:
> + proprietary_id = ""
> + if proprietary_id != "":
> + comment=tool_syntax[tool].replace("VID",proprietary_id)
> + # Regex to capture the index of the Xen tag and the schema
> + xen_tag = re.compile(get_xen_tag_index_type_regex(tool))\
> + .match(entry["id"])
> + if xen_tag and xen_tag.group(1) and xen_tag.group(2):
> + # Save in safe or false-positive-* the key {#id: "comment"}
> + id_number = int(xen_tag.group(1))
> + key = xen_tag.group(2)
> + ret[key][id_number] = "/* {} */\n".format(comment)
> + else:
> + raise TagDatabaseError(
> + "Error in database file, entry {} has unexpected "
> + "format.".format(entry["id"])
> + )
> +
> + return ret
> +
> +
> +def substitute_tags(tool, input_file, grep_struct, subs_rules):
> + try:
> + with open(grep_struct["file"], "wt") as outfile:
> +
> + try:
> + with open(input_file, "rt") as infile:
> + parsed_content = infile.readlines()
> + except Exception as e:
> + raise TagDatabaseError("Issue with reading file {}: {}"
> + .format(input_file, e))
> +
> + # grep_struct contains the line number where the comments are, the
> + # line number starts from 1 but in the array the first line is zero.
> + # For every line where there is a Xen tag comment, get the Xen tag
> + # that is in the capture group zero, extract from the Xen tag the
> + # unique index and the type (safe, false-positive-*) and with those
> + # information access the subs_rules dictionary to see if there is
> + # a match
> + for line_number in grep_struct["matches"]:
> + xen_tag = grep_struct["matches"][line_number][0]
> + xen_tag_regex_obj = re.compile(
> + get_xen_tag_index_type_regex(tool)).match(xen_tag)
> + id_number = int(xen_tag_regex_obj.group(1))
> + key = xen_tag_regex_obj.group(2)
> + if id_number in subs_rules[key]:
> + parsed_content[line_number-1] = subs_rules[key][id_number]
> +
> + outfile.writelines(parsed_content)
> + except Exception as e:
> + raise TagDatabaseError("Issue with writing file {}: {}"
> + .format(grep_struct["file"], e))
> diff --git a/xen/scripts/xen_analysis/utils.py b/xen/scripts/xen_analysis/utils.py
> new file mode 100644
> index 000000000000..a912d812c3df
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/utils.py
> @@ -0,0 +1,37 @@
> +#!/usr/bin/env python3
> +
> +import os, re
> +
> +
> +def grep(filepath, regex):
> + regObj = re.compile(regex)
> + res = { "file": filepath, "matches": {} }
> + try:
> + with open(filepath, "rt") as f:
> + line_number = 1
> + for line in f:
> + match = regObj.match(line)
> + if match:
> + res["matches"][line_number] = match.groups()
> + line_number = line_number + 1
> + except Exception as e:
> + print("WARNING: Can't open {}: {}".format(filepath, e))
> +
> + # Return filename and line matches if there are
> + return res if res["matches"] else {}
> +
> +
> +def recursive_find_file(path, filename_regex, action = None):
> + filename_reg_obj = re.compile(filename_regex)
> + res = []
> + for root, dirs, fnames in os.walk(path):
> + for fname in fnames:
> + if filename_reg_obj.match(fname):
> + if action is None:
> + res.append(os.path.join(root, fname))
> + else:
> + out = action(os.path.join(root, fname))
> + if out:
> + res.append(out)
> +
> + return res
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 4/4] xen: Justify linker script defined symbols in include/xen/kernel.h
2022-11-28 14:10 ` [PATCH 4/4] xen: Justify linker script defined symbols in include/xen/kernel.h Luca Fancellu
2022-11-28 15:19 ` Jan Beulich
@ 2022-11-29 1:55 ` Stefano Stabellini
1 sibling, 0 replies; 31+ messages in thread
From: Stefano Stabellini @ 2022-11-29 1:55 UTC (permalink / raw)
To: Luca Fancellu
Cc: xen-devel, bertrand.marquis, wei.chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Stefano Stabellini,
Wei Liu
On Mon, 28 Nov 2022, Luca Fancellu wrote:
> Eclair and Coverity found violation of the MISRA rule 8.6 for the
> symbols _start, _end, start, _stext, _etext, _srodata, _erodata,
> _sinittext, _einittext which are declared in
> xen/include/xen/kernel.h.
> All those symbols are defined by the liker script so we can deviate
> from the rule 8.6 for these cases.
>
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> docs/misra/safe.json | 9 +++++++++
> xen/include/xen/kernel.h | 4 ++++
> 2 files changed, 13 insertions(+)
>
> diff --git a/docs/misra/safe.json b/docs/misra/safe.json
> index e079d3038120..e3c8a1d8eb36 100644
> --- a/docs/misra/safe.json
> +++ b/docs/misra/safe.json
> @@ -3,6 +3,15 @@
> "content": [
> {
> "id": "SAF-0-safe",
> + "analyser": {
> + "eclair": "MC3R1.R8.6",
> + "coverity": "misra_c_2012_rule_8_6_violation"
> + },
> + "name": "Rule 8.6: linker script defined symbols",
> + "text": "It is safe to declare this symbol because it is defined in the linker script."
> + },
> + {
> + "id": "SAF-1-safe",
> "analyser": {},
> "name": "Sentinel",
> "text": "Next ID to be used"
> diff --git a/xen/include/xen/kernel.h b/xen/include/xen/kernel.h
> index 8cd142032d3b..f1a7713784fc 100644
> --- a/xen/include/xen/kernel.h
> +++ b/xen/include/xen/kernel.h
> @@ -65,24 +65,28 @@
> 1; \
> })
>
> +/* SAF-0-safe */
> extern char _start[], _end[], start[];
> #define is_kernel(p) ({ \
> char *__p = (char *)(unsigned long)(p); \
> (__p >= _start) && (__p < _end); \
> })
>
> +/* SAF-0-safe */
> extern char _stext[], _etext[];
> #define is_kernel_text(p) ({ \
> char *__p = (char *)(unsigned long)(p); \
> (__p >= _stext) && (__p < _etext); \
> })
>
> +/* SAF-0-safe */
> extern const char _srodata[], _erodata[];
> #define is_kernel_rodata(p) ({ \
> const char *__p = (const char *)(unsigned long)(p); \
> (__p >= _srodata) && (__p < _erodata); \
> })
>
> +/* SAF-0-safe */
> extern char _sinittext[], _einittext[];
> #define is_kernel_inittext(p) ({ \
> char *__p = (char *)(unsigned long)(p); \
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 0/4] Static analyser finding deviation
2022-11-28 14:10 [PATCH 0/4] Static analyser finding deviation Luca Fancellu
` (3 preceding siblings ...)
2022-11-28 14:10 ` [PATCH 4/4] xen: Justify linker script defined symbols in include/xen/kernel.h Luca Fancellu
@ 2022-11-29 1:55 ` Stefano Stabellini
2022-11-29 9:46 ` Luca Fancellu
4 siblings, 1 reply; 31+ messages in thread
From: Stefano Stabellini @ 2022-11-29 1:55 UTC (permalink / raw)
To: Luca Fancellu
Cc: xen-devel, bertrand.marquis, wei.chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Stefano Stabellini,
Wei Liu
On Mon, 28 Nov 2022, Luca Fancellu wrote:
> This serie introduces a way to suppress a static analyser finding providing a
> proper justification for it.
> The process is explained in the docs/misra/documenting-violations.rst document
> that this serie will provide.
> The tools currently supported are eclair, coverity and cppcheck, but the design
> is open to support many other static analysis tool.
>
> The changes are split between the first two patches to reduce the review effort,
> the first patch is introducing the deviation process for the eclair and coverity
> tools, this is because their analysis system is similar.
>
> The second patch is introducing the same deviation process for cppcheck,
> modifying the current way it is called from the makefile and improving its
> analysis.
>
> The third patch is a fix for a tool used for cppcheck and the fourth patch
> is an example of how a deviation can be applied for some MISRA findings.
I tried testing this series with:
# scripts/xen-analysis.py --build-only --cppcheck-html --run-cppcheck --cppcheck-bin=/local/repos/cppcheck/cppcheck --cppcheck-html-bin=/local/repos/cppcheck/htmlreport/cppcheck-htmlreport
But I get this error:
ERROR: Can't find cppcheck version or version is not 2.7
Note that my cppcheck is 2.7.4:
# ./cppcheck --version
Cppcheck 2.7.4
After removing the version check in cppcheck_analysis.py, the process
starts correctly.
Also, where is the output html report created by cppcheck-html by
default?
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-28 15:37 ` Luca Fancellu
@ 2022-11-29 9:42 ` Jan Beulich
2022-11-29 9:49 ` Luca Fancellu
0 siblings, 1 reply; 31+ messages in thread
From: Jan Beulich @ 2022-11-29 9:42 UTC (permalink / raw)
To: Luca Fancellu
Cc: Bertrand Marquis, Wei Chen, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Wei Liu, xen-devel
On 28.11.2022 16:37, Luca Fancellu wrote:
>> On 28 Nov 2022, at 15:19, Jan Beulich <jbeulich@suse.com> wrote:
>> On 28.11.2022 15:10, Luca Fancellu wrote:
>>> Change cppcheck invocation method by using the xen-analysis.py
>>> script using the arguments --run-cppcheck.
>>>
>>> Now cppcheck analysis will build Xen while the analysis is performed
>>> on the source files, it will produce a text report and an additional
>>> html output when the script is called with --cppcheck-html.
>>>
>>> With this patch cppcheck will benefit of platform configuration files
>>> that will help it to understand the target of the compilation and
>>> improve the analysis.
>>>
>>> To do so:
>>> - remove cppcheck rules from Makefile and move them to the script.
>>> - Update xen-analysis.py with the code to integrate cppcheck.
>>> - merge the script merge_cppcheck_reports.py into the xen-analysis
>>> script package and rework the code to integrate it.
>>> - add platform configuration files for cppcheck..
>>> - add cppcheck-cc.sh script that is a wrapper for cppcheck and it's
>>> used as Xen compiler, it will intercept all flags given from the
>>> make build system and it will execute cppcheck on the compiled
>>> file together with the file compilation.
>>> - guarded hypercall-defs.c with CPPCHECK define because cppcheck
>>> gets confused as the file does not contain c code.
>>> - add false-positive-cppcheck.json file
>>> - update documentation.
>>> - update .gitignore
>>>
>>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>>
>> Just two and a half questions, not a full review:
>>
>>> ---
>>> .gitignore | 8 +-
>>> docs/misra/cppcheck.txt | 27 +-
>>> docs/misra/documenting-violations.rst | 7 +-
>>> docs/misra/false-positive-cppcheck.json | 12 +
>>> docs/misra/xen-static-analysis.rst | 42 ++-
>>> xen/Makefile | 116 +-------
>>> xen/include/hypercall-defs.c | 9 +
>>> xen/scripts/xen-analysis.py | 18 +-
>>> xen/scripts/xen_analysis/cppcheck_analysis.py | 272 ++++++++++++++++++
>>> .../xen_analysis/cppcheck_report_utils.py | 130 +++++++++
>>> xen/scripts/xen_analysis/generic_analysis.py | 21 +-
>>> xen/scripts/xen_analysis/settings.py | 77 ++++-
>>> xen/scripts/xen_analysis/utils.py | 21 +-
>>> xen/tools/cppcheck-cc.sh | 223 ++++++++++++++
>>> xen/tools/cppcheck-plat/arm32-wchar_t4.xml | 17 ++
>>> xen/tools/cppcheck-plat/arm64-wchar_t2.xml | 17 ++
>>> xen/tools/cppcheck-plat/arm64-wchar_t4.xml | 17 ++
>>> xen/tools/cppcheck-plat/x86_64-wchar_t2.xml | 17 ++
>>> xen/tools/cppcheck-plat/x86_64-wchar_t4.xml | 17 ++
>>
>> What are these last five files about? There's nothing about them in
>> the description afaics.
>
> They are cppcheck platform configuration files, they help cppcheck to understand
> the size of the types depending on the target of the compilation.
>
> This section in the commit message is to introduce them:
>
> With this patch cppcheck will benefit of platform configuration files
> that will help it to understand the target of the compilation and
> improve the analysis.
>
> Do you think I should say it differently? Or maybe say that they reside in xen/tools/cppcheck-plat/ ?
Perhaps (I didn't read that paragraph as relating to _anything_ in
tree), e.g.:
With this patch cppcheck will benefit from platform configuration files
that will help it to understand the target of the compilation and
improve the analysis. These are XML files placed in
xen/tools/cppcheck-plat/, describing ... (I don't know what to put here).
Please write the description here such that people not familiar with
cppcheck (or more generally with any external tool) can still follow
what you're talking about and what the patch is doing.
>>> --- /dev/null
>>> +++ b/xen/scripts/xen_analysis/cppcheck_analysis.py
>>> @@ -0,0 +1,272 @@
>>> +#!/usr/bin/env python3
>>> +
>>> +import os, re, shutil
>>> +from . import settings, utils, cppcheck_report_utils
>>> +
>>> +class GetMakeVarsPhaseError(Exception):
>>> + pass
>>> +
>>> +class CppcheckDepsPhaseError(Exception):
>>> + pass
>>> +
>>> +class CppcheckReportPhaseError(Exception):
>>> + pass
>>> +
>>> +CPPCHECK_BUILD_DIR = "build-dir-cppcheck"
>>> +CPPCHECK_HTMLREPORT_OUTDIR = "cppcheck-htmlreport"
>>> +CPPCHECK_REPORT_OUTDIR = "cppcheck-report"
>>> +cppcheck_extra_make_args = ""
>>> +xen_cc = ""
>>> +
>>> +def get_make_vars():
>>> + global xen_cc
>>> + invoke_make = utils.invoke_command(
>>> + "make -C {} {} export-variable-CC"
>>> + .format(settings.xen_dir, settings.make_forward_args),
>>> + True, GetMakeVarsPhaseError,
>>> + "Error occured retrieving make vars:\n{}"
>>> + )
>>> +
>>> + cc_var_regex = re.search('^CC=(.*)$', invoke_make, flags=re.M)
>>> + if cc_var_regex:
>>> + xen_cc = cc_var_regex.group(1)
>>> +
>>> + if xen_cc == "":
>>> + raise GetMakeVarsPhaseError("CC variable not found in Xen make output")
>>
>> What use is CC without CFLAGS? Once again the description could do
>> with containing some information on what's going on here, and why
>> you need to export any variables in the first place.
>
> We don’t need CFLAGS here, we need only CC to generate include/generated/compiler-def.h and
> to pass CC to the cppcheck-cc.sh --compiler argument.
Hmm, I see that include/generated/compiler-def.h is generated already
now without any use of CFLAGS. Which looks suspicious to me. Sadly
the uses in xen/Makefile are lacking any details on what this is for,
and Bertrand's commit introducing it doesn't explain its purpose
either. Maybe again something entirely obvious to people knowing
cppcheck sufficiently well ...
> Would a comment in the code be ok?
Not sure (yet).
Jan
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 0/4] Static analyser finding deviation
2022-11-29 1:55 ` [PATCH 0/4] Static analyser finding deviation Stefano Stabellini
@ 2022-11-29 9:46 ` Luca Fancellu
2022-11-29 13:02 ` Luca Fancellu
0 siblings, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-11-29 9:46 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Bertrand Marquis, Wei Chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Wei Liu
> On 29 Nov 2022, at 01:55, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Mon, 28 Nov 2022, Luca Fancellu wrote:
>> This serie introduces a way to suppress a static analyser finding providing a
>> proper justification for it.
>> The process is explained in the docs/misra/documenting-violations.rst document
>> that this serie will provide.
>> The tools currently supported are eclair, coverity and cppcheck, but the design
>> is open to support many other static analysis tool.
>>
>> The changes are split between the first two patches to reduce the review effort,
>> the first patch is introducing the deviation process for the eclair and coverity
>> tools, this is because their analysis system is similar.
>>
>> The second patch is introducing the same deviation process for cppcheck,
>> modifying the current way it is called from the makefile and improving its
>> analysis.
>>
>> The third patch is a fix for a tool used for cppcheck and the fourth patch
>> is an example of how a deviation can be applied for some MISRA findings.
Hi Stefano,
>
> I tried testing this series with:
>
> # scripts/xen-analysis.py --build-only --cppcheck-html --run-cppcheck --cppcheck-bin=/local/repos/cppcheck/cppcheck --cppcheck-html-bin=/local/repos/cppcheck/htmlreport/cppcheck-htmlreport
>
> But I get this error:
>
> ERROR: Can't find cppcheck version or version is not 2.7
>
>
> Note that my cppcheck is 2.7.4:
>
> # ./cppcheck --version
> Cppcheck 2.7.4
Yes this is a bug, I’m strictly checking for 2.7, I will modify it to 2.7.x if you agree
>
>
> After removing the version check in cppcheck_analysis.py, the process
> starts correctly.
>
> Also, where is the output html report created by cppcheck-html by
> default?
The html output should be in the xen folder [xen_repo]/xen/cppcheck-htmlreport/html but when you specify --build-only the reports are not generated, only the build phase is executed.
Have you tried without --build-only to test the report generations?
Cheers,
Luca
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-29 9:42 ` Jan Beulich
@ 2022-11-29 9:49 ` Luca Fancellu
0 siblings, 0 replies; 31+ messages in thread
From: Luca Fancellu @ 2022-11-29 9:49 UTC (permalink / raw)
To: Jan Beulich
Cc: Bertrand Marquis, Wei Chen, Andrew Cooper, George Dunlap,
Julien Grall, Stefano Stabellini, Wei Liu, xen-devel
> On 29 Nov 2022, at 09:42, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 28.11.2022 16:37, Luca Fancellu wrote:
>>> On 28 Nov 2022, at 15:19, Jan Beulich <jbeulich@suse.com> wrote:
>>> On 28.11.2022 15:10, Luca Fancellu wrote:
>>>> Change cppcheck invocation method by using the xen-analysis.py
>>>> script using the arguments --run-cppcheck.
>>>>
>>>> Now cppcheck analysis will build Xen while the analysis is performed
>>>> on the source files, it will produce a text report and an additional
>>>> html output when the script is called with --cppcheck-html.
>>>>
>>>> With this patch cppcheck will benefit of platform configuration files
>>>> that will help it to understand the target of the compilation and
>>>> improve the analysis.
>>>>
>>>> To do so:
>>>> - remove cppcheck rules from Makefile and move them to the script.
>>>> - Update xen-analysis.py with the code to integrate cppcheck.
>>>> - merge the script merge_cppcheck_reports.py into the xen-analysis
>>>> script package and rework the code to integrate it.
>>>> - add platform configuration files for cppcheck..
>>>> - add cppcheck-cc.sh script that is a wrapper for cppcheck and it's
>>>> used as Xen compiler, it will intercept all flags given from the
>>>> make build system and it will execute cppcheck on the compiled
>>>> file together with the file compilation.
>>>> - guarded hypercall-defs.c with CPPCHECK define because cppcheck
>>>> gets confused as the file does not contain c code.
>>>> - add false-positive-cppcheck.json file
>>>> - update documentation.
>>>> - update .gitignore
>>>>
>>>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>>>
>>> Just two and a half questions, not a full review:
>>>
>>>> ---
>>>> .gitignore | 8 +-
>>>> docs/misra/cppcheck.txt | 27 +-
>>>> docs/misra/documenting-violations.rst | 7 +-
>>>> docs/misra/false-positive-cppcheck.json | 12 +
>>>> docs/misra/xen-static-analysis.rst | 42 ++-
>>>> xen/Makefile | 116 +-------
>>>> xen/include/hypercall-defs.c | 9 +
>>>> xen/scripts/xen-analysis.py | 18 +-
>>>> xen/scripts/xen_analysis/cppcheck_analysis.py | 272 ++++++++++++++++++
>>>> .../xen_analysis/cppcheck_report_utils.py | 130 +++++++++
>>>> xen/scripts/xen_analysis/generic_analysis.py | 21 +-
>>>> xen/scripts/xen_analysis/settings.py | 77 ++++-
>>>> xen/scripts/xen_analysis/utils.py | 21 +-
>>>> xen/tools/cppcheck-cc.sh | 223 ++++++++++++++
>>>> xen/tools/cppcheck-plat/arm32-wchar_t4.xml | 17 ++
>>>> xen/tools/cppcheck-plat/arm64-wchar_t2.xml | 17 ++
>>>> xen/tools/cppcheck-plat/arm64-wchar_t4.xml | 17 ++
>>>> xen/tools/cppcheck-plat/x86_64-wchar_t2.xml | 17 ++
>>>> xen/tools/cppcheck-plat/x86_64-wchar_t4.xml | 17 ++
>>>
>>> What are these last five files about? There's nothing about them in
>>> the description afaics.
>>
>> They are cppcheck platform configuration files, they help cppcheck to understand
>> the size of the types depending on the target of the compilation.
>>
>> This section in the commit message is to introduce them:
>>
>> With this patch cppcheck will benefit of platform configuration files
>> that will help it to understand the target of the compilation and
>> improve the analysis.
>>
>> Do you think I should say it differently? Or maybe say that they reside in xen/tools/cppcheck-plat/ ?
>
> Perhaps (I didn't read that paragraph as relating to _anything_ in
> tree), e.g.:
>
> With this patch cppcheck will benefit from platform configuration files
> that will help it to understand the target of the compilation and
> improve the analysis. These are XML files placed in
> xen/tools/cppcheck-plat/, describing ... (I don't know what to put here).
>
> Please write the description here such that people not familiar with
> cppcheck (or more generally with any external tool) can still follow
> what you're talking about and what the patch is doing.
Ok I can modify the description to add more details
>
>>>> --- /dev/null
>>>> +++ b/xen/scripts/xen_analysis/cppcheck_analysis.py
>>>> @@ -0,0 +1,272 @@
>>>> +#!/usr/bin/env python3
>>>> +
>>>> +import os, re, shutil
>>>> +from . import settings, utils, cppcheck_report_utils
>>>> +
>>>> +class GetMakeVarsPhaseError(Exception):
>>>> + pass
>>>> +
>>>> +class CppcheckDepsPhaseError(Exception):
>>>> + pass
>>>> +
>>>> +class CppcheckReportPhaseError(Exception):
>>>> + pass
>>>> +
>>>> +CPPCHECK_BUILD_DIR = "build-dir-cppcheck"
>>>> +CPPCHECK_HTMLREPORT_OUTDIR = "cppcheck-htmlreport"
>>>> +CPPCHECK_REPORT_OUTDIR = "cppcheck-report"
>>>> +cppcheck_extra_make_args = ""
>>>> +xen_cc = ""
>>>> +
>>>> +def get_make_vars():
>>>> + global xen_cc
>>>> + invoke_make = utils.invoke_command(
>>>> + "make -C {} {} export-variable-CC"
>>>> + .format(settings.xen_dir, settings.make_forward_args),
>>>> + True, GetMakeVarsPhaseError,
>>>> + "Error occured retrieving make vars:\n{}"
>>>> + )
>>>> +
>>>> + cc_var_regex = re.search('^CC=(.*)$', invoke_make, flags=re.M)
>>>> + if cc_var_regex:
>>>> + xen_cc = cc_var_regex.group(1)
>>>> +
>>>> + if xen_cc == "":
>>>> + raise GetMakeVarsPhaseError("CC variable not found in Xen make output")
>>>
>>> What use is CC without CFLAGS? Once again the description could do
>>> with containing some information on what's going on here, and why
>>> you need to export any variables in the first place.
>>
>> We don’t need CFLAGS here, we need only CC to generate include/generated/compiler-def.h and
>> to pass CC to the cppcheck-cc.sh --compiler argument.
>
> Hmm, I see that include/generated/compiler-def.h is generated already
> now without any use of CFLAGS. Which looks suspicious to me. Sadly
> the uses in xen/Makefile are lacking any details on what this is for,
> and Bertrand's commit introducing it doesn't explain its purpose
> either. Maybe again something entirely obvious to people knowing
> cppcheck sufficiently well ...
>
>> Would a comment in the code be ok?
>
> Not sure (yet).
>
> Jan
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 0/4] Static analyser finding deviation
2022-11-29 9:46 ` Luca Fancellu
@ 2022-11-29 13:02 ` Luca Fancellu
0 siblings, 0 replies; 31+ messages in thread
From: Luca Fancellu @ 2022-11-29 13:02 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Bertrand Marquis, Wei Chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Wei Liu
> On 29 Nov 2022, at 09:46, Luca Fancellu <Luca.Fancellu@arm.com> wrote:
>
>
>
>> On 29 Nov 2022, at 01:55, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>
>> On Mon, 28 Nov 2022, Luca Fancellu wrote:
>>> This serie introduces a way to suppress a static analyser finding providing a
>>> proper justification for it.
>>> The process is explained in the docs/misra/documenting-violations.rst document
>>> that this serie will provide.
>>> The tools currently supported are eclair, coverity and cppcheck, but the design
>>> is open to support many other static analysis tool.
>>>
>>> The changes are split between the first two patches to reduce the review effort,
>>> the first patch is introducing the deviation process for the eclair and coverity
>>> tools, this is because their analysis system is similar.
>>>
>>> The second patch is introducing the same deviation process for cppcheck,
>>> modifying the current way it is called from the makefile and improving its
>>> analysis.
>>>
>>> The third patch is a fix for a tool used for cppcheck and the fourth patch
>>> is an example of how a deviation can be applied for some MISRA findings.
>
> Hi Stefano,
>
>>
>> I tried testing this series with:
>>
>> # scripts/xen-analysis.py --build-only --cppcheck-html --run-cppcheck --cppcheck-bin=/local/repos/cppcheck/cppcheck --cppcheck-html-bin=/local/repos/cppcheck/htmlreport/cppcheck-htmlreport
>>
>> But I get this error:
>>
>> ERROR: Can't find cppcheck version or version is not 2.7
>>
>>
>> Note that my cppcheck is 2.7.4:
>>
>> # ./cppcheck --version
>> Cppcheck 2.7.4
>
> Yes this is a bug, I’m strictly checking for 2.7, I will modify it to 2.7.x if you agree
>
>>
>>
>> After removing the version check in cppcheck_analysis.py, the process
>> starts correctly.
>>
>> Also, where is the output html report created by cppcheck-html by
>> default?
>
>
> The html output should be in the xen folder [xen_repo]/xen/cppcheck-htmlreport/html but when you specify --build-only the reports are not generated, only the build phase is executed.
>
> Have you tried without --build-only to test the report generations?
However I’ve found another bug, when building using your command line (at least on my x86 machine)
I have that xen is not building and it’s ending with this:
ld -melf_x86_64 -T arch/x86/xen.lds -N prelink.o --build-id=sha1 \
./common/symbols-dummy.o -o ./.xen-syms.0
nm -pa --format=sysv ./.xen-syms.0 \
| ./tools/symbols --all-symbols --sort-by-name --sysv --sort \
>./.xen-syms.0.S
make -f ./Rules.mk obj=. ./.xen-syms.0.o
CC .xen-syms.0.o
ld -melf_x86_64 -T arch/x86/xen.lds -N prelink.o --build-id=sha1 \
./.xen-syms.0.o -o ./.xen-syms.1
nm -pa --format=sysv ./.xen-syms.1 \
| ./tools/symbols --all-symbols --sort-by-name --sysv --sort --error-dup \
>./.xen-syms.1.S
make -f ./Rules.mk obj=. ./.xen-syms.1.o
CC .xen-syms.1.o
ld -melf_x86_64 -T arch/x86/xen.lds -N prelink.o --build-id=sha1 \
--orphan-handling=warn ./.xen-syms.1.o -o xen-syms
nm -pa --format=sysv ./xen-syms \
| ./tools/symbols --all-symbols --xensyms --sysv --sort \
>./xen-syms.map
rm -f ./.xen-syms.[0-9]* ./..xen-syms.[0-9]*
HOSTCC arch/x86/efi/mkreloc
Checking arch/x86/efi/mkreloc.c ...
Checking arch/x86/efi/mkreloc.c: CPPCHECK=1;...
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
nm: 'arch/x86/efi/relocs-dummy.o': No such file
echo "Will strip debug info from xen.efi"
Will strip debug info from xen.efi
ld -mi386pep --subsystem=10 --strip-debug --image-base=0x --stack=0,0 --heap=0,0 --section-alignment=0x200000 --file-alignment=0x20 --major-image-version=4 --minor-image-version=17 --major-os-version=2 --minor-os-version=0 --major-subsystem-version=2 --minor-subsystem-version=0 --build-id=sha1 -T arch/x86/efi.lds -N prelink.o arch/x86/efi/relocs-dummy.o ./common/symbols-dummy.o -b pe-x86-64 arch/x86/efi/buildid.o -o ./.xen.efi.0x.0 && ld -mi386pep --subsystem=10 --strip-debug --image-base=0x --stack=0,0 --heap=0,0 --section-alignment=0x200000 --file-alignment=0x20 --major-image-version=4 --minor-image-version=17 --major-os-version=2 --minor-os-version=0 --major-subsystem-version=2 --minor-subsystem-version=0 --build-id=sha1 -T arch/x86/efi.lds -N prelink.o arch/x86/efi/relocs-dummy.o ./common/symbols-dummy.o -b pe-x86-64 arch/x86/efi/buildid.o -o ./.xen.efi.0x.0 && :
ld: cannot find arch/x86/efi/relocs-dummy.o: No such file or directory
ld: cannot find arch/x86/efi/buildid.o: No such file or directory
arch/x86/Makefile:207: recipe for target 'xen.efi' failed
make[2]: *** [xen.efi] Error 1
build.mk:90: recipe for target 'xen' failed
make[1]: *** [xen] Error 2
Makefile:585: recipe for target 'xen' failed
make: *** [xen] Error 2
make: Leaving directory '/data_sdc1/lucfan01/kirkstone_xen/xen/xen'
ERROR: Build error occured when running:
make -C /data_sdc1/lucfan01/kirkstone_xen/xen/xen CC="/data_sdc1/lucfan01/kirkstone_xen/xen/xen/tools/cppcheck-cc.sh --compiler=gcc --cppcheck-cmd=cppcheck --cppcheck-build-dir=/data_sdc1/lucfan01/kirkstone_xen/xen/xen/build-dir-cppcheck --max-ctu-depth=10 --enable=style,information,missingInclude --template='{file}({line},{column}):{id}:{severity}:{message}' --relative-paths=/data_sdc1/lucfan01/kirkstone_xen/xen/xen --inline-suppr --suppressions-list=/data_sdc1/lucfan01/kirkstone_xen/xen/xen/suppression-list.txt --suppress='unmatchedSuppression:*generated/compiler-def.h' --include=/data_sdc1/lucfan01/kirkstone_xen/xen/xen/include/xen/config.h -DCPPCHECK --cppcheck-plat=/data_sdc1/lucfan01/kirkstone_xen/xen/xen/tools/cppcheck-plat --ignore-path=tools/ --cppcheck-html --“ build
I’ve investigated why and it turns out that this line 94 in xen/xen/arch/x86/arch.mk:
XEN_BUILD_EFI := $(call if-success,$(CC) $(CFLAGS) -c $(srctree)/$(efi-check).c -o $(efi-check).o,y)
is calling the compiler on a c file, so the wrapper is using cppcheck on it, but seems that $(ARCH) variable is not set at this point so the call fails and you don’t see why because the output is silenced.
The fix is simple:
diff --git a/xen/scripts/xen_analysis/cppcheck_analysis.py b/xen/scripts/xen_analysis/cppcheck_analysis.py
index e5c2f3be3e85..646826851f0b 100644
--- a/xen/scripts/xen_analysis/cppcheck_analysis.py
+++ b/xen/scripts/xen_analysis/cppcheck_analysis.py
@@ -181,6 +181,7 @@ def generate_cppcheck_deps():
cppcheck_cc_flags = """--compiler={} --cppcheck-cmd={} {}
--cppcheck-plat={}/cppcheck-plat --ignore-path=tools/
+ --ignore-path=arch/x86/efi/check.c
""".format(xen_cc, settings.cppcheck_binpath, cppcheck_flags,
settings.tools_dir)
This will instruct the cppcheck-cc.sh wrapper to don’t call the cppcheck code on the *arch/x86/efi/check.c file.
I will add it in the next serie version as well as all the comments in the serie
Cheers,
Luca
>
> Cheers,
> Luca
^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: [PATCH 3/4] tools/misra: fix skipped rule numbers
2022-11-28 14:10 ` [PATCH 3/4] tools/misra: fix skipped rule numbers Luca Fancellu
@ 2022-11-29 23:51 ` Stefano Stabellini
2022-11-30 8:53 ` Luca Fancellu
0 siblings, 1 reply; 31+ messages in thread
From: Stefano Stabellini @ 2022-11-29 23:51 UTC (permalink / raw)
To: Luca Fancellu
Cc: xen-devel, bertrand.marquis, wei.chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Stefano Stabellini,
Wei Liu
On Mon, 28 Nov 2022, Luca Fancellu wrote:
> Currently the script convert_misra_doc.py is using a loop through
> range(1,22) to enumerate rules that needs to be skipped, however
> range function does not include the stop counter in the enumeration
> ending up into list rules until 21.21 instead of including rule 22.
>
> Fix the issue using a dictionary that list the rules in misra c2012.
I think I understand the problem you are trying to solve with this
patch. But I am confused about the proposed solution.
The original code is trying to list all the possible MISRA C rules that
are not in docs/misra/rules.rst. Instead of list(range(1,22)) now we
have a dictionary: misra_c2012_rules. But misra_c2012_rules doesn't have
all the possible MISRA C rules missing from docs/misra/rules.rst.
As an example Rule 13.1 is missing from docs/misra/rules.rst but it is
also missing from misra_c2012_rules.
Can you please help me understand why misra_c2012_rules has only a small
subset of MISRA C rules to be skipped?
> Fixes: 57caa5375321 ("xen: Add MISRA support to cppcheck make rule")
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
> ---
> xen/tools/convert_misra_doc.py | 32 ++++++++++++++++++++++++++++++--
> 1 file changed, 30 insertions(+), 2 deletions(-)
>
> diff --git a/xen/tools/convert_misra_doc.py b/xen/tools/convert_misra_doc.py
> index caa4487f645f..13074d8a2e91 100755
> --- a/xen/tools/convert_misra_doc.py
> +++ b/xen/tools/convert_misra_doc.py
> @@ -14,6 +14,34 @@ Usage:
>
> import sys, getopt, re
>
> +# MISRA rule are identified by two numbers, e.g. Rule 1.2, the main rule number
> +# and a sub-number. This dictionary contains the number of the MISRA rule as key
> +# and the maximum sub-number for that rule as value.
> +misra_c2012_rules = {
> + 1:4,
> + 2:7,
> + 3:2,
> + 4:2,
> + 5:9,
> + 6:2,
> + 7:4,
> + 8:14,
> + 9:5,
> + 10:8,
> + 11:9,
> + 12:5,
> + 13:6,
> + 14:4,
> + 15:7,
> + 16:7,
> + 17:8,
> + 18:8,
> + 19:2,
> + 20:14,
> + 21:21,
> + 22:10
> +}
> +
> def main(argv):
> infile = ''
> outfile = ''
> @@ -142,8 +170,8 @@ def main(argv):
> skip_list = []
>
> # Search for missing rules and add a dummy text with the rule number
> - for i in list(range(1,22)):
> - for j in list(range(1,22)):
> + for i in misra_c2012_rules:
> + for j in list(range(1,misra_c2012_rules[i]+1)):
> if str(i) + '.' + str(j) not in rule_list:
> outstr.write('Rule ' + str(i) + '.' + str(j) + '\n')
> outstr.write('No description for rule ' + str(i) + '.' + str(j)
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-28 14:10 ` [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script Luca Fancellu
2022-11-28 15:19 ` Jan Beulich
@ 2022-11-30 1:05 ` Stefano Stabellini
2022-11-30 11:59 ` Luca Fancellu
1 sibling, 1 reply; 31+ messages in thread
From: Stefano Stabellini @ 2022-11-30 1:05 UTC (permalink / raw)
To: Luca Fancellu
Cc: xen-devel, bertrand.marquis, wei.chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Stefano Stabellini,
Wei Liu
[-- Attachment #1: Type: text/plain, Size: 58389 bytes --]
On Mon, 28 Nov 2022, Luca Fancellu wrote:
> Change cppcheck invocation method by using the xen-analysis.py
> script using the arguments --run-cppcheck.
>
> Now cppcheck analysis will build Xen while the analysis is performed
> on the source files, it will produce a text report and an additional
> html output when the script is called with --cppcheck-html.
>
> With this patch cppcheck will benefit of platform configuration files
> that will help it to understand the target of the compilation and
> improve the analysis.
>
> To do so:
> - remove cppcheck rules from Makefile and move them to the script.
> - Update xen-analysis.py with the code to integrate cppcheck.
> - merge the script merge_cppcheck_reports.py into the xen-analysis
> script package and rework the code to integrate it.
> - add platform configuration files for cppcheck..
> - add cppcheck-cc.sh script that is a wrapper for cppcheck and it's
> used as Xen compiler, it will intercept all flags given from the
> make build system and it will execute cppcheck on the compiled
> file together with the file compilation.
> - guarded hypercall-defs.c with CPPCHECK define because cppcheck
> gets confused as the file does not contain c code.
> - add false-positive-cppcheck.json file
> - update documentation.
> - update .gitignore
>
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
I think the revert of the cppcheck integration in xen/Makefile and
xen/tools/merge_cppcheck_reports.py could be a separate patch. There is
no need to make sure cppcheck support in the xen Makefile is
"bisectable". That patch could have my acked-by already.
Also the document changes introduced in this patch have my reviewed-by:
- docs/misra/cppcheck.txt
- docs/misra/documenting-violations.rst
- docs/misra/false-positive-cppcheck.json
- docs/misra/xen-static-analysis.rst
More below
> ---
> .gitignore | 8 +-
> docs/misra/cppcheck.txt | 27 +-
> docs/misra/documenting-violations.rst | 7 +-
> docs/misra/false-positive-cppcheck.json | 12 +
> docs/misra/xen-static-analysis.rst | 42 ++-
> xen/Makefile | 116 +-------
> xen/include/hypercall-defs.c | 9 +
> xen/scripts/xen-analysis.py | 18 +-
> xen/scripts/xen_analysis/cppcheck_analysis.py | 272 ++++++++++++++++++
> .../xen_analysis/cppcheck_report_utils.py | 130 +++++++++
> xen/scripts/xen_analysis/generic_analysis.py | 21 +-
> xen/scripts/xen_analysis/settings.py | 77 ++++-
> xen/scripts/xen_analysis/utils.py | 21 +-
> xen/tools/cppcheck-cc.sh | 223 ++++++++++++++
> xen/tools/cppcheck-plat/arm32-wchar_t4.xml | 17 ++
> xen/tools/cppcheck-plat/arm64-wchar_t2.xml | 17 ++
> xen/tools/cppcheck-plat/arm64-wchar_t4.xml | 17 ++
> xen/tools/cppcheck-plat/x86_64-wchar_t2.xml | 17 ++
> xen/tools/cppcheck-plat/x86_64-wchar_t4.xml | 17 ++
> xen/tools/merge_cppcheck_reports.py | 86 ------
> 20 files changed, 899 insertions(+), 255 deletions(-)
> create mode 100644 docs/misra/false-positive-cppcheck.json
> create mode 100644 xen/scripts/xen_analysis/cppcheck_analysis.py
> create mode 100644 xen/scripts/xen_analysis/cppcheck_report_utils.py
> create mode 100755 xen/tools/cppcheck-cc.sh
> create mode 100644 xen/tools/cppcheck-plat/arm32-wchar_t4.xml
> create mode 100644 xen/tools/cppcheck-plat/arm64-wchar_t2.xml
> create mode 100644 xen/tools/cppcheck-plat/arm64-wchar_t4.xml
> create mode 100644 xen/tools/cppcheck-plat/x86_64-wchar_t2.xml
> create mode 100644 xen/tools/cppcheck-plat/x86_64-wchar_t4.xml
> delete mode 100755 xen/tools/merge_cppcheck_reports.py
>
> diff --git a/.gitignore b/.gitignore
> index f5a66f6194dd..68566d0c2587 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -7,9 +7,11 @@
> *.o
> *.d
> *.d2
> -*.c.cppcheck
> +*.cppcheck.txt
> +*.cppcheck.xml
> *.opic
> *.a
> +*.c.json
> *.safparse
> *.so
> *.so.[0-9]*
> @@ -282,9 +284,11 @@ xen/arch/*/efi/efi.h
> xen/arch/*/efi/pe.c
> xen/arch/*/efi/runtime.c
> xen/arch/*/include/asm/asm-offsets.h
> +xen/build-dir-cppcheck/
> xen/common/config_data.S
> xen/common/config.gz
> xen/cppcheck-htmlreport/
> +xen/cppcheck-report/
> xen/cppcheck-misra.*
> xen/include/headers*.chk
> xen/include/compat/*
> @@ -315,7 +319,7 @@ xen/xsm/flask/xenpolicy-*
> tools/flask/policy/policy.conf
> tools/flask/policy/xenpolicy-*
> xen/xen
> -xen/xen-cppcheck.xml
> +xen/suppression-list.txt
> xen/xen-syms
> xen/xen-syms.map
> xen/xen.*
> diff --git a/docs/misra/cppcheck.txt b/docs/misra/cppcheck.txt
> index 25d8c3050b72..f7b9f678b4d5 100644
> --- a/docs/misra/cppcheck.txt
> +++ b/docs/misra/cppcheck.txt
> @@ -3,8 +3,7 @@ Cppcheck for Xen static and MISRA analysis
>
> Xen can be analysed for both static analysis problems and MISRA violation using
> cppcheck, the open source tool allows the creation of a report with all the
> -findings. Xen has introduced the support in the Makefile so it's very easy to
> -use and in this document we can see how.
> +findings.
>
> The minimum version required for cppcheck is 2.7. Note that at the time of
> writing (June 2022), the version 2.8 is known to be broken [1].
> @@ -38,27 +37,7 @@ Dependencies are listed in the readme.md of the project repository.
> Use cppcheck to analyse Xen
> ===========================
>
> -Using cppcheck integration is very simple, it requires few steps:
> -
> - 1) Compile Xen
> - 2) call the cppcheck make target to generate a report in xml format:
> - make CPPCHECK_MISRA=y cppcheck
> - 3) call the cppcheck-html make target to generate a report in xml and html
> - format:
> - make CPPCHECK_MISRA=y cppcheck-html
> -
> - In case the cppcheck binaries are not in the PATH, CPPCHECK and
> - CPPCHECK_HTMLREPORT variables can be overridden with the full path to the
> - binaries:
> -
> - make -C xen \
> - CPPCHECK=/path/to/cppcheck \
> - CPPCHECK_HTMLREPORT=/path/to/cppcheck-htmlreport \
> - CPPCHECK_MISRA=y \
> - cppcheck-html
> -
> -The output is by default in a folder named cppcheck-htmlreport, but the name
> -can be changed by passing it in the CPPCHECK_HTMLREPORT_OUTDIR variable.
> -
> +To analyse Xen using cppcheck, please refer to the document
> +xen-static-analysis.rst, section "Analyse Xen with Cppcheck".
>
> [1] https://sourceforge.net/p/cppcheck/discussion/general/thread/bfc3ab6c41/?limit=25
> diff --git a/docs/misra/documenting-violations.rst b/docs/misra/documenting-violations.rst
> index 1d23447556d2..31dafd5d4ece 100644
> --- a/docs/misra/documenting-violations.rst
> +++ b/docs/misra/documenting-violations.rst
> @@ -51,6 +51,7 @@ Here is an example to add a new justification in safe.json::
> | {
> | "id": "SAF-0-safe",
> | "analyser": {
> +| "cppcheck": "misra-c2012-20.7",
> | "coverity": "misra_c_2012_rule_20_7_violation",
> | "eclair": "MC3R1.R20.7"
> | },
> @@ -77,9 +78,9 @@ Here is an explanation of the fields inside an object of the "content" array:
> It tells the tool to substitute a Xen in-code comment having this structure:
> /* SAF-0-safe [...] \*/
> - analyser: it is an object containing pair of key-value strings, the key is
> - the analyser, so it can be coverity or eclair, the value is the proprietary
> - id corresponding on the finding, for example when coverity is used as
> - analyser, the tool will translate the Xen in-code coment in this way:
> + the analyser, so it can be cppcheck, coverity or eclair, the value is the
> + proprietary id corresponding on the finding, for example when coverity is
> + used as analyser, the tool will translate the Xen in-code coment in this way:
> /* SAF-0-safe [...] \*/ -> /* coverity[misra_c_2012_rule_20_7_violation] \*/
> if the object doesn't have a key-value, then the corresponding in-code
> comment won't be translated.
> diff --git a/docs/misra/false-positive-cppcheck.json b/docs/misra/false-positive-cppcheck.json
> new file mode 100644
> index 000000000000..5d4da2ce6170
> --- /dev/null
> +++ b/docs/misra/false-positive-cppcheck.json
> @@ -0,0 +1,12 @@
> +{
> + "version": "1.0",
> + "content": [
> + {
> + "id": "SAF-0-false-positive-cppcheck",
> + "violation-id": "",
> + "tool-version": "",
> + "name": "Sentinel",
> + "text": "Next ID to be used"
> + }
> + ]
> +}
> diff --git a/docs/misra/xen-static-analysis.rst b/docs/misra/xen-static-analysis.rst
> index 5b886474d4a0..2712255db1b0 100644
> --- a/docs/misra/xen-static-analysis.rst
> +++ b/docs/misra/xen-static-analysis.rst
> @@ -7,9 +7,8 @@ The Xen codebase integrates some scripts and tools that helps the developer to
> perform static analysis of the code, currently Xen supports three analysis tool
> that are eclair, coverity and cppcheck.
> The Xen tree has a script (xen-analysis.py) available to ease the analysis
> -process and it integrates a way to suppress findings on these tools (only Eclair
> -and Coverity are currently supported by the script), please check the
> -documenting-violation.rst document to know more about it.
> +process and it integrates a way to suppress findings on these tools, please
> +check the documenting-violation.rst document to know more about it.
>
> Analyse Xen with Coverity or Eclair
> -----------------------------------
> @@ -52,3 +51,40 @@ When invoking the script, the procedure below will be followed:
> this step, call the script adding the --no-clean argument, but before
> running again the script, call it with the --clean-only argument, that will
> execute only this cleaning step.
> +
> +
> +Analyse Xen with Cppcheck
> +-------------------------
> +
> +Cppcheck tool is integrated in xen-analysis.py script, when using the script,
> +the tool will be called on every source file compiled by the make build system.
> +Here how to start the analysis with Cppcheck:
> +
> + - xen-analysis.py --run-cppcheck [--cppcheck-misra] [--cppcheck-html] --
> + [optional make arguments]
> +
> +The command above tells the script to prepare the codebase and use Cppcheck tool
> +for the analysis.
> +The optional argument --cppcheck-misra activates the analysis also for MISRA
> +compliance.
> +The optional argument --cppcheck-html instruct cppcheck to produce an additional
> +HTML report.
> +
> +When invoking the script for Cppcheck analysis, the followed procedure is
> +similar to the one above for Coverity or Eclair, but it has some additional
> +steps:
> +
> + 1. This step is the same as step 1 for Coverity/Eclair.
> + 2. The cppcheck dependency are created, build directory for cppcheck analysis
> + and an header file containing internal compiler macro
> + (include/generated/compiler-def.h) are generated
> + 3. Xen compilation starts using every <additional make parameters> supplied
> + at the script invocation, but because cppcheck is not able to intercept the
> + compiled files and flags on compiler invocation, a script (cppcheck-cc.sh)
> + is passed as CC to the make system, it is a wrapper for the compiler that
> + will also execute cppcheck on every compiled file.
> + 4. After the compilation and analysis, the cppcheck report will be created
> + putting together all the cppcheck report fragments for every analysed file.
> + Cppcheck will produce a text fragment and an additional XML report fragment
> + if the script is configured to produce the HTML output.
> + 5. This step is the same as step 3 for Coverity/Eclair.
> diff --git a/xen/Makefile b/xen/Makefile
> index 9d0df5e2c543..77926724bcd7 100644
> --- a/xen/Makefile
> +++ b/xen/Makefile
> @@ -457,7 +457,7 @@ endif # need-config
>
> __all: build
>
> -main-targets := build install uninstall clean distclean MAP cppcheck cppcheck-html
> +main-targets := build install uninstall clean distclean MAP
> .PHONY: $(main-targets)
> ifneq ($(XEN_TARGET_ARCH),x86_32)
> $(main-targets): %: _% ;
> @@ -566,18 +566,16 @@ _clean:
> $(Q)$(MAKE) $(clean)=tools/kconfig
> find . \( -name "*.o" -o -name ".*.d" -o -name ".*.d2" \
> -o -name ".*.o.tmp" -o -name "*~" -o -name "core" \
> - -o -name '*.lex.c' -o -name '*.tab.[ch]' -o -name '*.c.cppcheck' \
> - -o -name "*.gcno" -o -name ".*.cmd" -o -name "lib.a" \) -exec rm -f {} \;
> + -o -name '*.lex.c' -o -name '*.tab.[ch]' -o -name "*.gcno" \
> + -o -name ".*.cmd" -o -name "lib.a" \) -exec rm -f {} \;
> rm -f include/asm $(TARGET) $(TARGET).gz $(TARGET)-syms $(TARGET)-syms.map
> rm -f $(TARGET).efi $(TARGET).efi.map $(TARGET).efi.stripped
> rm -f asm-offsets.s arch/*/include/asm/asm-offsets.h
> rm -f .banner .allconfig.tmp include/xen/compile.h
> - rm -f cppcheck-misra.* xen-cppcheck.xml
>
> .PHONY: _distclean
> _distclean: clean
> rm -f tags TAGS cscope.files cscope.in.out cscope.out cscope.po.out GTAGS GPATH GRTAGS GSYMS .config source
> - rm -rf $(CPPCHECK_HTMLREPORT_OUTDIR)
>
> $(TARGET).gz: $(TARGET)
> gzip -n -f -9 < $< > $@.new
> @@ -651,111 +649,9 @@ cloc:
> done; \
> done | cloc --list-file=-
>
> -# What cppcheck command to use.
> -# To get proper results, it is recommended to build cppcheck manually from the
> -# latest source and use CPPCHECK to give the full path to the built version.
> -CPPCHECK ?= cppcheck
> -
> -# What cppcheck-htmlreport to use.
> -# If you give the full path to a self compiled cppcheck, this should be set
> -# to the full path to cppcheck-html in the htmlreport directory of cppcheck.
> -# On recent distribution, this is available in the standard path.
> -CPPCHECK_HTMLREPORT ?= cppcheck-htmlreport
> -
> -# By default we generate the report in cppcheck-htmlreport directory in the
> -# build directory. This can be changed by giving a directory in this variable.
> -CPPCHECK_HTMLREPORT_OUTDIR ?= cppcheck-htmlreport
> -
> -# By default we do not check misra rules, to enable pass "CPPCHECK_MISRA=y" to
> -# make command line.
> -CPPCHECK_MISRA ?= n
> -
> -# Compile flags to pass to cppcheck:
> -# - include directories and defines Xen Makefile is passing (from CFLAGS)
> -# - include config.h as this is passed directly to the compiler.
> -# - define CPPCHECK as we use to disable or enable some specific part of the
> -# code to solve some cppcheck issues.
> -# - explicitely enable some cppcheck checks as we do not want to use "all"
> -# which includes unusedFunction which gives wrong positives as we check file
> -# per file.
> -#
> -# Compiler defines are in compiler-def.h which is included in config.h
> -#
> -CPPCHECKFLAGS := -DCPPCHECK --max-ctu-depth=10 \
> - --enable=style,information,missingInclude \
> - --include=$(srctree)/include/xen/config.h \
> - -I $(srctree)/xsm/flask/include \
> - -I $(srctree)/include/xen/libfdt \
> - $(filter -D% -I%,$(CFLAGS))
> -
> -# We need to find all C files (as we are not checking assembly files) so
> -# we find all generated .o files which have a .c corresponding file.
> -CPPCHECKFILES := $(wildcard $(patsubst $(objtree)/%.o,$(srctree)/%.c, \
> - $(filter-out $(objtree)/tools/%, \
> - $(shell find $(objtree) -name "*.o"))))
> -
> -# Headers and files required to run cppcheck on a file
> -CPPCHECKDEPS := $(objtree)/include/generated/autoconf.h \
> - $(objtree)/include/generated/compiler-def.h
> -
> -ifeq ($(CPPCHECK_MISRA),y)
> - CPPCHECKFLAGS += --addon=cppcheck-misra.json
> - CPPCHECKDEPS += cppcheck-misra.json
> -endif
> -
> -quiet_cmd_cppcheck_xml = CPPCHECK $(patsubst $(srctree)/%,%,$<)
> -cmd_cppcheck_xml = $(CPPCHECK) -v -q --xml $(CPPCHECKFLAGS) \
> - --output-file=$@ $<
> -
> -quiet_cmd_merge_cppcheck_reports = CPPCHECK-MERGE $@
> -cmd_merge_cppcheck_reports = $(PYTHON) $(srctree)/tools/merge_cppcheck_reports.py $^ $@
> -
> -quiet_cmd_cppcheck_html = CPPCHECK-HTML $<
> -cmd_cppcheck_html = $(CPPCHECK_HTMLREPORT) --file=$< --source-dir=$(srctree) \
> - --report-dir=$(CPPCHECK_HTMLREPORT_OUTDIR) --title=Xen
> -
> -PHONY += _cppcheck _cppcheck-html cppcheck-version
> -
> -_cppcheck-html: xen-cppcheck.xml
> - $(call if_changed,cppcheck_html)
> -
> -_cppcheck: xen-cppcheck.xml
> -
> -xen-cppcheck.xml: $(patsubst $(srctree)/%.c,$(objtree)/%.c.cppcheck,$(CPPCHECKFILES))
> -ifeq ($(CPPCHECKFILES),)
> - $(error Please build Xen before running cppcheck)
> -endif
> - $(call if_changed,merge_cppcheck_reports)
> -
> -$(objtree)/%.c.cppcheck: $(srctree)/%.c $(CPPCHECKDEPS) | cppcheck-version
> - $(call if_changed,cppcheck_xml)
> -
> -cppcheck-version:
> - $(Q)if ! which $(CPPCHECK) > /dev/null 2>&1; then \
> - echo "Cannot find cppcheck executable: $(CPPCHECK)"; \
> - exit 1; \
> - fi
> - $(Q)if [ "$$($(CPPCHECK) --version | awk '{print ($$2 < 2.7)}')" -eq 1 ]; then \
> - echo "Please upgrade your cppcheck to version 2.7 or greater"; \
> - exit 1; \
> - fi
> -
> -# List of Misra rules to respect is written inside a doc.
> -# In order to have some helpful text in the cppcheck output, generate a text
> -# file containing the rules identifier, classification and text from the Xen
> -# documentation file. Also generate a json file with the right arguments for
> -# cppcheck in json format including the list of rules to ignore.
> -#
> -# convert_misra_doc.py, producing both targets at the same time, should be
> -# executed only once. Utilize a pattern rule to achieve this effect, with the
> -# stem kind of arbitrarily chosen to be "cppcheck".
> -.PRECIOUS: %-misra.json
> -%-misra.txt %-misra.json: $(XEN_ROOT)/docs/misra/rules.rst $(srctree)/tools/convert_misra_doc.py
> - $(Q)$(PYTHON) $(srctree)/tools/convert_misra_doc.py -i $< -o $*-misra.txt -j $*-misra.json
> -
> -# Put this in generated headers this way it is cleaned by include/Makefile
> -$(objtree)/include/generated/compiler-def.h:
> - $(Q)$(CC) -dM -E -o $@ - < /dev/null
> +# Target used by xen-analysis.sh script to retrieve Xen build system variables
> +export-variable-%:
> + $(info $*=$($*))
>
> endif #config-build
> endif # need-sub-make
> diff --git a/xen/include/hypercall-defs.c b/xen/include/hypercall-defs.c
> index 45b6f969d2ab..3d1eb7f04a73 100644
> --- a/xen/include/hypercall-defs.c
> +++ b/xen/include/hypercall-defs.c
> @@ -60,6 +60,13 @@
> * are possible.
> */
>
> +/*
> + * Cppcheck thinks this file needs to be analysed because it is preprocessed by
> + * the compiler, but it gets confused because this file does not contains C
> + * code. Hence protect the code when CPPCHECK is used.
> + */
> +#ifndef CPPCHECK
> +
> #ifdef CONFIG_HVM
> #define PREFIX_hvm hvm
> #else
> @@ -286,3 +293,5 @@ mca do do - - -
> #ifndef CONFIG_PV_SHIM_EXCLUSIVE
> paging_domctl_cont do do do do -
> #endif
> +
> +#endif /* !CPPCHECK */
> diff --git a/xen/scripts/xen-analysis.py b/xen/scripts/xen-analysis.py
> index b5d9ef1862c9..8e50c27cd898 100755
> --- a/xen/scripts/xen-analysis.py
> +++ b/xen/scripts/xen-analysis.py
> @@ -1,28 +1,42 @@
> #!/usr/bin/env python3
>
> import sys
> -from xen_analysis import settings, generic_analysis
> +from xen_analysis import settings, generic_analysis, cppcheck_analysis
> from xen_analysis.generic_analysis import *
> +from xen_analysis.cppcheck_analysis import *
> +
> +PhaseExceptions = (GetMakeVarsPhaseError, ParseTagPhaseError,
> + CppcheckDepsPhaseError, BuildPhaseError,
> + CppcheckReportPhaseError)
>
>
> def main(argv):
> ret_code = 0
> settings.parse_commandline(argv)
> try:
> + if settings.step_get_make_vars:
> + cppcheck_analysis.get_make_vars()
> if settings.step_parse_tags:
> generic_analysis.parse_xen_tags()
> + if settings.step_cppcheck_deps:
> + cppcheck_analysis.generate_cppcheck_deps()
> if settings.step_build_xen:
> generic_analysis.build_xen()
> - except (ParseTagPhaseError, BuildPhaseError) as e:
> + if settings.step_cppcheck_report:
> + cppcheck_analysis.generate_cppcheck_report()
> + except PhaseExceptions as e:
> print("ERROR: {}".format(e))
> if hasattr(e, "errorcode"):
> ret_code = e.errorcode
> finally:
> if settings.step_clean_analysis:
> + cppcheck_analysis.clean_analysis_artifacts()
> e = generic_analysis.clean_analysis_artifacts()
> if e:
> print("ERROR: {}".format(e))
> ret_code = 1
> + if settings.step_distclean_analysis:
> + cppcheck_analysis.clean_reports()
>
> sys.exit(ret_code)
>
> diff --git a/xen/scripts/xen_analysis/cppcheck_analysis.py b/xen/scripts/xen_analysis/cppcheck_analysis.py
> new file mode 100644
> index 000000000000..e5c2f3be3e85
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/cppcheck_analysis.py
> @@ -0,0 +1,272 @@
> +#!/usr/bin/env python3
> +
> +import os, re, shutil
> +from . import settings, utils, cppcheck_report_utils
> +
> +class GetMakeVarsPhaseError(Exception):
> + pass
> +
> +class CppcheckDepsPhaseError(Exception):
> + pass
> +
> +class CppcheckReportPhaseError(Exception):
> + pass
> +
> +CPPCHECK_BUILD_DIR = "build-dir-cppcheck"
> +CPPCHECK_HTMLREPORT_OUTDIR = "cppcheck-htmlreport"
> +CPPCHECK_REPORT_OUTDIR = "cppcheck-report"
> +cppcheck_extra_make_args = ""
> +xen_cc = ""
> +
> +def get_make_vars():
> + global xen_cc
> + invoke_make = utils.invoke_command(
> + "make -C {} {} export-variable-CC"
> + .format(settings.xen_dir, settings.make_forward_args),
> + True, GetMakeVarsPhaseError,
> + "Error occured retrieving make vars:\n{}"
> + )
> +
> + cc_var_regex = re.search('^CC=(.*)$', invoke_make, flags=re.M)
> + if cc_var_regex:
> + xen_cc = cc_var_regex.group(1)
> +
> + if xen_cc == "":
> + raise GetMakeVarsPhaseError("CC variable not found in Xen make output")
> +
> +
> +def __generate_suppression_list(out_file):
> + # The following lambda function will return a file if it contains lines with
> + # a comment containing "cppcheck-suppress[*]" on a single line.
> + grep_action = lambda x: utils.grep(x,
> + r'^[ \t]*/\* cppcheck-suppress\[(.*)\] \*/$')
> + # Look for a list of .h files that matches the condition above
> + headers = utils.recursive_find_file(settings.xen_dir, r'.*\.h$',
> + grep_action)
> +
> + try:
> + with open(out_file, "wt") as supplist_file:
> + # Add this rule to skip every finding in the autogenerated
> + # header for cppcheck
> + supplist_file.write("*:*generated/compiler-def.h\n")
> +
> + for entry in headers:
> + filename = entry["file"]
> + try:
> + with open(filename, "rt") as infile:
> + header_content = infile.readlines()
> + except OSError as e:
> + raise CppcheckDepsPhaseError(
> + "Issue with reading file {}: {}"
> + .format(filename, e)
> + )
> + header_lines_len = len(header_content)
> + # line_num in entry will be header_content[line_num-1], here we
> + # are going to search the first line after line_num that have
> + # anything different from comments or empty line, because the
> + # in-code comment suppression is related to that line then.
> + for line_num in entry["matches"]:
> + cppcheck_violation_id = ""
> + tmp_line = line_num
> + # look up to which line is referring the comment at
> + # line_num (which would be header_content[tmp_line-1])
> + comment_section = False
> + while tmp_line < header_lines_len:
> + line = header_content[tmp_line]
> + # Matches a line with just optional spaces/tabs and the
> + # start of a comment '/*'
> + comment_line_starts = re.match('^[ \t]*/\*.*$', line)
> + # Matches a line with text and the end of a comment '*/'
> + comment_line_stops = re.match('^.*\*/$', line)
> + if (not comment_section) and comment_line_starts:
> + comment_section = True
> + if (len(line.strip()) != 0) and (not comment_section):
> + cppcheck_violation_id = entry["matches"][line_num][0]
> + break
> + if comment_section and comment_line_stops:
> + comment_section = False
> + tmp_line = tmp_line + 1
> +
> + if cppcheck_violation_id == "":
> + raise CppcheckDepsPhaseError(
> + "Error matching cppcheck comment in {} at line {}."
> + .format(filename, line_num)
> + )
> + # Write [error id]:[filename]:[line]
> + # tmp_line refers to the array index, so translated to the
> + # file line (that begins with 1) it is tmp_line+1
> + supplist_file.write(
> + "{}:{}:{}\n".format(cppcheck_violation_id, filename,
> + (tmp_line + 1))
> + )
> + except OSError as e:
> + raise CppcheckDepsPhaseError("Issue with writing file {}: {}"
> + .format(out_file, e))
> +
> +
> +def generate_cppcheck_deps():
> + global cppcheck_extra_make_args
> +
> + # Compile flags to pass to cppcheck:
> + # - include config.h as this is passed directly to the compiler.
> + # - define CPPCHECK as we use it to disable or enable some specific part of
> + # the code to solve some cppcheck issues.
> + # - explicitely enable some cppcheck checks as we do not want to use "all"
> + # which includes unusedFunction which gives wrong positives as we check
> + # file per file.
> + # - Explicitly suppress warnings on compiler-def.h because cppcheck throws
> + # an unmatchedSuppression due to the rule we put in suppression-list.txt
> + # to skip every finding in the file.
> + #
> + # Compiler defines are in compiler-def.h which is included in config.h
> + #
> + cppcheck_flags="""
> +--cppcheck-build-dir={}/{}
> + --max-ctu-depth=10
> + --enable=style,information,missingInclude
> + --template=\'{{file}}({{line}},{{column}}):{{id}}:{{severity}}:{{message}}\'
> + --relative-paths={}
> + --inline-suppr
> + --suppressions-list={}/suppression-list.txt
> + --suppress='unmatchedSuppression:*generated/compiler-def.h'
> + --include={}/include/xen/config.h
I noticed that some of the includes we used to have like
xsm/flask/include are missing here. Is that intended?
> + -DCPPCHECK
> +""".format(settings.outdir, CPPCHECK_BUILD_DIR, settings.xen_dir,
> + settings.outdir, settings.xen_dir)
> +
> + invoke_cppcheck = utils.invoke_command(
> + "{} --version".format(settings.cppcheck_binpath),
> + True, CppcheckDepsPhaseError,
> + "Error occured retrieving cppcheck version:\n{}\n\n{}"
> + )
> +
> + version_regex = re.search('^Cppcheck (.*)$', invoke_cppcheck, flags=re.M)
> + # Currently, only cppcheck version >= 2.7 is supported, but version 2.8 is
> + # known to be broken, please refer to docs/misra/cppcheck.txt
> + if (not version_regex) or (version_regex.group(1) != "2.7"):
> + raise CppcheckDepsPhaseError(
> + "Can't find cppcheck version or version is not 2.7"
> + )
> +
> + # If misra option is selected, append misra addon and generate cppcheck
> + # files for misra analysis
> + if settings.cppcheck_misra:
> + cppcheck_flags = cppcheck_flags + " --addon=cppcheck-misra.json"
> +
> + utils.invoke_command(
> + "{}/convert_misra_doc.py -i {}/docs/misra/rules.rst"
> + " -o {}/cppcheck-misra.txt -j {}/cppcheck-misra.json"
> + .format(settings.tools_dir, settings.repo_dir,
> + settings.outdir, settings.outdir),
> + False, CppcheckDepsPhaseError,
> + "An error occured when running:\n{}"
> + )
> +
> + # Generate compiler macros
> + os.makedirs("{}/include/generated".format(settings.outdir), exist_ok=True)
> + utils.invoke_command(
> + "{} -dM -E -o \"{}/include/generated/compiler-def.h\" - < /dev/null"
> + .format(xen_cc, settings.outdir),
> + False, CppcheckDepsPhaseError,
> + "An error occured when running:\n{}"
> + )
> +
> + # Generate cppcheck suppression list
> + __generate_suppression_list(
> + "{}/suppression-list.txt".format(settings.outdir))
> +
> + # Generate cppcheck build folder
> + os.makedirs("{}/{}".format(settings.outdir, CPPCHECK_BUILD_DIR),
> + exist_ok=True)
> +
> + cppcheck_cc_flags = """--compiler={} --cppcheck-cmd={} {}
> + --cppcheck-plat={}/cppcheck-plat --ignore-path=tools/
> +""".format(xen_cc, settings.cppcheck_binpath, cppcheck_flags,
> + settings.tools_dir)
> +
> + if settings.cppcheck_html:
> + cppcheck_cc_flags = cppcheck_cc_flags + " --cppcheck-html"
> +
> + # Generate the extra make argument to pass the cppcheck-cc.sh wrapper as CC
> + cppcheck_extra_make_args = "CC=\"{}/cppcheck-cc.sh {} --\"".format(
> + settings.tools_dir,
> + cppcheck_cc_flags
> + ).replace("\n", "")
> +
> +
> +def generate_cppcheck_report():
> + # Prepare text report
> + # Look for a list of .cppcheck.txt files, those are the txt report
> + # fragments
> + fragments = utils.recursive_find_file(settings.outdir, r'.*\.cppcheck.txt$')
> + text_report_dir = "{}/{}".format(settings.outdir,
> + CPPCHECK_REPORT_OUTDIR)
> + report_filename = "{}/xen-cppcheck.txt".format(text_report_dir)
> + os.makedirs(text_report_dir, exist_ok=True)
> + try:
> + cppcheck_report_utils.cppcheck_merge_txt_fragments(fragments,
> + report_filename,
> + [settings.xen_dir])
> + except cppcheck_report_utils.CppcheckTXTReportError as e:
> + raise CppcheckReportPhaseError(e)
> +
> + # If HTML output is requested
> + if settings.cppcheck_html:
> + # Look for a list of .cppcheck.xml files, those are the XML report
> + # fragments
> + fragments = utils.recursive_find_file(settings.outdir,
> + r'.*\.cppcheck.xml$')
> + html_report_dir = "{}/{}".format(settings.outdir,
> + CPPCHECK_HTMLREPORT_OUTDIR)
> + xml_filename = "{}/xen-cppcheck.xml".format(html_report_dir)
> + os.makedirs(html_report_dir, exist_ok=True)
> + try:
> + cppcheck_report_utils.cppcheck_merge_xml_fragments(fragments,
> + xml_filename,
> + settings.xen_dir,
> + settings.outdir)
> + except cppcheck_report_utils.CppcheckHTMLReportError as e:
> + raise CppcheckReportPhaseError(e)
> + # Call cppcheck-htmlreport utility to generate the HTML output
> + utils.invoke_command(
> + "{} --file={} --source-dir={} --report-dir={}/html --title=Xen"
> + .format(settings.cppcheck_htmlreport_binpath, xml_filename,
> + settings.xen_dir, html_report_dir),
> + False, CppcheckReportPhaseError,
> + "Error occured generating Cppcheck HTML report:\n{}"
> + )
> + # Strip src and obj path from *.html files
> + html_files = utils.recursive_find_file(html_report_dir, r'.*\.html$')
> + try:
> + cppcheck_report_utils.cppcheck_strip_path_html(html_files,
> + (settings.xen_dir,
> + settings.outdir))
> + except cppcheck_report_utils.CppcheckHTMLReportError as e:
> + raise CppcheckReportPhaseError(e)
> +
> +
> +def clean_analysis_artifacts():
> + clean_files = ("suppression-list.txt", "cppcheck-misra.txt",
> + "cppcheck-misra.json")
> + cppcheck_build_dir = "{}/{}".format(settings.outdir, CPPCHECK_BUILD_DIR)
> + if os.path.isdir(cppcheck_build_dir):
> + shutil.rmtree(cppcheck_build_dir)
> + artifact_files = utils.recursive_find_file(settings.outdir,
> + r'.*\.(?:c\.json|cppcheck\.txt|cppcheck\.xml)$')
> + for file in clean_files:
> + file = "{}/{}".format(settings.outdir, file)
> + if os.path.isfile(file):
> + artifact_files.append(file)
> + for delfile in artifact_files:
> + os.remove(delfile)
> +
> +
> +def clean_reports():
> + text_report_dir = "{}/{}".format(settings.outdir,
> + CPPCHECK_REPORT_OUTDIR)
> + html_report_dir = "{}/{}".format(settings.outdir,
> + CPPCHECK_HTMLREPORT_OUTDIR)
> + if os.path.isdir(text_report_dir):
> + shutil.rmtree(text_report_dir)
> + if os.path.isdir(html_report_dir):
> + shutil.rmtree(html_report_dir)
> diff --git a/xen/scripts/xen_analysis/cppcheck_report_utils.py b/xen/scripts/xen_analysis/cppcheck_report_utils.py
> new file mode 100644
> index 000000000000..02440aefdfec
> --- /dev/null
> +++ b/xen/scripts/xen_analysis/cppcheck_report_utils.py
> @@ -0,0 +1,130 @@
> +#!/usr/bin/env python3
> +
> +import os
> +from xml.etree import ElementTree
> +
> +class CppcheckHTMLReportError(Exception):
> + pass
> +
> +class CppcheckTXTReportError(Exception):
> + pass
> +
> +
> +def __elements_equal(el1, el2):
> + if type(el1) != type(el2): return False
> +
> + if el1.find('location') is None: return False
> + if el2.find('location') is None: return False
> +
> + el1_location = str(el1.find('location').attrib)
> + el2_location = str(el2.find('location').attrib)
> +
> + if el1_location != el2_location: return False
> +
> + return True
> +
> +
> +def __contain_element(new, lst):
> + for elem in lst:
> + if __elements_equal(new, elem):
> + return True
> + return False
> +
> +
> +def __get_xml_root_file(filename):
> + try:
> + result_xml_root = ElementTree.parse(filename).getroot()
> + except ElementTree.ParseError as e:
> + raise CppcheckHTMLReportError(
> + "XML parsing error in {}: {}".format(filename, e)
> + )
> + return result_xml_root
> +
> +
> +def __sanitize_cppcheck_xml_path(xml_tree, src_path, obj_path):
> + # Some path are relative to the source tree but some others are generated
> + # in the obj tree, for cppcheck when using cppcheck-htmlreport we can pass
> + # only one source tree where the files will be fetched if relative path are
> + # found. So for every path that does not exists in src tree, we guess it
> + # comes from obj tree and we put explicit absolute path to it
> + error_item_root = xml_tree.findall("errors")[0]
> + for error_item in error_item_root:
> + for location_item in error_item.findall("location"):
> + path = location_item.attrib["file"]
> + new_obj_path = obj_path + "/" + path
> + new_src_path = src_path + "/" + path
> + if (path[0] != "/") and (not os.path.isfile(new_src_path)) \
> + and os.path.isfile(new_obj_path):
> + location_item.attrib["file"] = new_obj_path
> +
> +
> +def cppcheck_merge_xml_fragments(fragments_list, out_xml_file, src_path,
> + obj_path):
> +
> + result_xml = __get_xml_root_file(fragments_list[0])
> + insert_point = result_xml.findall("errors")[0]
> + for xml_file in fragments_list[1:]:
> + xml_root = __get_xml_root_file(xml_file)
> + curr_elem_list = list(insert_point)
> + new_elem_list = list(xml_root.findall("errors")[0])
> + for xml_error_elem in new_elem_list:
> + if not __contain_element(xml_error_elem, curr_elem_list):
> + insert_point.insert(1, xml_error_elem)
> +
> + if result_xml is None:
> + return False
> +
> + __sanitize_cppcheck_xml_path(result_xml, src_path, obj_path)
> +
> + ElementTree.ElementTree(result_xml).write(out_xml_file)
> +
> + return True
> +
> +
> +def cppcheck_merge_txt_fragments(fragments_list, out_txt_file, strip_paths):
> + try:
> + with open(out_txt_file, "wt") as outfile:
> + # Using a set will remove automatically the duplicate lines
> + text_report_content = set()
> + for file in fragments_list:
> + try:
> + with open(file, "rt") as infile:
> + frag_lines = infile.readlines()
> + except OSError as e:
> + raise CppcheckTXTReportError(
> + "Issue with reading file {}: {}"
> + .format(file, e)
> + )
> + text_report_content.update(frag_lines)
> +
> + # Back to modifiable list
> + text_report_content = list(text_report_content)
> + # Strip path from report lines
> + for i in list(range(0, len(text_report_content))):
> + for path in strip_paths:
> + text_report_content[i] = text_report_content[i].replace(
> + path + "/", "")
> + # Write the final text report
> + outfile.writelines(text_report_content)
> + except OSError as e:
> + raise CppcheckTXTReportError("Issue with writing file {}: {}"
> + .format(out_txt_file, e))
> +
> +
> +def cppcheck_strip_path_html(html_files, strip_paths):
> + for file in html_files:
> + try:
> + with open(file, "rt") as infile:
> + html_lines = infile.readlines()
> + except OSError as e:
> + raise CppcheckHTMLReportError("Issue with reading file {}: {}"
> + .format(file, e))
> + for i in list(range(0, len(html_lines))):
> + for path in strip_paths:
> + html_lines[i] = html_lines[i].replace(path + "/", "")
> + try:
> + with open(file, "wt") as outfile:
> + outfile.writelines(html_lines)
> + except OSError as e:
> + raise CppcheckHTMLReportError("Issue with writing file {}: {}"
> + .format(file, e))
> diff --git a/xen/scripts/xen_analysis/generic_analysis.py b/xen/scripts/xen_analysis/generic_analysis.py
> index 0b470c4ecf7d..94122aebace0 100644
> --- a/xen/scripts/xen_analysis/generic_analysis.py
> +++ b/xen/scripts/xen_analysis/generic_analysis.py
> @@ -1,7 +1,7 @@
> #!/usr/bin/env python3
>
> -import os, subprocess
> -from . import settings, utils, tag_database
> +import os
> +from . import settings, utils, tag_database, cppcheck_analysis
>
> class ParseTagPhaseError(Exception):
> pass
> @@ -60,18 +60,13 @@ def parse_xen_tags():
>
>
> def build_xen():
> - try:
> - subprocess.run(
> - "make -C {} {} build"
> - .format(settings.xen_dir, settings.make_forward_args),
> - shell=True, check=True
> + utils.invoke_command(
> + "make -C {} {} {} build"
> + .format(settings.xen_dir, settings.make_forward_args,
> + cppcheck_analysis.cppcheck_extra_make_args),
> + False, BuildPhaseError,
> + "Build error occured when running:\n{}"
> )
> - except (subprocess.CalledProcessError, subprocess.SubprocessError) as e:
> - excp = BuildPhaseError(
> - "Build error occured when running:\n{}".format(e.cmd)
> - )
> - excp.errorcode = e.returncode if hasattr(e, 'returncode') else 1
> - raise excp
>
>
> def clean_analysis_artifacts():
> diff --git a/xen/scripts/xen_analysis/settings.py b/xen/scripts/xen_analysis/settings.py
> index 947dfa2d50af..bd1faafe79a3 100644
> --- a/xen/scripts/xen_analysis/settings.py
> +++ b/xen/scripts/xen_analysis/settings.py
> @@ -7,14 +7,23 @@ xen_dir = os.path.realpath(module_dir + "/../..")
> repo_dir = os.path.realpath(xen_dir + "/..")
> tools_dir = os.path.realpath(xen_dir + "/tools")
>
> +step_get_make_vars = False
> step_parse_tags = True
> +step_cppcheck_deps = False
> step_build_xen = True
> +step_cppcheck_report = False
> step_clean_analysis = True
> +step_distclean_analysis = False
>
> target_build = False
> target_clean = False
> +target_distclean = False
>
> analysis_tool = ""
> +cppcheck_binpath = "cppcheck"
> +cppcheck_html = False
> +cppcheck_htmlreport_binpath = "cppcheck-htmlreport"
> +cppcheck_misra = False
> make_forward_args = ""
> outdir = xen_dir
>
> @@ -26,29 +35,47 @@ Usage: {} [OPTION] ... [-- [make arguments]]
> This script runs the analysis on the Xen codebase.
>
> Options:
> - --build-only Run only the commands to build Xen with the optional make
> - arguments passed to the script
> - --clean-only Run only the commands to clean the analysis artifacts
> - -h, --help Print this help
> - --no-build Skip the build Xen phase
> - --no-clean Don\'t clean the analysis artifacts on exit
> - --run-coverity Run the analysis for the Coverity tool
> - --run-eclair Run the analysis for the Eclair tool
> + --build-only Run only the commands to build Xen with the optional
> + make arguments passed to the script
> + --clean-only Run only the commands to clean the analysis artifacts
> + --cppcheck-bin= Path to the cppcheck binary (Default: {})
> + --cppcheck-html Produce an additional HTML output report for Cppcheck
> + --cppcheck-html-bin= Path to the cppcheck-html binary (Default: {})
> + --cppcheck-misra Activate the Cppcheck MISRA analysis
> + --distclean Clean analysis artifacts and reports
> + -h, --help Print this help
> + --no-build Skip the build Xen phase
> + --no-clean Don\'t clean the analysis artifacts on exit
> + --run-coverity Run the analysis for the Coverity tool
> + --run-cppcheck Run the Cppcheck analysis tool on Xen
> + --run-eclair Run the analysis for the Eclair tool
> """
> - print(msg.format(sys.argv[0]))
> + print(msg.format(sys.argv[0], cppcheck_binpath,
> + cppcheck_htmlreport_binpath))
>
>
> def parse_commandline(argv):
> global analysis_tool
> + global cppcheck_binpath
> + global cppcheck_html
> + global cppcheck_htmlreport_binpath
> + global cppcheck_misra
> global make_forward_args
> global outdir
> + global step_get_make_vars
> global step_parse_tags
> + global step_cppcheck_deps
> global step_build_xen
> + global step_cppcheck_report
> global step_clean_analysis
> + global step_distclean_analysis
> global target_build
> global target_clean
> + global target_distclean
> forward_to_make = False
> for option in argv:
> + args_with_content_regex = re.match(r'^(--[a-z]+[a-z-]*)=(.*)$', option)
> +
> if forward_to_make:
> # Intercept outdir
> outdir_regex = re.match("^O=(.*)$", option)
> @@ -60,6 +87,18 @@ def parse_commandline(argv):
> target_build = True
> elif option == "--clean-only":
> target_clean = True
> + elif args_with_content_regex and \
> + args_with_content_regex.group(1) == "--cppcheck-bin":
> + cppcheck_binpath = args_with_content_regex.group(2)
> + elif option == "--cppcheck-html":
> + cppcheck_html = True
> + elif args_with_content_regex and \
> + args_with_content_regex.group(1) == "--cppcheck-html-bin":
> + cppcheck_htmlreport_binpath = args_with_content_regex.group(2)
> + elif option == "--cppcheck-misra":
> + cppcheck_misra = True
> + elif option == "--distclean":
> + target_distclean = True
> elif (option == "--help") or (option == "-h"):
> help()
> sys.exit(0)
> @@ -69,6 +108,11 @@ def parse_commandline(argv):
> step_clean_analysis = False
> elif (option == "--run-coverity") or (option == "--run-eclair"):
> analysis_tool = option[6:]
> + elif (option == "--run-cppcheck"):
> + analysis_tool = "cppcheck"
> + step_get_make_vars = True
> + step_cppcheck_deps = True
> + step_cppcheck_report = True
> elif option == "--":
> forward_to_make = True
> else:
> @@ -76,13 +120,23 @@ def parse_commandline(argv):
> help()
> sys.exit(1)
>
> - if target_build and target_clean:
> - print("--build-only is not compatible with --clean-only argument.")
> + if target_build and (target_clean or target_distclean):
> + print("--build-only is not compatible with --clean-only/--distclean "
> + "argument.")
> sys.exit(1)
>
> + if target_distclean:
> + # Implicit activation of clean target
> + target_clean = True
> +
> + step_distclean_analysis = True
> +
> if target_clean:
> + step_get_make_vars = False
> step_parse_tags = False
> + step_cppcheck_deps = False
> step_build_xen = False
> + step_cppcheck_report = False
> step_clean_analysis = True
> return
>
> @@ -95,3 +149,4 @@ def parse_commandline(argv):
> step_parse_tags = False
> step_build_xen = True
> step_clean_analysis = False
> + step_cppcheck_report = False
> diff --git a/xen/scripts/xen_analysis/utils.py b/xen/scripts/xen_analysis/utils.py
> index a912d812c3df..1193e3f4631e 100644
> --- a/xen/scripts/xen_analysis/utils.py
> +++ b/xen/scripts/xen_analysis/utils.py
> @@ -1,6 +1,6 @@
> #!/usr/bin/env python3
>
> -import os, re
> +import os, re, subprocess
>
>
> def grep(filepath, regex):
> @@ -35,3 +35,22 @@ def recursive_find_file(path, filename_regex, action = None):
> res.append(out)
>
> return res
> +
> +
> +def invoke_command(command, needs_output, exeption_type = Exception,
> + exeption_msg = ""):
> + try:
> + pipe_stdout = subprocess.PIPE if (needs_output == True) else None
> + output = subprocess.run(command, shell=True, check=True,
> + stdout=pipe_stdout, stderr=subprocess.STDOUT,
> + encoding='utf8')
> + except (subprocess.CalledProcessError, subprocess.SubprocessError) as e:
> + if needs_output == True:
> + exeption_msg = exeption_msg.format(e.cmd, output.stdout)
> + else:
> + exeption_msg = exeption_msg.format(e.cmd)
> + excp = exeption_type(exeption_msg)
> + excp.errorcode = e.returncode if hasattr(e, 'returncode') else 1
> + raise excp
> +
> + return output.stdout
> diff --git a/xen/tools/cppcheck-cc.sh b/xen/tools/cppcheck-cc.sh
> new file mode 100755
> index 000000000000..e682f6b9d79d
> --- /dev/null
> +++ b/xen/tools/cppcheck-cc.sh
> @@ -0,0 +1,223 @@
> +#!/usr/bin/env bash
> +
> +set -e
> +
> +function help() {
> + cat <<EOF
> +Usage: ${0} [OPTION] ... -- <compiler arguments>
> +
> +This script is a wrapper for cppcheck that enables it to analyse the files that
> +are the target for the build, it is used in place of a selected compiler and the
> +make process will run it on every file that needs to be built.
> +All the arguments passed to the original compiler are forwarded to it without
> +modification, furthermore, they are used to improve the cppcheck analysis.
> +
> +Options:
> + --compiler= Use this compiler for the build
> + --cppcheck-cmd= Command line for the cppcheck analysis.
> + --cppcheck-html Prepare for cppcheck HTML output
> + --cppcheck-plat= Path to the cppcheck platform folder
> + --ignore-path= This script won't run cppcheck on the files having this
> + path, the compiler will run anyway on them. This argument
> + can be specified multiple times.
> + -h, --help Print this help
> +EOF
> +}
> +
> +CC_FILE=""
> +COMPILER=""
> +CPPCHECK_HTML="n"
> +CPPCHECK_PLAT_PATH=""
> +CPPCHECK_TOOL=""
> +CPPCHECK_TOOL_ARGS=""
> +FORWARD_FLAGS=""
> +IGNORE_PATH="n"
> +IGNORE_PATH_LIST=""
> +JDB_FILE=""
> +OBJTREE_PATH=""
> +
> +# Variable used for arg parsing
> +forward_to_cc="n"
> +sm_tool_args="n"
> +obj_arg_content="n"
> +
> +for OPTION in "$@"
> +do
> + if [ "${forward_to_cc}" = "y" ]; then
> + if [[ ${OPTION} == *.c ]]
> + then
> + CC_FILE="${OPTION}"
> + elif [ "${OPTION}" = "-o" ]
> + then
> + # After -o there is the path to the obj file, flag it
> + obj_arg_content="y"
> + elif [ "${obj_arg_content}" = "y" ]
> + then
> + # This must be the path to the obj file, turn off flag and save path
> + OBJTREE_PATH="$(dirname "${OPTION}")"
> + obj_arg_content="n"
> + fi
> + # Forward any argument to the compiler
> + FORWARD_FLAGS="${FORWARD_FLAGS} ${OPTION}"
> + continue
> + fi
> + case ${OPTION} in
> + -h|--help)
> + help
> + exit 0
> + ;;
> + --compiler=*)
> + COMPILER="$(eval echo "${OPTION#*=}")"
This can be:
COMPILER="${OPTION#*=}"
and same for all the other below
> + sm_tool_args="n"
> + ;;
> + --cppcheck-cmd=*)
> + CPPCHECK_TOOL="$(eval echo "${OPTION#*=}")"
> + sm_tool_args="y"
> + ;;
> + --cppcheck-html)
> + CPPCHECK_HTML="y"
> + sm_tool_args="n"
> + ;;
> + --cppcheck-plat=*)
> + CPPCHECK_PLAT_PATH="$(eval echo "${OPTION#*=}")"
> + sm_tool_args="n"
> + ;;
> + --ignore-path=*)
> + IGNORE_PATH_LIST="${IGNORE_PATH_LIST} $(eval echo "${OPTION#*=}")"
> + sm_tool_args="n"
> + ;;
> + --)
> + forward_to_cc="y"
> + sm_tool_args="n"
> + ;;
> + *)
> + if [ "${sm_tool_args}" = "y" ]; then
> + CPPCHECK_TOOL_ARGS="${CPPCHECK_TOOL_ARGS} ${OPTION}"
> + else
> + echo "Invalid option ${OPTION}"
> + exit 1
It doesn't look like sm_tool_args is really needed? It is only set to
'y' in the case of --cppcheck-cmd, and in that case we also set
CPPCHECK_TOOL. CPPCHECK_TOOL is the variable used below. Am I missing
something?
> + fi
> + ;;
> + esac
> +done
> +
> +if [ "${COMPILER}" = "" ]
> +then
> + echo "--compiler arg is mandatory."
> + exit 1
> +fi
> +
> +function print_file() {
> + local text="${1}"
> + local init_file="${2}"
> +
> + if [ "${init_file}" = "y" ]
> + then
> + echo -e -n "${text}" > "${JDB_FILE}"
> + else
> + echo -e -n "${text}" >> "${JDB_FILE}"
> + fi
The >> can be used to create a file if the file is not already present.
So why the need for this if? In fact, we don't need print_file at all
and we can just
echo -e -n "something" >> "${JDB_FILE}"
directly from create_jcd. If you are concerned about a preexisting file,
then at the beginning of create_jcd you can:
rm "${JDB_FILE}"
> +}
> +
> +function create_jcd() {
> + local line="${1}"
> + local arg_num=0
> + local same_line=0
> +
> + print_file "[\n" "y"
> + print_file " {\n"
> + print_file " \"arguments\": [\n"
> +
> + for arg in ${line}; do
> + # This code prevents to put comma in the last element of the list or on
> + # sequential lines that are going to be merged
> + [ "${arg_num}" -ne 0 ] && [ "${same_line}" -eq 0 ] && print_file ",\n"
> + if [ "${same_line}" -ne 0 ]
> + then
> + print_file "${arg}\""
> + same_line=0
> + elif [ "${arg}" = "-iquote" ] || [ "${arg}" = "-I" ]
> + then
> + # cppcheck doesn't understand -iquote, substitute with -I
> + print_file " \"-I"
> + same_line=1
> + else
> + print_file " \"${arg}\""
> + fi
> + arg_num=$(( arg_num + 1 ))
> + done
> + print_file "\n"
> + print_file " ],\n"
> + print_file " \"directory\": \"$(pwd -P)\",\n"
> + print_file " \"file\": \"${CC_FILE}\"\n"
> + print_file " }\n"
> + print_file "]\n"
> +}
> +
> +
> +# Execute compiler with forwarded flags
> +# Shellcheck complains about missing quotes on FORWARD_FLAGS, but they can't be
> +# used here
> +# shellcheck disable=SC2086
> +${COMPILER} ${FORWARD_FLAGS}
> +
> +if [ -n "${CC_FILE}" ];
> +then
> + for path in ${IGNORE_PATH_LIST}
> + do
> + if [[ ${CC_FILE} == *${path}* ]]
> + then
> + IGNORE_PATH="y"
> + echo "${0}: ${CC_FILE} ignored by --ignore-path matching *${path}*"
> + fi
> + done
> + if [ "${IGNORE_PATH}" = "n" ]
> + then
> + JDB_FILE="${OBJTREE_PATH}/$(basename "${CC_FILE}".json)"
> +
> + # Prepare the Json Compilation Database for the file
> + create_jcd "${COMPILER} ${FORWARD_FLAGS}"
> +
> + out_file="${OBJTREE_PATH}/$(basename "${CC_FILE%.c}".cppcheck.txt)"
> +
> + # Check wchar size
> + wchar_plat_suffix="t4"
> + # sed prints the last occurence of -f(no-)short-wchar which is the one
> + # applied to the file by the compiler
> + wchar_option=$(echo "${FORWARD_FLAGS}" | \
> + sed -nre 's,.*(-f(no-)?short-wchar).*,\1,p')
> + if [ "${wchar_option}" = "-fshort-wchar" ]
> + then
> + wchar_plat_suffix="t2"
> + fi
This seems a bit unnecessary: we should be able to find the right
platform file from XEN_TARGET_ARCH alone. No need to reverse engineer
the compiler command line?
> +
> + # Select the right target platform, ARCH is generated from Xen Makefile
> + platform="${CPPCHECK_PLAT_PATH}/${ARCH}-wchar_${wchar_plat_suffix}.xml"
> + if [ ! -f "${platform}" ]
> + then
> + echo "${platform} not found!"
> + exit 1
> + fi
> +
> + # Shellcheck complains about missing quotes on CPPCHECK_TOOL_ARGS, but
> + # they can't be used here
> + # shellcheck disable=SC2086
> + ${CPPCHECK_TOOL} ${CPPCHECK_TOOL_ARGS} \
> + --project="${JDB_FILE}" \
> + --output-file="${out_file}" \
> + --platform=${platform}
> +
> + if [ "${CPPCHECK_HTML}" = "y" ]
> + then
> + # Shellcheck complains about missing quotes on CPPCHECK_TOOL_ARGS,
> + # but they can't be used here
> + # shellcheck disable=SC2086
> + ${CPPCHECK_TOOL} ${CPPCHECK_TOOL_ARGS} \
> + --project="${JDB_FILE}" \
> + --output-file="${out_file%.txt}.xml" \
> + --platform=${platform} \
> + -q \
> + --xml
This is showing my ignorance in cppcheck, but does it actually need to
be called twice in the html generation case? Actually three times if we
count the extra cppcheck-htmlreport call?
> + fi
> + fi
> +fi
> diff --git a/xen/tools/cppcheck-plat/arm32-wchar_t4.xml b/xen/tools/cppcheck-plat/arm32-wchar_t4.xml
> new file mode 100644
> index 000000000000..3aefa7ba5c98
> --- /dev/null
> +++ b/xen/tools/cppcheck-plat/arm32-wchar_t4.xml
> @@ -0,0 +1,17 @@
> +<?xml version="1.0"?>
> +<platform>
> + <char_bit>8</char_bit>
> + <default-sign>unsigned</default-sign>
usually in C the default is actually "signed" not "unsigned". If you
write:
int i;
i is signed
> + <sizeof>
> + <short>2</short>
> + <int>4</int>
> + <long>4</long>
> + <long-long>8</long-long>
> + <float>4</float>
> + <double>8</double>
> + <long-double>8</long-double>
> + <pointer>4</pointer>
> + <size_t>4</size_t>
> + <wchar_t>4</wchar_t>
> + </sizeof>
> +</platform>
> diff --git a/xen/tools/cppcheck-plat/arm64-wchar_t2.xml b/xen/tools/cppcheck-plat/arm64-wchar_t2.xml
> new file mode 100644
> index 000000000000..e345b934a986
> --- /dev/null
> +++ b/xen/tools/cppcheck-plat/arm64-wchar_t2.xml
> @@ -0,0 +1,17 @@
> +<?xml version="1.0"?>
> +<platform>
> + <char_bit>8</char_bit>
> + <default-sign>unsigned</default-sign>
> + <sizeof>
> + <short>2</short>
> + <int>4</int>
> + <long>8</long>
> + <long-long>8</long-long>
> + <float>4</float>
> + <double>8</double>
> + <long-double>16</long-double>
> + <pointer>8</pointer>
> + <size_t>4</size_t>
Isn't size_t 8 bytes on arm64?
> + <wchar_t>2</wchar_t>
> + </sizeof>
> +</platform>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 3/4] tools/misra: fix skipped rule numbers
2022-11-29 23:51 ` Stefano Stabellini
@ 2022-11-30 8:53 ` Luca Fancellu
2022-11-30 23:34 ` Stefano Stabellini
0 siblings, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-11-30 8:53 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Bertrand Marquis, Wei Chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Wei Liu
> On 29 Nov 2022, at 23:51, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Mon, 28 Nov 2022, Luca Fancellu wrote:
>> Currently the script convert_misra_doc.py is using a loop through
>> range(1,22) to enumerate rules that needs to be skipped, however
>> range function does not include the stop counter in the enumeration
>> ending up into list rules until 21.21 instead of including rule 22.
>>
>> Fix the issue using a dictionary that list the rules in misra c2012.
>
> I think I understand the problem you are trying to solve with this
> patch. But I am confused about the proposed solution.
>
> The original code is trying to list all the possible MISRA C rules that
> are not in docs/misra/rules.rst. Instead of list(range(1,22)) now we
> have a dictionary: misra_c2012_rules. But misra_c2012_rules doesn't have
> all the possible MISRA C rules missing from docs/misra/rules.rst.
>
> As an example Rule 13.1 is missing from docs/misra/rules.rst but it is
> also missing from misra_c2012_rules.
>
> Can you please help me understand why misra_c2012_rules has only a small
> subset of MISRA C rules to be skipped?
Hi Stefano,
MISRA rules are in this format X.Y, misra_c2012_rules is a dictionary where the key is
X and the value is the maximum number that Y can have.
For example rule 13.Y goes from 13.1 to 13.6 (in the dictionary misra_c2012_rules[13] == 6),
so the code can now check which among (13.1 .. 13.6) is not in the rule_list and add it to the
list of skipped rules.
Here an example:
{
"script": "misra.py",
"args": [
"--rule-texts=/path/to/cppcheck-misra.txt",
"--suppress-rules=1.1,1.2,1.4,2.2,2.3,2.4,2.5,2.6,2.7,3.1,4.1,4.2,5.5,5.6,5.7,5.8,5.9,6.1,7.1,7.2,7.3,7.4,8.2,8.3,8.7,8.9,8.11,8.13,8.14,9.3,9.4,9.5,10.1,10.2,10.3,10.4,10.5,10.6,10.7,10.8,11.1,11.2,11.3,11.4,11.5,11.6,11.7,11.8,11.9,12.1,12.2,12.3,12.4,12.5,13.1,13.2,13.3,13.4,13.5,14.2,14.3,14.4,15.1,15.2,15.3,15.4,15.5,15.6,15.7,16.1,16.2,16.3,16.4,16.5,16.6,17.1,17.2,17.5,17.6,17.7,17.8,18.1,18.2,18.3,18.4,18.5,18.6,18.7,18.8,19.1,19.2,20.1,20.2,20.3,20.4,20.5,20.6,20.8,20.9,20.10,20.11,20.12,21.1,21.2,21.3,21.4,21.5,21.6,21.7,21.8,21.9,21.10,21.11,21.12,21.13,21.14,21.15,21.16,21.17,21.18,21.19,21.20,21.21,22.1,22.2,22.3,22.4,22.5,22.6,22.7,22.8,22.9,22.10"
]
}
So this patch is solving two issues, the first one was that rule 22.Y was never included in the suppressed
list because range(1,22) produces a range in [1..21], the second issue is that the code was producing
Invalid MISRA C 2012 rules, for example 1.21 and so on.
>
>
>> Fixes: 57caa5375321 ("xen: Add MISRA support to cppcheck make rule")
>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>> ---
>> xen/tools/convert_misra_doc.py | 32 ++++++++++++++++++++++++++++++--
>> 1 file changed, 30 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/tools/convert_misra_doc.py b/xen/tools/convert_misra_doc.py
>> index caa4487f645f..13074d8a2e91 100755
>> --- a/xen/tools/convert_misra_doc.py
>> +++ b/xen/tools/convert_misra_doc.py
>> @@ -14,6 +14,34 @@ Usage:
>>
>> import sys, getopt, re
>>
>> +# MISRA rule are identified by two numbers, e.g. Rule 1.2, the main rule number
>> +# and a sub-number. This dictionary contains the number of the MISRA rule as key
>> +# and the maximum sub-number for that rule as value.
>> +misra_c2012_rules = {
>> + 1:4,
>> + 2:7,
>> + 3:2,
>> + 4:2,
>> + 5:9,
>> + 6:2,
>> + 7:4,
>> + 8:14,
>> + 9:5,
>> + 10:8,
>> + 11:9,
>> + 12:5,
>> + 13:6,
>> + 14:4,
>> + 15:7,
>> + 16:7,
>> + 17:8,
>> + 18:8,
>> + 19:2,
>> + 20:14,
>> + 21:21,
>> + 22:10
>> +}
>> +
>> def main(argv):
>> infile = ''
>> outfile = ''
>> @@ -142,8 +170,8 @@ def main(argv):
>> skip_list = []
>>
>> # Search for missing rules and add a dummy text with the rule number
>> - for i in list(range(1,22)):
>> - for j in list(range(1,22)):
>> + for i in misra_c2012_rules:
>> + for j in list(range(1,misra_c2012_rules[i]+1)):
>> if str(i) + '.' + str(j) not in rule_list:
>> outstr.write('Rule ' + str(i) + '.' + str(j) + '\n')
>> outstr.write('No description for rule ' + str(i) + '.' + str(j)
>> --
>> 2.17.1
>>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-30 1:05 ` Stefano Stabellini
@ 2022-11-30 11:59 ` Luca Fancellu
2022-11-30 20:26 ` Stefano Stabellini
0 siblings, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-11-30 11:59 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Bertrand Marquis, Wei Chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Wei Liu
Hi Stefano,
> I think the revert of the cppcheck integration in xen/Makefile and
> xen/tools/merge_cppcheck_reports.py could be a separate patch. There is
> no need to make sure cppcheck support in the xen Makefile is
> "bisectable". That patch could have my acked-by already.
Ok I will split these changes in a following patch
>
> Also the document changes introduced in this patch have my reviewed-by:
> - docs/misra/cppcheck.txt
> - docs/misra/documenting-violations.rst
> - docs/misra/false-positive-cppcheck.json
> - docs/misra/xen-static-analysis.rst
Thank you, should I put those files in a separate patch with your rev-by before
this patch or this is just a comment for you to remember which file you already
reviewed?
>>
>> +
>> +def generate_cppcheck_deps():
>> + global cppcheck_extra_make_args
>> +
>> + # Compile flags to pass to cppcheck:
>> + # - include config.h as this is passed directly to the compiler.
>> + # - define CPPCHECK as we use it to disable or enable some specific part of
>> + # the code to solve some cppcheck issues.
>> + # - explicitely enable some cppcheck checks as we do not want to use "all"
>> + # which includes unusedFunction which gives wrong positives as we check
>> + # file per file.
>> + # - Explicitly suppress warnings on compiler-def.h because cppcheck throws
>> + # an unmatchedSuppression due to the rule we put in suppression-list.txt
>> + # to skip every finding in the file.
>> + #
>> + # Compiler defines are in compiler-def.h which is included in config.h
>> + #
>> + cppcheck_flags="""
>> +--cppcheck-build-dir={}/{}
>> + --max-ctu-depth=10
>> + --enable=style,information,missingInclude
>> + --template=\'{{file}}({{line}},{{column}}):{{id}}:{{severity}}:{{message}}\'
>> + --relative-paths={}
>> + --inline-suppr
>> + --suppressions-list={}/suppression-list.txt
>> + --suppress='unmatchedSuppression:*generated/compiler-def.h'
>> + --include={}/include/xen/config.h
>
> I noticed that some of the includes we used to have like
> xsm/flask/include are missing here. Is that intended?
Yes it is, now that cppcheck is using the JSON compilation database, it can understand
by the compilation argument “-I” what include path it needs to add, before we were
adding it to every file, resulting in some false positive from the tool.
Just --include={}/include/xen/config.h is needed because in the Xen makefile we are doing
the same, passing the option to the compiler, resulting in every compiled file to have that
header included.
>>
>> + case ${OPTION} in
>> + -h|--help)
>> + help
>> + exit 0
>> + ;;
>> + --compiler=*)
>> + COMPILER="$(eval echo "${OPTION#*=}")"
>
> This can be:
>
> COMPILER="${OPTION#*=}"
>
> and same for all the other below
Ok I’ll fix that
>
>
>> + sm_tool_args="n"
>> + ;;
>> + --cppcheck-cmd=*)
>> + CPPCHECK_TOOL="$(eval echo "${OPTION#*=}")"
>> + sm_tool_args="y"
>> + ;;
>> + --cppcheck-html)
>> + CPPCHECK_HTML="y"
>> + sm_tool_args="n"
>> + ;;
>> + --cppcheck-plat=*)
>> + CPPCHECK_PLAT_PATH="$(eval echo "${OPTION#*=}")"
>> + sm_tool_args="n"
>> + ;;
>> + --ignore-path=*)
>> + IGNORE_PATH_LIST="${IGNORE_PATH_LIST} $(eval echo "${OPTION#*=}")"
>> + sm_tool_args="n"
>> + ;;
>> + --)
>> + forward_to_cc="y"
>> + sm_tool_args="n"
>> + ;;
>> + *)
>> + if [ "${sm_tool_args}" = "y" ]; then
>> + CPPCHECK_TOOL_ARGS="${CPPCHECK_TOOL_ARGS} ${OPTION}"
>> + else
>> + echo "Invalid option ${OPTION}"
>> + exit 1
>
> It doesn't look like sm_tool_args is really needed? It is only set to
> 'y' in the case of --cppcheck-cmd, and in that case we also set
> CPPCHECK_TOOL. CPPCHECK_TOOL is the variable used below. Am I missing
> something?
We use sm_tool_args to fill CPPCHECK_TOOL_ARGS, basically it’s a state machine where
when we find --cppcheck-cmd=<xxx> we expect that every other space separated arguments
passed afterwards are the args for cppcheck, so we append to CPPCHECK_TOOL_ARGS
until we find an argument that is supposed to be only for this script.
>
>
>> + fi
>> + ;;
>> + esac
>> +done
>> +
>> +if [ "${COMPILER}" = "" ]
>> +then
>> + echo "--compiler arg is mandatory."
>> + exit 1
>> +fi
>> +
>> +function print_file() {
>> + local text="${1}"
>> + local init_file="${2}"
>> +
>> + if [ "${init_file}" = "y" ]
>> + then
>> + echo -e -n "${text}" > "${JDB_FILE}"
>> + else
>> + echo -e -n "${text}" >> "${JDB_FILE}"
>> + fi
>
> The >> can be used to create a file if the file is not already present.
> So why the need for this if? In fact, we don't need print_file at all
> and we can just
>
> echo -e -n "something" >> "${JDB_FILE}"
>
> directly from create_jcd. If you are concerned about a preexisting file,
> then at the beginning of create_jcd you can:
>
> rm "${JDB_FILE}"
Ok I’ll remove the file in the top of create_jcd and use echo -e -n "something" >> "${JDB_FILE}”
>>
>> +
>> + # Check wchar size
>> + wchar_plat_suffix="t4"
>> + # sed prints the last occurence of -f(no-)short-wchar which is the one
>> + # applied to the file by the compiler
>> + wchar_option=$(echo "${FORWARD_FLAGS}" | \
>> + sed -nre 's,.*(-f(no-)?short-wchar).*,\1,p')
>> + if [ "${wchar_option}" = "-fshort-wchar" ]
>> + then
>> + wchar_plat_suffix="t2"
>> + fi
>
> This seems a bit unnecessary: we should be able to find the right
> platform file from XEN_TARGET_ARCH alone. No need to reverse engineer
> the compiler command line?
The efi code is compiled with -fshort-wchar, but the rest of the file uses default length wchar,
now maybe it was a bit of overthinking because I guess we have only these cases:
arm64: arm64-wchar_t2 (efi code uses -fshort-wchar)
arm32: arm32-wchar_t4 (efi code is not in, but common-stub compiled with -f-no-short-wchar)
x86_64: x86_64-wchar_t2 (efi code uses -fshort-wchar)
Am I right?
>
>
>> +
>> + # Select the right target platform, ARCH is generated from Xen Makefile
>> + platform="${CPPCHECK_PLAT_PATH}/${ARCH}-wchar_${wchar_plat_suffix}.xml"
>> + if [ ! -f "${platform}" ]
>> + then
>> + echo "${platform} not found!"
>> + exit 1
>> + fi
>> +
>> + # Shellcheck complains about missing quotes on CPPCHECK_TOOL_ARGS, but
>> + # they can't be used here
>> + # shellcheck disable=SC2086
>> + ${CPPCHECK_TOOL} ${CPPCHECK_TOOL_ARGS} \
>> + --project="${JDB_FILE}" \
>> + --output-file="${out_file}" \
>> + --platform=${platform}
>> +
>> + if [ "${CPPCHECK_HTML}" = "y" ]
>> + then
>> + # Shellcheck complains about missing quotes on CPPCHECK_TOOL_ARGS,
>> + # but they can't be used here
>> + # shellcheck disable=SC2086
>> + ${CPPCHECK_TOOL} ${CPPCHECK_TOOL_ARGS} \
>> + --project="${JDB_FILE}" \
>> + --output-file="${out_file%.txt}.xml" \
>> + --platform=${platform} \
>> + -q \
>> + --xml
>
> This is showing my ignorance in cppcheck, but does it actually need to
> be called twice in the html generation case? Actually three times if we
> count the extra cppcheck-htmlreport call?
Cppcheck is not able to output a text report and an XML report at the same time,
hence we need to call it twice, but the second call will use the cppcheck build directory
As a “cache” to generate the results so it will be much more faster than the first one.
>
>
>> + fi
>> + fi
>> +fi
>> diff --git a/xen/tools/cppcheck-plat/arm32-wchar_t4.xml b/xen/tools/cppcheck-plat/arm32-wchar_t4.xml
>> new file mode 100644
>> index 000000000000..3aefa7ba5c98
>> --- /dev/null
>> +++ b/xen/tools/cppcheck-plat/arm32-wchar_t4.xml
>> @@ -0,0 +1,17 @@
>> +<?xml version="1.0"?>
>> +<platform>
>> + <char_bit>8</char_bit>
>> + <default-sign>unsigned</default-sign>
>
> usually in C the default is actually "signed" not "unsigned". If you
> write:
>
> int i;
>
> i is signed
It took me a bit to understand this field, as the documentation is not clear at all, the default-sign is referring
to the default char sign, which should be unsigned for arm, right?
Here the code to cppcheck that clarifies the field:
https://github.com/danmar/cppcheck/blob/2.7.5/lib/platform.cpp
At line 204, defaultSign is taking the value of <default-sign>, at line 64, when the platform is Native,
defaultSign = (std::numeric_limits<char>::is_signed) ? 's' : 'u';
I’ve done some tests with this code in arm/arm64/x86_64:
#define is_type_signed(my_type) (((my_type)-1) < 0)
if (is_type_signed(char))
printf("signed\n");
else
printf("unsigned\n");
And I have unsigned for arm/arm64 and signed for x86_64 (which I will change as it is wrong in this patch)
Can you confirm my results are right?
>>
>> +++ b/xen/tools/cppcheck-plat/arm64-wchar_t2.xml
>> @@ -0,0 +1,17 @@
>> +<?xml version="1.0"?>
>> +<platform>
>> + <char_bit>8</char_bit>
>> + <default-sign>unsigned</default-sign>
>> + <sizeof>
>> + <short>2</short>
>> + <int>4</int>
>> + <long>8</long>
>> + <long-long>8</long-long>
>> + <float>4</float>
>> + <double>8</double>
>> + <long-double>16</long-double>
>> + <pointer>8</pointer>
>> + <size_t>4</size_t>
>
> Isn't size_t 8 bytes on arm64?
Yes you are right, I will fix it
>
>
>> + <wchar_t>2</wchar_t>
>> + </sizeof>
>> +</platform>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-30 11:59 ` Luca Fancellu
@ 2022-11-30 20:26 ` Stefano Stabellini
2022-12-01 8:33 ` Jan Beulich
2022-12-01 15:34 ` Luca Fancellu
0 siblings, 2 replies; 31+ messages in thread
From: Stefano Stabellini @ 2022-11-30 20:26 UTC (permalink / raw)
To: Luca Fancellu
Cc: Stefano Stabellini, Xen-devel, Bertrand Marquis, Wei Chen,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall, Wei Liu
[-- Attachment #1: Type: text/plain, Size: 10766 bytes --]
On Wed, 30 Nov 2022, Luca Fancellu wrote:
> Hi Stefano,
>
> > I think the revert of the cppcheck integration in xen/Makefile and
> > xen/tools/merge_cppcheck_reports.py could be a separate patch. There is
> > no need to make sure cppcheck support in the xen Makefile is
> > "bisectable". That patch could have my acked-by already.
>
> Ok I will split these changes in a following patch
>
> >
> > Also the document changes introduced in this patch have my reviewed-by:
> > - docs/misra/cppcheck.txt
> > - docs/misra/documenting-violations.rst
> > - docs/misra/false-positive-cppcheck.json
> > - docs/misra/xen-static-analysis.rst
>
> Thank you, should I put those files in a separate patch with your rev-by before
> this patch or this is just a comment for you to remember which file you already
> reviewed?
If Jan and the other reviewers are OK, I think you could split them out
in a separate patch and add my reviewed-by. If Jan prefers to keep it
all together in one patch, then I wrote it down so that I remember what
I have already acked :-)
> >> +
> >> +def generate_cppcheck_deps():
> >> + global cppcheck_extra_make_args
> >> +
> >> + # Compile flags to pass to cppcheck:
> >> + # - include config.h as this is passed directly to the compiler.
> >> + # - define CPPCHECK as we use it to disable or enable some specific part of
> >> + # the code to solve some cppcheck issues.
> >> + # - explicitely enable some cppcheck checks as we do not want to use "all"
> >> + # which includes unusedFunction which gives wrong positives as we check
> >> + # file per file.
> >> + # - Explicitly suppress warnings on compiler-def.h because cppcheck throws
> >> + # an unmatchedSuppression due to the rule we put in suppression-list.txt
> >> + # to skip every finding in the file.
> >> + #
> >> + # Compiler defines are in compiler-def.h which is included in config.h
> >> + #
> >> + cppcheck_flags="""
> >> +--cppcheck-build-dir={}/{}
> >> + --max-ctu-depth=10
> >> + --enable=style,information,missingInclude
> >> + --template=\'{{file}}({{line}},{{column}}):{{id}}:{{severity}}:{{message}}\'
> >> + --relative-paths={}
> >> + --inline-suppr
> >> + --suppressions-list={}/suppression-list.txt
> >> + --suppress='unmatchedSuppression:*generated/compiler-def.h'
> >> + --include={}/include/xen/config.h
> >
> > I noticed that some of the includes we used to have like
> > xsm/flask/include are missing here. Is that intended?
>
> Yes it is, now that cppcheck is using the JSON compilation database, it can understand
> by the compilation argument “-I” what include path it needs to add, before we were
> adding it to every file, resulting in some false positive from the tool.
> Just --include={}/include/xen/config.h is needed because in the Xen makefile we are doing
> the same, passing the option to the compiler, resulting in every compiled file to have that
> header included.
OK, good to hear the process is improving
> >>
> >> + case ${OPTION} in
> >> + -h|--help)
> >> + help
> >> + exit 0
> >> + ;;
> >> + --compiler=*)
> >> + COMPILER="$(eval echo "${OPTION#*=}")"
> >
> > This can be:
> >
> > COMPILER="${OPTION#*=}"
> >
> > and same for all the other below
>
> Ok I’ll fix that
>
> >
> >
> >> + sm_tool_args="n"
> >> + ;;
> >> + --cppcheck-cmd=*)
> >> + CPPCHECK_TOOL="$(eval echo "${OPTION#*=}")"
> >> + sm_tool_args="y"
> >> + ;;
> >> + --cppcheck-html)
> >> + CPPCHECK_HTML="y"
> >> + sm_tool_args="n"
> >> + ;;
> >> + --cppcheck-plat=*)
> >> + CPPCHECK_PLAT_PATH="$(eval echo "${OPTION#*=}")"
> >> + sm_tool_args="n"
> >> + ;;
> >> + --ignore-path=*)
> >> + IGNORE_PATH_LIST="${IGNORE_PATH_LIST} $(eval echo "${OPTION#*=}")"
> >> + sm_tool_args="n"
> >> + ;;
> >> + --)
> >> + forward_to_cc="y"
> >> + sm_tool_args="n"
> >> + ;;
> >> + *)
> >> + if [ "${sm_tool_args}" = "y" ]; then
> >> + CPPCHECK_TOOL_ARGS="${CPPCHECK_TOOL_ARGS} ${OPTION}"
> >> + else
> >> + echo "Invalid option ${OPTION}"
> >> + exit 1
> >
> > It doesn't look like sm_tool_args is really needed? It is only set to
> > 'y' in the case of --cppcheck-cmd, and in that case we also set
> > CPPCHECK_TOOL. CPPCHECK_TOOL is the variable used below. Am I missing
> > something?
>
> We use sm_tool_args to fill CPPCHECK_TOOL_ARGS, basically it’s a state machine where
> when we find --cppcheck-cmd=<xxx> we expect that every other space separated arguments
> passed afterwards are the args for cppcheck, so we append to CPPCHECK_TOOL_ARGS
> until we find an argument that is supposed to be only for this script.
That seems a bit unnecessary: if the user wants to pass arguments to
cppcheck, the user would do --cppcheck-cmd="cppcheck arg1 arg2" with ""
quotes. Doing that should make --cppcheck-cmd="cppcheck arg1 arg2" be
seen as a single argument from this script point of view. CPPCHECK_TOOL
would end up being set to "cppcheck arg1 arg2" which is what we want
anyway? And if we need to distinguish between the cppcheck binary and
its argument we could use "cut" to extract "cppcheck", "arg1", and
"arg2" from CPPCHECK_TOOL. Would that work?
> >
> >> + fi
> >> + ;;
> >> + esac
> >> +done
> >> +
> >> +if [ "${COMPILER}" = "" ]
> >> +then
> >> + echo "--compiler arg is mandatory."
> >> + exit 1
> >> +fi
> >> +
> >> +function print_file() {
> >> + local text="${1}"
> >> + local init_file="${2}"
> >> +
> >> + if [ "${init_file}" = "y" ]
> >> + then
> >> + echo -e -n "${text}" > "${JDB_FILE}"
> >> + else
> >> + echo -e -n "${text}" >> "${JDB_FILE}"
> >> + fi
> >
> > The >> can be used to create a file if the file is not already present.
> > So why the need for this if? In fact, we don't need print_file at all
> > and we can just
> >
> > echo -e -n "something" >> "${JDB_FILE}"
> >
> > directly from create_jcd. If you are concerned about a preexisting file,
> > then at the beginning of create_jcd you can:
> >
> > rm "${JDB_FILE}"
>
> Ok I’ll remove the file in the top of create_jcd and use echo -e -n "something" >> "${JDB_FILE}”
>
> >>
> >> +
> >> + # Check wchar size
> >> + wchar_plat_suffix="t4"
> >> + # sed prints the last occurence of -f(no-)short-wchar which is the one
> >> + # applied to the file by the compiler
> >> + wchar_option=$(echo "${FORWARD_FLAGS}" | \
> >> + sed -nre 's,.*(-f(no-)?short-wchar).*,\1,p')
> >> + if [ "${wchar_option}" = "-fshort-wchar" ]
> >> + then
> >> + wchar_plat_suffix="t2"
> >> + fi
> >
> > This seems a bit unnecessary: we should be able to find the right
> > platform file from XEN_TARGET_ARCH alone. No need to reverse engineer
> > the compiler command line?
>
> The efi code is compiled with -fshort-wchar, but the rest of the file uses default length wchar,
> now maybe it was a bit of overthinking because I guess we have only these cases:
>
> arm64: arm64-wchar_t2 (efi code uses -fshort-wchar)
> arm32: arm32-wchar_t4 (efi code is not in, but common-stub compiled with -f-no-short-wchar)
> x86_64: x86_64-wchar_t2 (efi code uses -fshort-wchar)
>
> Am I right?
Yes I think so too
> >
> >> +
> >> + # Select the right target platform, ARCH is generated from Xen Makefile
> >> + platform="${CPPCHECK_PLAT_PATH}/${ARCH}-wchar_${wchar_plat_suffix}.xml"
> >> + if [ ! -f "${platform}" ]
> >> + then
> >> + echo "${platform} not found!"
> >> + exit 1
> >> + fi
> >> +
> >> + # Shellcheck complains about missing quotes on CPPCHECK_TOOL_ARGS, but
> >> + # they can't be used here
> >> + # shellcheck disable=SC2086
> >> + ${CPPCHECK_TOOL} ${CPPCHECK_TOOL_ARGS} \
> >> + --project="${JDB_FILE}" \
> >> + --output-file="${out_file}" \
> >> + --platform=${platform}
> >> +
> >> + if [ "${CPPCHECK_HTML}" = "y" ]
> >> + then
> >> + # Shellcheck complains about missing quotes on CPPCHECK_TOOL_ARGS,
> >> + # but they can't be used here
> >> + # shellcheck disable=SC2086
> >> + ${CPPCHECK_TOOL} ${CPPCHECK_TOOL_ARGS} \
> >> + --project="${JDB_FILE}" \
> >> + --output-file="${out_file%.txt}.xml" \
> >> + --platform=${platform} \
> >> + -q \
> >> + --xml
> >
> > This is showing my ignorance in cppcheck, but does it actually need to
> > be called twice in the html generation case? Actually three times if we
> > count the extra cppcheck-htmlreport call?
>
> Cppcheck is not able to output a text report and an XML report at the same time,
> hence we need to call it twice, but the second call will use the cppcheck build directory
> As a “cache” to generate the results so it will be much more faster than the first one.
OK
> >
> >> + fi
> >> + fi
> >> +fi
> >> diff --git a/xen/tools/cppcheck-plat/arm32-wchar_t4.xml b/xen/tools/cppcheck-plat/arm32-wchar_t4.xml
> >> new file mode 100644
> >> index 000000000000..3aefa7ba5c98
> >> --- /dev/null
> >> +++ b/xen/tools/cppcheck-plat/arm32-wchar_t4.xml
> >> @@ -0,0 +1,17 @@
> >> +<?xml version="1.0"?>
> >> +<platform>
> >> + <char_bit>8</char_bit>
> >> + <default-sign>unsigned</default-sign>
> >
> > usually in C the default is actually "signed" not "unsigned". If you
> > write:
> >
> > int i;
> >
> > i is signed
>
> It took me a bit to understand this field, as the documentation is not clear at all, the default-sign is referring
> to the default char sign, which should be unsigned for arm, right?
OK
> Here the code to cppcheck that clarifies the field:
>
> https://github.com/danmar/cppcheck/blob/2.7.5/lib/platform.cpp
>
> At line 204, defaultSign is taking the value of <default-sign>, at line 64, when the platform is Native,
> defaultSign = (std::numeric_limits<char>::is_signed) ? 's' : 'u';
>
> I’ve done some tests with this code in arm/arm64/x86_64:
>
> #define is_type_signed(my_type) (((my_type)-1) < 0)
> if (is_type_signed(char))
> printf("signed\n");
> else
> printf("unsigned\n");
>
> And I have unsigned for arm/arm64 and signed for x86_64 (which I will change as it is wrong in this patch)
>
> Can you confirm my results are right?
It looks like this is compiler specific. Yes, surprisingly with gcc I
got the same results as you.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 3/4] tools/misra: fix skipped rule numbers
2022-11-30 8:53 ` Luca Fancellu
@ 2022-11-30 23:34 ` Stefano Stabellini
2022-12-01 11:27 ` Luca Fancellu
0 siblings, 1 reply; 31+ messages in thread
From: Stefano Stabellini @ 2022-11-30 23:34 UTC (permalink / raw)
To: Luca Fancellu
Cc: Stefano Stabellini, Xen-devel, Bertrand Marquis, Wei Chen,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall, Wei Liu
On Wed, 30 Nov 2022, Luca Fancellu wrote:
> > On 29 Nov 2022, at 23:51, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >
> > On Mon, 28 Nov 2022, Luca Fancellu wrote:
> >> Currently the script convert_misra_doc.py is using a loop through
> >> range(1,22) to enumerate rules that needs to be skipped, however
> >> range function does not include the stop counter in the enumeration
> >> ending up into list rules until 21.21 instead of including rule 22.
> >>
> >> Fix the issue using a dictionary that list the rules in misra c2012.
> >
> > I think I understand the problem you are trying to solve with this
> > patch. But I am confused about the proposed solution.
> >
> > The original code is trying to list all the possible MISRA C rules that
> > are not in docs/misra/rules.rst. Instead of list(range(1,22)) now we
> > have a dictionary: misra_c2012_rules. But misra_c2012_rules doesn't have
> > all the possible MISRA C rules missing from docs/misra/rules.rst.
> >
> > As an example Rule 13.1 is missing from docs/misra/rules.rst but it is
> > also missing from misra_c2012_rules.
> >
> > Can you please help me understand why misra_c2012_rules has only a small
> > subset of MISRA C rules to be skipped?
>
> Hi Stefano,
>
> MISRA rules are in this format X.Y, misra_c2012_rules is a dictionary where the key is
> X and the value is the maximum number that Y can have.
>
> For example rule 13.Y goes from 13.1 to 13.6 (in the dictionary misra_c2012_rules[13] == 6),
> so the code can now check which among (13.1 .. 13.6) is not in the rule_list and add it to the
> list of skipped rules.
>
> Here an example:
> {
> "script": "misra.py",
> "args": [
> "--rule-texts=/path/to/cppcheck-misra.txt",
> "--suppress-rules=1.1,1.2,1.4,2.2,2.3,2.4,2.5,2.6,2.7,3.1,4.1,4.2,5.5,5.6,5.7,5.8,5.9,6.1,7.1,7.2,7.3,7.4,8.2,8.3,8.7,8.9,8.11,8.13,8.14,9.3,9.4,9.5,10.1,10.2,10.3,10.4,10.5,10.6,10.7,10.8,11.1,11.2,11.3,11.4,11.5,11.6,11.7,11.8,11.9,12.1,12.2,12.3,12.4,12.5,13.1,13.2,13.3,13.4,13.5,14.2,14.3,14.4,15.1,15.2,15.3,15.4,15.5,15.6,15.7,16.1,16.2,16.3,16.4,16.5,16.6,17.1,17.2,17.5,17.6,17.7,17.8,18.1,18.2,18.3,18.4,18.5,18.6,18.7,18.8,19.1,19.2,20.1,20.2,20.3,20.4,20.5,20.6,20.8,20.9,20.10,20.11,20.12,21.1,21.2,21.3,21.4,21.5,21.6,21.7,21.8,21.9,21.10,21.11,21.12,21.13,21.14,21.15,21.16,21.17,21.18,21.19,21.20,21.21,22.1,22.2,22.3,22.4,22.5,22.6,22.7,22.8,22.9,22.10"
> ]
> }
>
> So this patch is solving two issues, the first one was that rule 22.Y was never included in the suppressed
> list because range(1,22) produces a range in [1..21], the second issue is that the code was producing
> Invalid MISRA C 2012 rules, for example 1.21 and so on.
I see, that makes sense. Please improve the commit message with this
information and add
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-30 20:26 ` Stefano Stabellini
@ 2022-12-01 8:33 ` Jan Beulich
2022-12-01 11:18 ` Luca Fancellu
2022-12-01 15:34 ` Luca Fancellu
1 sibling, 1 reply; 31+ messages in thread
From: Jan Beulich @ 2022-12-01 8:33 UTC (permalink / raw)
To: Stefano Stabellini, Luca Fancellu
Cc: Xen-devel, Bertrand Marquis, Wei Chen, Andrew Cooper,
George Dunlap, Julien Grall, Wei Liu
On 30.11.2022 21:26, Stefano Stabellini wrote:
> On Wed, 30 Nov 2022, Luca Fancellu wrote:
>>> I think the revert of the cppcheck integration in xen/Makefile and
>>> xen/tools/merge_cppcheck_reports.py could be a separate patch. There is
>>> no need to make sure cppcheck support in the xen Makefile is
>>> "bisectable". That patch could have my acked-by already.
>>
>> Ok I will split these changes in a following patch
>>
>>>
>>> Also the document changes introduced in this patch have my reviewed-by:
>>> - docs/misra/cppcheck.txt
>>> - docs/misra/documenting-violations.rst
>>> - docs/misra/false-positive-cppcheck.json
>>> - docs/misra/xen-static-analysis.rst
>>
>> Thank you, should I put those files in a separate patch with your rev-by before
>> this patch or this is just a comment for you to remember which file you already
>> reviewed?
>
> If Jan and the other reviewers are OK, I think you could split them out
> in a separate patch and add my reviewed-by. If Jan prefers to keep it
> all together in one patch, then I wrote it down so that I remember what
> I have already acked :-)
Docs changes being split off and going in first is okay as long as what
is being documented is present behavior. If other changes are needed to
make (parts of) new documentation actually correct, then it should imo
go together. If new documentation describes future behavior (e.g.
design docs), then of course it's fine as well to go in early, as then
there simply is no code anywhere which this would (temporarily) not
describe correctly.
Jan
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-12-01 8:33 ` Jan Beulich
@ 2022-12-01 11:18 ` Luca Fancellu
2022-12-01 11:20 ` Jan Beulich
0 siblings, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-12-01 11:18 UTC (permalink / raw)
To: Jan Beulich
Cc: Stefano Stabellini, Xen-devel, Bertrand Marquis, Wei Chen,
Andrew Cooper, George Dunlap, Julien Grall, Wei Liu
> On 1 Dec 2022, at 08:33, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 30.11.2022 21:26, Stefano Stabellini wrote:
>> On Wed, 30 Nov 2022, Luca Fancellu wrote:
>>>> I think the revert of the cppcheck integration in xen/Makefile and
>>>> xen/tools/merge_cppcheck_reports.py could be a separate patch. There is
>>>> no need to make sure cppcheck support in the xen Makefile is
>>>> "bisectable". That patch could have my acked-by already.
>>>
>>> Ok I will split these changes in a following patch
>>>
>>>>
>>>> Also the document changes introduced in this patch have my reviewed-by:
>>>> - docs/misra/cppcheck.txt
>>>> - docs/misra/documenting-violations.rst
>>>> - docs/misra/false-positive-cppcheck.json
>>>> - docs/misra/xen-static-analysis.rst
>>>
>>> Thank you, should I put those files in a separate patch with your rev-by before
>>> this patch or this is just a comment for you to remember which file you already
>>> reviewed?
>>
>> If Jan and the other reviewers are OK, I think you could split them out
>> in a separate patch and add my reviewed-by. If Jan prefers to keep it
>> all together in one patch, then I wrote it down so that I remember what
>> I have already acked :-)
>
> Docs changes being split off and going in first is okay as long as what
> is being documented is present behavior. If other changes are needed to
> make (parts of) new documentation actually correct, then it should imo
> go together. If new documentation describes future behavior (e.g.
> design docs), then of course it's fine as well to go in early, as then
> there simply is no code anywhere which this would (temporarily) not
> describe correctly.
Yeah I thought so, I would prefer to keep these files here otherwise I would need to
change them somehow and I would lose the r-by anyway.
Regarding the revert of cppcheck from makefile and xen/tools/merge_cppcheck_reports.py,
are you ok if I send a patch with only those changes? Would it be ok for you if the new patch
is after this one?
Just asking to prevent back and forth.
Thank you
>
> Jan
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-12-01 11:18 ` Luca Fancellu
@ 2022-12-01 11:20 ` Jan Beulich
2022-12-01 15:15 ` Stefano Stabellini
0 siblings, 1 reply; 31+ messages in thread
From: Jan Beulich @ 2022-12-01 11:20 UTC (permalink / raw)
To: Luca Fancellu
Cc: Stefano Stabellini, Xen-devel, Bertrand Marquis, Wei Chen,
Andrew Cooper, George Dunlap, Julien Grall, Wei Liu
On 01.12.2022 12:18, Luca Fancellu wrote:
>> On 1 Dec 2022, at 08:33, Jan Beulich <jbeulich@suse.com> wrote:
>> On 30.11.2022 21:26, Stefano Stabellini wrote:
>>> On Wed, 30 Nov 2022, Luca Fancellu wrote:
>>>>> I think the revert of the cppcheck integration in xen/Makefile and
>>>>> xen/tools/merge_cppcheck_reports.py could be a separate patch. There is
>>>>> no need to make sure cppcheck support in the xen Makefile is
>>>>> "bisectable". That patch could have my acked-by already.
>>>>
>>>> Ok I will split these changes in a following patch
>>>>
>>>>>
>>>>> Also the document changes introduced in this patch have my reviewed-by:
>>>>> - docs/misra/cppcheck.txt
>>>>> - docs/misra/documenting-violations.rst
>>>>> - docs/misra/false-positive-cppcheck.json
>>>>> - docs/misra/xen-static-analysis.rst
>>>>
>>>> Thank you, should I put those files in a separate patch with your rev-by before
>>>> this patch or this is just a comment for you to remember which file you already
>>>> reviewed?
>>>
>>> If Jan and the other reviewers are OK, I think you could split them out
>>> in a separate patch and add my reviewed-by. If Jan prefers to keep it
>>> all together in one patch, then I wrote it down so that I remember what
>>> I have already acked :-)
>>
>> Docs changes being split off and going in first is okay as long as what
>> is being documented is present behavior. If other changes are needed to
>> make (parts of) new documentation actually correct, then it should imo
>> go together. If new documentation describes future behavior (e.g.
>> design docs), then of course it's fine as well to go in early, as then
>> there simply is no code anywhere which this would (temporarily) not
>> describe correctly.
>
> Yeah I thought so, I would prefer to keep these files here otherwise I would need to
> change them somehow and I would lose the r-by anyway.
>
> Regarding the revert of cppcheck from makefile and xen/tools/merge_cppcheck_reports.py,
> are you ok if I send a patch with only those changes? Would it be ok for you if the new patch
> is after this one?
I don't mind you doing so, but I guess the question is mainly to people
actually / possibly making use of those make goals.
Jan
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 3/4] tools/misra: fix skipped rule numbers
2022-11-30 23:34 ` Stefano Stabellini
@ 2022-12-01 11:27 ` Luca Fancellu
2022-12-01 15:16 ` Stefano Stabellini
0 siblings, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-12-01 11:27 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Bertrand Marquis, Wei Chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Wei Liu
>>
>> Hi Stefano,
>>
>> MISRA rules are in this format X.Y, misra_c2012_rules is a dictionary where the key is
>> X and the value is the maximum number that Y can have.
>>
>> For example rule 13.Y goes from 13.1 to 13.6 (in the dictionary misra_c2012_rules[13] == 6),
>> so the code can now check which among (13.1 .. 13.6) is not in the rule_list and add it to the
>> list of skipped rules.
>>
>> Here an example:
>> {
>> "script": "misra.py",
>> "args": [
>> "--rule-texts=/path/to/cppcheck-misra.txt",
>> "--suppress-rules=1.1,1.2,1.4,2.2,2.3,2.4,2.5,2.6,2.7,3.1,4.1,4.2,5.5,5.6,5.7,5.8,5.9,6.1,7.1,7.2,7.3,7.4,8.2,8.3,8.7,8.9,8.11,8.13,8.14,9.3,9.4,9.5,10.1,10.2,10.3,10.4,10.5,10.6,10.7,10.8,11.1,11.2,11.3,11.4,11.5,11.6,11.7,11.8,11.9,12.1,12.2,12.3,12.4,12.5,13.1,13.2,13.3,13.4,13.5,14.2,14.3,14.4,15.1,15.2,15.3,15.4,15.5,15.6,15.7,16.1,16.2,16.3,16.4,16.5,16.6,17.1,17.2,17.5,17.6,17.7,17.8,18.1,18.2,18.3,18.4,18.5,18.6,18.7,18.8,19.1,19.2,20.1,20.2,20.3,20.4,20.5,20.6,20.8,20.9,20.10,20.11,20.12,21.1,21.2,21.3,21.4,21.5,21.6,21.7,21.8,21.9,21.10,21.11,21.12,21.13,21.14,21.15,21.16,21.17,21.18,21.19,21.20,21.21,22.1,22.2,22.3,22.4,22.5,22.6,22.7,22.8,22.9,22.10"
>> ]
>> }
>>
>> So this patch is solving two issues, the first one was that rule 22.Y was never included in the suppressed
>> list because range(1,22) produces a range in [1..21], the second issue is that the code was producing
>> Invalid MISRA C 2012 rules, for example 1.21 and so on.
>
> I see, that makes sense. Please improve the commit message with this
> information and add
>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Thank you,
If you agree, I will change the commit message to be this one:
MISRA rules are in the format Rule X.Y, currently the script
convert_misra_doc.py is using two nested loop through range(1,22) to
enumerate rules that needs to be skipped, using combination of X.Y in
that range, however there are two issues in the code:
- rule 22 is never included because the range(1,22) produces a range
in [1..21]
- the second issue is that the code is producing invalid MISRA C 2012
rules, for example 1.21 and so on
Fix the issue using a dictionary that list the rules in misra c2012.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-12-01 11:20 ` Jan Beulich
@ 2022-12-01 15:15 ` Stefano Stabellini
0 siblings, 0 replies; 31+ messages in thread
From: Stefano Stabellini @ 2022-12-01 15:15 UTC (permalink / raw)
To: Jan Beulich
Cc: Luca Fancellu, Stefano Stabellini, Xen-devel, Bertrand Marquis,
Wei Chen, Andrew Cooper, George Dunlap, Julien Grall, Wei Liu
On Thu, 1 Dec 2022, Jan Beulich wrote:
> On 01.12.2022 12:18, Luca Fancellu wrote:
> >> On 1 Dec 2022, at 08:33, Jan Beulich <jbeulich@suse.com> wrote:
> >> On 30.11.2022 21:26, Stefano Stabellini wrote:
> >>> On Wed, 30 Nov 2022, Luca Fancellu wrote:
> >>>>> I think the revert of the cppcheck integration in xen/Makefile and
> >>>>> xen/tools/merge_cppcheck_reports.py could be a separate patch. There is
> >>>>> no need to make sure cppcheck support in the xen Makefile is
> >>>>> "bisectable". That patch could have my acked-by already.
> >>>>
> >>>> Ok I will split these changes in a following patch
> >>>>
> >>>>>
> >>>>> Also the document changes introduced in this patch have my reviewed-by:
> >>>>> - docs/misra/cppcheck.txt
> >>>>> - docs/misra/documenting-violations.rst
> >>>>> - docs/misra/false-positive-cppcheck.json
> >>>>> - docs/misra/xen-static-analysis.rst
> >>>>
> >>>> Thank you, should I put those files in a separate patch with your rev-by before
> >>>> this patch or this is just a comment for you to remember which file you already
> >>>> reviewed?
> >>>
> >>> If Jan and the other reviewers are OK, I think you could split them out
> >>> in a separate patch and add my reviewed-by. If Jan prefers to keep it
> >>> all together in one patch, then I wrote it down so that I remember what
> >>> I have already acked :-)
> >>
> >> Docs changes being split off and going in first is okay as long as what
> >> is being documented is present behavior. If other changes are needed to
> >> make (parts of) new documentation actually correct, then it should imo
> >> go together. If new documentation describes future behavior (e.g.
> >> design docs), then of course it's fine as well to go in early, as then
> >> there simply is no code anywhere which this would (temporarily) not
> >> describe correctly.
> >
> > Yeah I thought so, I would prefer to keep these files here otherwise I would need to
> > change them somehow and I would lose the r-by anyway.
> >
> > Regarding the revert of cppcheck from makefile and xen/tools/merge_cppcheck_reports.py,
> > are you ok if I send a patch with only those changes? Would it be ok for you if the new patch
> > is after this one?
>
> I don't mind you doing so, but I guess the question is mainly to people
> actually / possibly making use of those make goals.
I think it is OK -- we are not at the stage where cppcheck is used in
production-worthy pipelines yet.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 3/4] tools/misra: fix skipped rule numbers
2022-12-01 11:27 ` Luca Fancellu
@ 2022-12-01 15:16 ` Stefano Stabellini
0 siblings, 0 replies; 31+ messages in thread
From: Stefano Stabellini @ 2022-12-01 15:16 UTC (permalink / raw)
To: Luca Fancellu
Cc: Stefano Stabellini, Xen-devel, Bertrand Marquis, Wei Chen,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall, Wei Liu
On Thu, 1 Dec 2022, Luca Fancellu wrote:
> >> Hi Stefano,
> >>
> >> MISRA rules are in this format X.Y, misra_c2012_rules is a dictionary where the key is
> >> X and the value is the maximum number that Y can have.
> >>
> >> For example rule 13.Y goes from 13.1 to 13.6 (in the dictionary misra_c2012_rules[13] == 6),
> >> so the code can now check which among (13.1 .. 13.6) is not in the rule_list and add it to the
> >> list of skipped rules.
> >>
> >> Here an example:
> >> {
> >> "script": "misra.py",
> >> "args": [
> >> "--rule-texts=/path/to/cppcheck-misra.txt",
> >> "--suppress-rules=1.1,1.2,1.4,2.2,2.3,2.4,2.5,2.6,2.7,3.1,4.1,4.2,5.5,5.6,5.7,5.8,5.9,6.1,7.1,7.2,7.3,7.4,8.2,8.3,8.7,8.9,8.11,8.13,8.14,9.3,9.4,9.5,10.1,10.2,10.3,10.4,10.5,10.6,10.7,10.8,11.1,11.2,11.3,11.4,11.5,11.6,11.7,11.8,11.9,12.1,12.2,12.3,12.4,12.5,13.1,13.2,13.3,13.4,13.5,14.2,14.3,14.4,15.1,15.2,15.3,15.4,15.5,15.6,15.7,16.1,16.2,16.3,16.4,16.5,16.6,17.1,17.2,17.5,17.6,17.7,17.8,18.1,18.2,18.3,18.4,18.5,18.6,18.7,18.8,19.1,19.2,20.1,20.2,20.3,20.4,20.5,20.6,20.8,20.9,20.10,20.11,20.12,21.1,21.2,21.3,21.4,21.5,21.6,21.7,21.8,21.9,21.10,21.11,21.12,21.13,21.14,21.15,21.16,21.17,21.18,21.19,21.20,21.21,22.1,22.2,22.3,22.4,22.5,22.6,22.7,22.8,22.9,22.10"
> >> ]
> >> }
> >>
> >> So this patch is solving two issues, the first one was that rule 22.Y was never included in the suppressed
> >> list because range(1,22) produces a range in [1..21], the second issue is that the code was producing
> >> Invalid MISRA C 2012 rules, for example 1.21 and so on.
> >
> > I see, that makes sense. Please improve the commit message with this
> > information and add
> >
> > Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>
> Thank you,
>
> If you agree, I will change the commit message to be this one:
>
> MISRA rules are in the format Rule X.Y, currently the script
> convert_misra_doc.py is using two nested loop through range(1,22) to
> enumerate rules that needs to be skipped, using combination of X.Y in
> that range, however there are two issues in the code:
> - rule 22 is never included because the range(1,22) produces a range
> in [1..21]
> - the second issue is that the code is producing invalid MISRA C 2012
> rules, for example 1.21 and so on
>
> Fix the issue using a dictionary that list the rules in misra c2012.
Sounds good
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-11-30 20:26 ` Stefano Stabellini
2022-12-01 8:33 ` Jan Beulich
@ 2022-12-01 15:34 ` Luca Fancellu
2022-12-01 20:23 ` Stefano Stabellini
1 sibling, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-12-01 15:34 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Bertrand Marquis, Wei Chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Wei Liu
Hi Stefano,
>>>
>>>
>>>> + sm_tool_args="n"
>>>> + ;;
>>>> + --cppcheck-cmd=*)
>>>> + CPPCHECK_TOOL="$(eval echo "${OPTION#*=}")"
>>>> + sm_tool_args="y"
>>>> + ;;
>>>> + --cppcheck-html)
>>>> + CPPCHECK_HTML="y"
>>>> + sm_tool_args="n"
>>>> + ;;
>>>> + --cppcheck-plat=*)
>>>> + CPPCHECK_PLAT_PATH="$(eval echo "${OPTION#*=}")"
>>>> + sm_tool_args="n"
>>>> + ;;
>>>> + --ignore-path=*)
>>>> + IGNORE_PATH_LIST="${IGNORE_PATH_LIST} $(eval echo "${OPTION#*=}")"
>>>> + sm_tool_args="n"
>>>> + ;;
>>>> + --)
>>>> + forward_to_cc="y"
>>>> + sm_tool_args="n"
>>>> + ;;
>>>> + *)
>>>> + if [ "${sm_tool_args}" = "y" ]; then
>>>> + CPPCHECK_TOOL_ARGS="${CPPCHECK_TOOL_ARGS} ${OPTION}"
>>>> + else
>>>> + echo "Invalid option ${OPTION}"
>>>> + exit 1
>>>
>>> It doesn't look like sm_tool_args is really needed? It is only set to
>>> 'y' in the case of --cppcheck-cmd, and in that case we also set
>>> CPPCHECK_TOOL. CPPCHECK_TOOL is the variable used below. Am I missing
>>> something?
>>
>> We use sm_tool_args to fill CPPCHECK_TOOL_ARGS, basically it’s a state machine where
>> when we find --cppcheck-cmd=<xxx> we expect that every other space separated arguments
>> passed afterwards are the args for cppcheck, so we append to CPPCHECK_TOOL_ARGS
>> until we find an argument that is supposed to be only for this script.
>
> That seems a bit unnecessary: if the user wants to pass arguments to
> cppcheck, the user would do --cppcheck-cmd="cppcheck arg1 arg2" with ""
> quotes. Doing that should make --cppcheck-cmd="cppcheck arg1 arg2" be
> seen as a single argument from this script point of view. CPPCHECK_TOOL
> would end up being set to "cppcheck arg1 arg2" which is what we want
> anyway? And if we need to distinguish between the cppcheck binary and
> its argument we could use "cut" to extract "cppcheck", "arg1", and
> "arg2" from CPPCHECK_TOOL. Would that work?
>
I gave a try for the quotes, the problem is that we need to have quotes in CC=“...”, so adding
quotes also to --cppcheck-cmd= which is inside CC=“...” is preventing the Makefile to work,
I tried escaping etc but I didn’t manage to have it working, so would you agree on keeping it
like that?
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-12-01 15:34 ` Luca Fancellu
@ 2022-12-01 20:23 ` Stefano Stabellini
2022-12-02 13:09 ` Luca Fancellu
0 siblings, 1 reply; 31+ messages in thread
From: Stefano Stabellini @ 2022-12-01 20:23 UTC (permalink / raw)
To: Luca Fancellu
Cc: Stefano Stabellini, Xen-devel, Bertrand Marquis, Wei Chen,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall, Wei Liu
[-- Attachment #1: Type: text/plain, Size: 4066 bytes --]
On Thu, 1 Dec 2022, Luca Fancellu wrote:
> Hi Stefano,
>
> >>>
> >>>
> >>>> + sm_tool_args="n"
> >>>> + ;;
> >>>> + --cppcheck-cmd=*)
> >>>> + CPPCHECK_TOOL="$(eval echo "${OPTION#*=}")"
> >>>> + sm_tool_args="y"
> >>>> + ;;
> >>>> + --cppcheck-html)
> >>>> + CPPCHECK_HTML="y"
> >>>> + sm_tool_args="n"
> >>>> + ;;
> >>>> + --cppcheck-plat=*)
> >>>> + CPPCHECK_PLAT_PATH="$(eval echo "${OPTION#*=}")"
> >>>> + sm_tool_args="n"
> >>>> + ;;
> >>>> + --ignore-path=*)
> >>>> + IGNORE_PATH_LIST="${IGNORE_PATH_LIST} $(eval echo "${OPTION#*=}")"
> >>>> + sm_tool_args="n"
> >>>> + ;;
> >>>> + --)
> >>>> + forward_to_cc="y"
> >>>> + sm_tool_args="n"
> >>>> + ;;
> >>>> + *)
> >>>> + if [ "${sm_tool_args}" = "y" ]; then
> >>>> + CPPCHECK_TOOL_ARGS="${CPPCHECK_TOOL_ARGS} ${OPTION}"
> >>>> + else
> >>>> + echo "Invalid option ${OPTION}"
> >>>> + exit 1
> >>>
> >>> It doesn't look like sm_tool_args is really needed? It is only set to
> >>> 'y' in the case of --cppcheck-cmd, and in that case we also set
> >>> CPPCHECK_TOOL. CPPCHECK_TOOL is the variable used below. Am I missing
> >>> something?
> >>
> >> We use sm_tool_args to fill CPPCHECK_TOOL_ARGS, basically it’s a state machine where
> >> when we find --cppcheck-cmd=<xxx> we expect that every other space separated arguments
> >> passed afterwards are the args for cppcheck, so we append to CPPCHECK_TOOL_ARGS
> >> until we find an argument that is supposed to be only for this script.
> >
> > That seems a bit unnecessary: if the user wants to pass arguments to
> > cppcheck, the user would do --cppcheck-cmd="cppcheck arg1 arg2" with ""
> > quotes. Doing that should make --cppcheck-cmd="cppcheck arg1 arg2" be
> > seen as a single argument from this script point of view. CPPCHECK_TOOL
> > would end up being set to "cppcheck arg1 arg2" which is what we want
> > anyway? And if we need to distinguish between the cppcheck binary and
> > its argument we could use "cut" to extract "cppcheck", "arg1", and
> > "arg2" from CPPCHECK_TOOL. Would that work?
> >
>
> I gave a try for the quotes, the problem is that we need to have quotes in CC=“...”, so adding
> quotes also to --cppcheck-cmd= which is inside CC=“...” is preventing the Makefile to work,
> I tried escaping etc but I didn’t manage to have it working, so would you agree on keeping it
> like that?
Is the problem coming from the following?
cppcheck_cc_flags = """--compiler={} --cppcheck-cmd={} {}
--cppcheck-plat={}/cppcheck-plat --ignore-path=tools/
""".format(xen_cc, settings.cppcheck_binpath, cppcheck_flags,
settings.tools_dir)
if settings.cppcheck_html:
cppcheck_cc_flags = cppcheck_cc_flags + " --cppcheck-html"
# Generate the extra make argument to pass the cppcheck-cc.sh wrapper as CC
cppcheck_extra_make_args = "CC=\"{}/cppcheck-cc.sh {} --\"".format(
settings.tools_dir,
cppcheck_cc_flags
).replace("\n", "")
Wouldn't something like the following solve the issue?
settings.cppcheck_binpath = settings.cppcheck_binpath + " " + cppcheck_cc_flags
cppcheck_cc_flags = """--compiler={} --cppcheck-cmd=\"{}\"
--cppcheck-plat={}/cppcheck-plat --ignore-path=tools/
""".format(xen_cc, settings.cppcheck_binpath, settings.tools_dir)
if settings.cppcheck_html:
cppcheck_cc_flags = cppcheck_cc_flags + " --cppcheck-html"
# Generate the extra make argument to pass the cppcheck-cc.sh wrapper as CC
cppcheck_extra_make_args = "CC=\"{}/cppcheck-cc.sh {} --\"".format(
settings.tools_dir,
cppcheck_cc_flags
).replace("\n", "")
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-12-01 20:23 ` Stefano Stabellini
@ 2022-12-02 13:09 ` Luca Fancellu
2022-12-03 0:41 ` Stefano Stabellini
0 siblings, 1 reply; 31+ messages in thread
From: Luca Fancellu @ 2022-12-02 13:09 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Bertrand Marquis, Wei Chen, Andrew Cooper,
George Dunlap, Jan Beulich, Julien Grall, Wei Liu
> On 1 Dec 2022, at 20:23, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Thu, 1 Dec 2022, Luca Fancellu wrote:
>> Hi Stefano,
>>
>>>>>
>>>>>
>>>>>> + sm_tool_args="n"
>>>>>> + ;;
>>>>>> + --cppcheck-cmd=*)
>>>>>> + CPPCHECK_TOOL="$(eval echo "${OPTION#*=}")"
>>>>>> + sm_tool_args="y"
>>>>>> + ;;
>>>>>> + --cppcheck-html)
>>>>>> + CPPCHECK_HTML="y"
>>>>>> + sm_tool_args="n"
>>>>>> + ;;
>>>>>> + --cppcheck-plat=*)
>>>>>> + CPPCHECK_PLAT_PATH="$(eval echo "${OPTION#*=}")"
>>>>>> + sm_tool_args="n"
>>>>>> + ;;
>>>>>> + --ignore-path=*)
>>>>>> + IGNORE_PATH_LIST="${IGNORE_PATH_LIST} $(eval echo "${OPTION#*=}")"
>>>>>> + sm_tool_args="n"
>>>>>> + ;;
>>>>>> + --)
>>>>>> + forward_to_cc="y"
>>>>>> + sm_tool_args="n"
>>>>>> + ;;
>>>>>> + *)
>>>>>> + if [ "${sm_tool_args}" = "y" ]; then
>>>>>> + CPPCHECK_TOOL_ARGS="${CPPCHECK_TOOL_ARGS} ${OPTION}"
>>>>>> + else
>>>>>> + echo "Invalid option ${OPTION}"
>>>>>> + exit 1
>>>>>
>>>>> It doesn't look like sm_tool_args is really needed? It is only set to
>>>>> 'y' in the case of --cppcheck-cmd, and in that case we also set
>>>>> CPPCHECK_TOOL. CPPCHECK_TOOL is the variable used below. Am I missing
>>>>> something?
>>>>
>>>> We use sm_tool_args to fill CPPCHECK_TOOL_ARGS, basically it’s a state machine where
>>>> when we find --cppcheck-cmd=<xxx> we expect that every other space separated arguments
>>>> passed afterwards are the args for cppcheck, so we append to CPPCHECK_TOOL_ARGS
>>>> until we find an argument that is supposed to be only for this script.
>>>
>>> That seems a bit unnecessary: if the user wants to pass arguments to
>>> cppcheck, the user would do --cppcheck-cmd="cppcheck arg1 arg2" with ""
>>> quotes. Doing that should make --cppcheck-cmd="cppcheck arg1 arg2" be
>>> seen as a single argument from this script point of view. CPPCHECK_TOOL
>>> would end up being set to "cppcheck arg1 arg2" which is what we want
>>> anyway? And if we need to distinguish between the cppcheck binary and
>>> its argument we could use "cut" to extract "cppcheck", "arg1", and
>>> "arg2" from CPPCHECK_TOOL. Would that work?
>>>
>>
>> I gave a try for the quotes, the problem is that we need to have quotes in CC=“...”, so adding
>> quotes also to --cppcheck-cmd= which is inside CC=“...” is preventing the Makefile to work,
>> I tried escaping etc but I didn’t manage to have it working, so would you agree on keeping it
>> like that?
>
> Is the problem coming from the following?
>
> cppcheck_cc_flags = """--compiler={} --cppcheck-cmd={} {}
> --cppcheck-plat={}/cppcheck-plat --ignore-path=tools/
> """.format(xen_cc, settings.cppcheck_binpath, cppcheck_flags,
> settings.tools_dir)
>
> if settings.cppcheck_html:
> cppcheck_cc_flags = cppcheck_cc_flags + " --cppcheck-html"
>
> # Generate the extra make argument to pass the cppcheck-cc.sh wrapper as CC
> cppcheck_extra_make_args = "CC=\"{}/cppcheck-cc.sh {} --\"".format(
> settings.tools_dir,
> cppcheck_cc_flags
> ).replace("\n", "")
>
>
> Wouldn't something like the following solve the issue?
>
> settings.cppcheck_binpath = settings.cppcheck_binpath + " " + cppcheck_cc_flags
>
> cppcheck_cc_flags = """--compiler={} --cppcheck-cmd=\"{}\"
> --cppcheck-plat={}/cppcheck-plat --ignore-path=tools/
> """.format(xen_cc, settings.cppcheck_binpath, settings.tools_dir)
>
> if settings.cppcheck_html:
> cppcheck_cc_flags = cppcheck_cc_flags + " --cppcheck-html"
>
> # Generate the extra make argument to pass the cppcheck-cc.sh wrapper as CC
> cppcheck_extra_make_args = "CC=\"{}/cppcheck-cc.sh {} --\"".format(
> settings.tools_dir,
> cppcheck_cc_flags
> ).replace("\n", "")
No unfortunately not, Makefile is very sensitive to quotes, I’ve tried with many combination of single/double quotes but nothing worked
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script
2022-12-02 13:09 ` Luca Fancellu
@ 2022-12-03 0:41 ` Stefano Stabellini
0 siblings, 0 replies; 31+ messages in thread
From: Stefano Stabellini @ 2022-12-03 0:41 UTC (permalink / raw)
To: Luca Fancellu
Cc: Stefano Stabellini, Xen-devel, Bertrand Marquis, Wei Chen,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall, Wei Liu
[-- Attachment #1: Type: text/plain, Size: 4690 bytes --]
On Fri, 2 Dec 2022, Luca Fancellu wrote:
> > On 1 Dec 2022, at 20:23, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >
> > On Thu, 1 Dec 2022, Luca Fancellu wrote:
> >> Hi Stefano,
> >>
> >>>>>
> >>>>>
> >>>>>> + sm_tool_args="n"
> >>>>>> + ;;
> >>>>>> + --cppcheck-cmd=*)
> >>>>>> + CPPCHECK_TOOL="$(eval echo "${OPTION#*=}")"
> >>>>>> + sm_tool_args="y"
> >>>>>> + ;;
> >>>>>> + --cppcheck-html)
> >>>>>> + CPPCHECK_HTML="y"
> >>>>>> + sm_tool_args="n"
> >>>>>> + ;;
> >>>>>> + --cppcheck-plat=*)
> >>>>>> + CPPCHECK_PLAT_PATH="$(eval echo "${OPTION#*=}")"
> >>>>>> + sm_tool_args="n"
> >>>>>> + ;;
> >>>>>> + --ignore-path=*)
> >>>>>> + IGNORE_PATH_LIST="${IGNORE_PATH_LIST} $(eval echo "${OPTION#*=}")"
> >>>>>> + sm_tool_args="n"
> >>>>>> + ;;
> >>>>>> + --)
> >>>>>> + forward_to_cc="y"
> >>>>>> + sm_tool_args="n"
> >>>>>> + ;;
> >>>>>> + *)
> >>>>>> + if [ "${sm_tool_args}" = "y" ]; then
> >>>>>> + CPPCHECK_TOOL_ARGS="${CPPCHECK_TOOL_ARGS} ${OPTION}"
> >>>>>> + else
> >>>>>> + echo "Invalid option ${OPTION}"
> >>>>>> + exit 1
> >>>>>
> >>>>> It doesn't look like sm_tool_args is really needed? It is only set to
> >>>>> 'y' in the case of --cppcheck-cmd, and in that case we also set
> >>>>> CPPCHECK_TOOL. CPPCHECK_TOOL is the variable used below. Am I missing
> >>>>> something?
> >>>>
> >>>> We use sm_tool_args to fill CPPCHECK_TOOL_ARGS, basically it’s a state machine where
> >>>> when we find --cppcheck-cmd=<xxx> we expect that every other space separated arguments
> >>>> passed afterwards are the args for cppcheck, so we append to CPPCHECK_TOOL_ARGS
> >>>> until we find an argument that is supposed to be only for this script.
> >>>
> >>> That seems a bit unnecessary: if the user wants to pass arguments to
> >>> cppcheck, the user would do --cppcheck-cmd="cppcheck arg1 arg2" with ""
> >>> quotes. Doing that should make --cppcheck-cmd="cppcheck arg1 arg2" be
> >>> seen as a single argument from this script point of view. CPPCHECK_TOOL
> >>> would end up being set to "cppcheck arg1 arg2" which is what we want
> >>> anyway? And if we need to distinguish between the cppcheck binary and
> >>> its argument we could use "cut" to extract "cppcheck", "arg1", and
> >>> "arg2" from CPPCHECK_TOOL. Would that work?
> >>>
> >>
> >> I gave a try for the quotes, the problem is that we need to have quotes in CC=“...”, so adding
> >> quotes also to --cppcheck-cmd= which is inside CC=“...” is preventing the Makefile to work,
> >> I tried escaping etc but I didn’t manage to have it working, so would you agree on keeping it
> >> like that?
> >
> > Is the problem coming from the following?
> >
> > cppcheck_cc_flags = """--compiler={} --cppcheck-cmd={} {}
> > --cppcheck-plat={}/cppcheck-plat --ignore-path=tools/
> > """.format(xen_cc, settings.cppcheck_binpath, cppcheck_flags,
> > settings.tools_dir)
> >
> > if settings.cppcheck_html:
> > cppcheck_cc_flags = cppcheck_cc_flags + " --cppcheck-html"
> >
> > # Generate the extra make argument to pass the cppcheck-cc.sh wrapper as CC
> > cppcheck_extra_make_args = "CC=\"{}/cppcheck-cc.sh {} --\"".format(
> > settings.tools_dir,
> > cppcheck_cc_flags
> > ).replace("\n", "")
> >
> >
> > Wouldn't something like the following solve the issue?
> >
> > settings.cppcheck_binpath = settings.cppcheck_binpath + " " + cppcheck_cc_flags
> >
> > cppcheck_cc_flags = """--compiler={} --cppcheck-cmd=\"{}\"
> > --cppcheck-plat={}/cppcheck-plat --ignore-path=tools/
> > """.format(xen_cc, settings.cppcheck_binpath, settings.tools_dir)
> >
> > if settings.cppcheck_html:
> > cppcheck_cc_flags = cppcheck_cc_flags + " --cppcheck-html"
> >
> > # Generate the extra make argument to pass the cppcheck-cc.sh wrapper as CC
> > cppcheck_extra_make_args = "CC=\"{}/cppcheck-cc.sh {} --\"".format(
> > settings.tools_dir,
> > cppcheck_cc_flags
> > ).replace("\n", "")
>
> No unfortunately not, Makefile is very sensitive to quotes, I’ve tried with many combination of single/double quotes but nothing worked
I spent a couple of hours to try to get it to work. I also admit defeat.
Keep your original code, that's better.
^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2022-12-03 0:41 UTC | newest]
Thread overview: 31+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-28 14:10 [PATCH 0/4] Static analyser finding deviation Luca Fancellu
2022-11-28 14:10 ` [PATCH 1/4] xen/scripts: add xen-analysis.py for coverity and eclair analysis Luca Fancellu
2022-11-29 1:39 ` Stefano Stabellini
2022-11-28 14:10 ` [PATCH 2/4] xen/scripts: add cppcheck tool to the xen-analysis.py script Luca Fancellu
2022-11-28 15:19 ` Jan Beulich
2022-11-28 15:37 ` Luca Fancellu
2022-11-29 9:42 ` Jan Beulich
2022-11-29 9:49 ` Luca Fancellu
2022-11-30 1:05 ` Stefano Stabellini
2022-11-30 11:59 ` Luca Fancellu
2022-11-30 20:26 ` Stefano Stabellini
2022-12-01 8:33 ` Jan Beulich
2022-12-01 11:18 ` Luca Fancellu
2022-12-01 11:20 ` Jan Beulich
2022-12-01 15:15 ` Stefano Stabellini
2022-12-01 15:34 ` Luca Fancellu
2022-12-01 20:23 ` Stefano Stabellini
2022-12-02 13:09 ` Luca Fancellu
2022-12-03 0:41 ` Stefano Stabellini
2022-11-28 14:10 ` [PATCH 3/4] tools/misra: fix skipped rule numbers Luca Fancellu
2022-11-29 23:51 ` Stefano Stabellini
2022-11-30 8:53 ` Luca Fancellu
2022-11-30 23:34 ` Stefano Stabellini
2022-12-01 11:27 ` Luca Fancellu
2022-12-01 15:16 ` Stefano Stabellini
2022-11-28 14:10 ` [PATCH 4/4] xen: Justify linker script defined symbols in include/xen/kernel.h Luca Fancellu
2022-11-28 15:19 ` Jan Beulich
2022-11-29 1:55 ` Stefano Stabellini
2022-11-29 1:55 ` [PATCH 0/4] Static analyser finding deviation Stefano Stabellini
2022-11-29 9:46 ` Luca Fancellu
2022-11-29 13:02 ` Luca Fancellu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.