* [PATCH 0/2] Enable clang tools for kernels build using gcc
@ 2022-06-28 12:27 Daniel Thompson
2022-06-28 12:27 ` [PATCH 1/2] clang-tools: Generate clang compatible output even with gcc builds Daniel Thompson
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Daniel Thompson @ 2022-06-28 12:27 UTC (permalink / raw)
To: Nathan Chancellor, Tom Rix
Cc: Daniel Thompson, Masahiro Yamada, Michal Marek, Nick Desaulniers,
linux-kbuild, llvm, linux-kernel
This patch set makes it possible to use clang based tools, such as
the clangd completion engine, regardless of our choice of compiler
for the actual kernel build.
Most of the fine details are included in the patch header for the
first patch (since we'd like them to appear in the kernel history so
they can help future archaeologists).
Daniel Thompson (2):
clang-tools: Generate clang compatible output even with gcc builds
Makefile: Enable clang-tidy and clang-analyzer for gcc builds
Makefile | 11 ++--
scripts/clang-tools/gen_compile_commands.py | 71 ++++++++++++++++++++-
2 files changed, 74 insertions(+), 8 deletions(-)
base-commit: a111daf0c53ae91e71fd2bfe7497862d14132e3e
--
2.35.1
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/2] clang-tools: Generate clang compatible output even with gcc builds 2022-06-28 12:27 [PATCH 0/2] Enable clang tools for kernels build using gcc Daniel Thompson @ 2022-06-28 12:27 ` Daniel Thompson 2022-06-28 18:35 ` Nick Desaulniers 2022-06-28 12:27 ` [PATCH 2/2] Makefile: Enable clang-tidy and clang-analyzer for " Daniel Thompson 2022-06-28 13:19 ` [PATCH 0/2] Enable clang tools for kernels build using gcc Tom Rix 2 siblings, 1 reply; 6+ messages in thread From: Daniel Thompson @ 2022-06-28 12:27 UTC (permalink / raw) To: Nathan Chancellor, Tom Rix Cc: Daniel Thompson, Masahiro Yamada, Michal Marek, Nick Desaulniers, linux-kbuild, llvm, linux-kernel Currently `make compile_commands.json` cannot produce useful output for kernels built with gcc. That is because kbuild will opportunistically enable gcc-specific command line options from recent versions of gcc. Options that are not compatible with clang cause trouble because most of the tools that consume compile_commands.json only understand the clang argument set. This is to be expected since it was the clang folks wrote the spec to help make those tools come alive (and AFAIK all the tools that consume the compilation database are closely linked to the clang tools): https://clang.llvm.org/docs/JSONCompilationDatabase.html Let's fix this by adding code to gen_compile_commands.py that will automatically strip not-supported-by-clang command line options from the compilation database. This allows the common consumers of the compilation database (clang-tidy, clangd code completion engine, CodeChecker, etc) to work without requiring the developer to build the kernel using a different C compiler. In theory this could cause problems if/when a not-based-on-clang tool emerges that reuses the clang compilation database format. This is not expected to be a problem in practice since the heuristics added to gen_compile_commands.py are pretty conservative. The should only ever disable some rather esoteric compiler options ("they must be esoteric otherwise clang would have implemented them..."). It is hard to reason about what will/won't break tools that are not yet written but we can hope the removing esoteric options will be benign! Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> --- Makefile | 5 +- scripts/clang-tools/gen_compile_commands.py | 71 ++++++++++++++++++++- 2 files changed, 74 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 513c1fbf7888..9ea6867aaf9c 100644 --- a/Makefile +++ b/Makefile @@ -1886,8 +1886,11 @@ nsdeps: modules # Clang Tooling # --------------------------------------------------------------------------- +ifdef CONFIG_CC_IS_GCC +gen_compile_commands-flags += --gcc +endif quiet_cmd_gen_compile_commands = GEN $@ - cmd_gen_compile_commands = $(PYTHON3) $< -a $(AR) -o $@ $(filter-out $<, $(real-prereqs)) + cmd_gen_compile_commands = $(PYTHON3) $< $(gen_compile_commands-flags) -a $(AR) -o $@ $(filter-out $<, $(real-prereqs)) $(extmod_prefix)compile_commands.json: scripts/clang-tools/gen_compile_commands.py \ $(if $(KBUILD_EXTMOD),,$(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS)) \ diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py index 1d1bde1fd45e..02f6a1408968 100755 --- a/scripts/clang-tools/gen_compile_commands.py +++ b/scripts/clang-tools/gen_compile_commands.py @@ -56,6 +56,9 @@ def parse_arguments(): ar_help = 'command used for parsing .a archives' parser.add_argument('-a', '--ar', type=str, default='llvm-ar', help=ar_help) + gcc_help = 'tidy up gcc invocations to work with clang' + parser.add_argument('-g', '--gcc', action='store_true', help=gcc_help) + paths_help = ('directories to search or files to parse ' '(files should be *.o, *.a, or modules.order). ' 'If nothing is specified, the current directory is searched') @@ -67,6 +70,7 @@ def parse_arguments(): os.path.abspath(args.directory), args.output, args.ar, + args.gcc, args.paths if len(args.paths) > 0 else [args.directory]) @@ -196,10 +200,73 @@ def process_line(root_directory, command_prefix, file_path): 'command': prefix + file_path, } +clang_options = {} + +def check_clang_compatibility(target, flag): + """Check that the supplied flag does not cause clang to return an error. + + The results of the check, which is expensive if repeated many times, is + cached in the clang_options variable and reused in subsequent calls. + """ + global clang_options + if flag in clang_options: + return clang_options[flag] + + c = 'echo "int f;"| clang {} {} - -E > /dev/null 2>&1'.format(target, flag) + rc = os.system(c) + compatible = rc == 0 + clang_options[flag] = compatible + if not compatible: + logging.info('Not supported by clang: %s', flag) + + return compatible + +def make_clang_compatible(entry): + """Scans and transforms the command line options to make the invocation + compatible with clang. + + There are two main heuristics: + + 1. Use the gcc compiler prefix to populate the clang --target variable + (which is needed for cross-compiles to work correctly) + + 2. Scan for any -f or -m options that are not supported by clang and + discard them. + + This allows us to use clang tools on our kernel builds even if we built the + kernel using gcc. + """ + newcmd = [] + target = '' + + # Splitting the command line like this isn't going to handle quoted + # strings transparently. However assuming the quoted string does not + # contain tabs, double spaces or words commencing with '-f' or '-c' + # (which is fairly reasonable) then this simple approach will be + # sufficient. + atoms = entry['command'].split() + + # Use the compiler prefix as the clang --target variable + if atoms[0].endswith('-gcc'): + target = '--target=' + os.path.basename(atoms[0][:-4]) + newcmd.append(atoms[0]) + newcmd.append(target) + del atoms[0] + + # Drop incompatible flags that provoke fatal errors for clang. Note that + # unsupported -Wenable-warning flags are not fatal so we don't have to + # worry about those. + for atom in atoms: + if atom.startswith('-f') or atom.startswith('-m'): + if not check_clang_compatibility(target, atom): + continue + newcmd.append(atom) + + entry['command'] = ' '.join(newcmd) def main(): """Walks through the directory and finds and parses .cmd files.""" - log_level, directory, output, ar, paths = parse_arguments() + log_level, directory, output, ar, gcc, paths = parse_arguments() level = getattr(logging, log_level) logging.basicConfig(format='%(levelname)s: %(message)s', level=level) @@ -232,6 +299,8 @@ def main(): try: entry = process_line(directory, result.group(1), result.group(2)) + if gcc: + make_clang_compatible(entry) compile_commands.append(entry) except ValueError as err: logging.info('Could not add line from %s: %s', -- 2.35.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] clang-tools: Generate clang compatible output even with gcc builds 2022-06-28 12:27 ` [PATCH 1/2] clang-tools: Generate clang compatible output even with gcc builds Daniel Thompson @ 2022-06-28 18:35 ` Nick Desaulniers 0 siblings, 0 replies; 6+ messages in thread From: Nick Desaulniers @ 2022-06-28 18:35 UTC (permalink / raw) To: Daniel Thompson Cc: Nathan Chancellor, Tom Rix, Masahiro Yamada, Michal Marek, linux-kbuild, llvm, linux-kernel On Tue, Jun 28, 2022 at 5:27 AM Daniel Thompson <daniel.thompson@linaro.org> wrote: > > Currently `make compile_commands.json` cannot produce useful output for > kernels built with gcc. That is because kbuild will opportunistically Right, compile_commands.json should reflect reality, as in what flags were actually used for the build. > enable gcc-specific command line options from recent versions of gcc. > Options that are not compatible with clang cause trouble because most of > the tools that consume compile_commands.json only understand the clang > argument set. This is to be expected since it was the clang folks wrote > the spec to help make those tools come alive (and AFAIK all the tools > that consume the compilation database are closely linked to the clang > tools): > https://clang.llvm.org/docs/JSONCompilationDatabase.html It sounds like the raison d'etre for this patch is to support clang-tidy and clang-scan kernel Makefile targets when CC=gcc? In that case, it sounds like compile_commands.json should be post processed only when using those Makefile targets with CC=gcc. As in: $ make compile_commands.json Should continue to produce a pristine/unmodified for this purpose compile_commands.json. Only when: $ make clang-analyzer or $ make clang-tidy are run should we: 1. produce a second compilation database in which we filter out command line flags clang cannot consume. 2. pass that explicitly to clang-tidy via -p. I suspect that second compile_commands.json will have to be placed in a new dir as well. I'm sympathetic to the intent of the series, but I think it corrupts the compilation_commands.json for gcc builds in order to support one new use case, at the cost of harming existing and future potential use cases by stripping out information that is precise. Such loss of fidelity makes me uncomfortable accepting this change as is. > > Let's fix this by adding code to gen_compile_commands.py that will > automatically strip not-supported-by-clang command line options from > the compilation database. This allows the common consumers of the > compilation database (clang-tidy, clangd code completion engine, > CodeChecker, etc) to work without requiring the developer to build the > kernel using a different C compiler. > > In theory this could cause problems if/when a not-based-on-clang tool > emerges that reuses the clang compilation database format. This is not > expected to be a problem in practice since the heuristics added to > gen_compile_commands.py are pretty conservative. The should only ever > disable some rather esoteric compiler options ("they must be esoteric > otherwise clang would have implemented them..."). It is hard to reason > about what will/won't break tools that are not yet written but we can > hope the removing esoteric options will be benign! > > Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> > --- > Makefile | 5 +- > scripts/clang-tools/gen_compile_commands.py | 71 ++++++++++++++++++++- > 2 files changed, 74 insertions(+), 2 deletions(-) > > diff --git a/Makefile b/Makefile > index 513c1fbf7888..9ea6867aaf9c 100644 > --- a/Makefile > +++ b/Makefile > @@ -1886,8 +1886,11 @@ nsdeps: modules > # Clang Tooling > # --------------------------------------------------------------------------- > > +ifdef CONFIG_CC_IS_GCC > +gen_compile_commands-flags += --gcc > +endif > quiet_cmd_gen_compile_commands = GEN $@ > - cmd_gen_compile_commands = $(PYTHON3) $< -a $(AR) -o $@ $(filter-out $<, $(real-prereqs)) > + cmd_gen_compile_commands = $(PYTHON3) $< $(gen_compile_commands-flags) -a $(AR) -o $@ $(filter-out $<, $(real-prereqs)) > > $(extmod_prefix)compile_commands.json: scripts/clang-tools/gen_compile_commands.py \ > $(if $(KBUILD_EXTMOD),,$(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS)) \ > diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py > index 1d1bde1fd45e..02f6a1408968 100755 > --- a/scripts/clang-tools/gen_compile_commands.py > +++ b/scripts/clang-tools/gen_compile_commands.py > @@ -56,6 +56,9 @@ def parse_arguments(): > ar_help = 'command used for parsing .a archives' > parser.add_argument('-a', '--ar', type=str, default='llvm-ar', help=ar_help) > > + gcc_help = 'tidy up gcc invocations to work with clang' > + parser.add_argument('-g', '--gcc', action='store_true', help=gcc_help) > + > paths_help = ('directories to search or files to parse ' > '(files should be *.o, *.a, or modules.order). ' > 'If nothing is specified, the current directory is searched') > @@ -67,6 +70,7 @@ def parse_arguments(): > os.path.abspath(args.directory), > args.output, > args.ar, > + args.gcc, > args.paths if len(args.paths) > 0 else [args.directory]) > > > @@ -196,10 +200,73 @@ def process_line(root_directory, command_prefix, file_path): > 'command': prefix + file_path, > } > > +clang_options = {} > + > +def check_clang_compatibility(target, flag): > + """Check that the supplied flag does not cause clang to return an error. > + > + The results of the check, which is expensive if repeated many times, is > + cached in the clang_options variable and reused in subsequent calls. > + """ > + global clang_options > + if flag in clang_options: > + return clang_options[flag] > + > + c = 'echo "int f;"| clang {} {} - -E > /dev/null 2>&1'.format(target, flag) > + rc = os.system(c) > + compatible = rc == 0 > + clang_options[flag] = compatible > + if not compatible: > + logging.info('Not supported by clang: %s', flag) > + > + return compatible > + > +def make_clang_compatible(entry): > + """Scans and transforms the command line options to make the invocation > + compatible with clang. > + > + There are two main heuristics: > + > + 1. Use the gcc compiler prefix to populate the clang --target variable > + (which is needed for cross-compiles to work correctly) > + > + 2. Scan for any -f or -m options that are not supported by clang and > + discard them. > + > + This allows us to use clang tools on our kernel builds even if we built the > + kernel using gcc. > + """ > + newcmd = [] > + target = '' > + > + # Splitting the command line like this isn't going to handle quoted > + # strings transparently. However assuming the quoted string does not > + # contain tabs, double spaces or words commencing with '-f' or '-c' > + # (which is fairly reasonable) then this simple approach will be > + # sufficient. > + atoms = entry['command'].split() > + > + # Use the compiler prefix as the clang --target variable > + if atoms[0].endswith('-gcc'): > + target = '--target=' + os.path.basename(atoms[0][:-4]) > + newcmd.append(atoms[0]) > + newcmd.append(target) > + del atoms[0] > + > + # Drop incompatible flags that provoke fatal errors for clang. Note that > + # unsupported -Wenable-warning flags are not fatal so we don't have to > + # worry about those. > + for atom in atoms: > + if atom.startswith('-f') or atom.startswith('-m'): > + if not check_clang_compatibility(target, atom): > + continue > + newcmd.append(atom) > + > + entry['command'] = ' '.join(newcmd) > > def main(): > """Walks through the directory and finds and parses .cmd files.""" > - log_level, directory, output, ar, paths = parse_arguments() > + log_level, directory, output, ar, gcc, paths = parse_arguments() > > level = getattr(logging, log_level) > logging.basicConfig(format='%(levelname)s: %(message)s', level=level) > @@ -232,6 +299,8 @@ def main(): > try: > entry = process_line(directory, result.group(1), > result.group(2)) > + if gcc: > + make_clang_compatible(entry) > compile_commands.append(entry) > except ValueError as err: > logging.info('Could not add line from %s: %s', > -- > 2.35.1 > -- Thanks, ~Nick Desaulniers ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/2] Makefile: Enable clang-tidy and clang-analyzer for gcc builds 2022-06-28 12:27 [PATCH 0/2] Enable clang tools for kernels build using gcc Daniel Thompson 2022-06-28 12:27 ` [PATCH 1/2] clang-tools: Generate clang compatible output even with gcc builds Daniel Thompson @ 2022-06-28 12:27 ` Daniel Thompson 2022-06-28 13:19 ` [PATCH 0/2] Enable clang tools for kernels build using gcc Tom Rix 2 siblings, 0 replies; 6+ messages in thread From: Daniel Thompson @ 2022-06-28 12:27 UTC (permalink / raw) To: Nathan Chancellor, Tom Rix Cc: Daniel Thompson, Masahiro Yamada, Michal Marek, Nick Desaulniers, linux-kbuild, llvm, linux-kernel It is now possible use clang-tidy and clang-analyzer even if your primary compiler is gcc. Remove the checks that prevent gcc builds from using these tools. Signed-off-by: Daniel Thompson <daniel.thompson@linaro.org> --- Makefile | 6 ------ 1 file changed, 6 deletions(-) diff --git a/Makefile b/Makefile index 9ea6867aaf9c..0afef56d8203 100644 --- a/Makefile +++ b/Makefile @@ -1901,17 +1901,11 @@ targets += $(extmod_prefix)compile_commands.json PHONY += clang-tidy clang-analyzer -ifdef CONFIG_CC_IS_CLANG quiet_cmd_clang_tools = CHECK $< cmd_clang_tools = $(PYTHON3) $(srctree)/scripts/clang-tools/run-clang-tools.py $@ $< clang-tidy clang-analyzer: $(extmod_prefix)compile_commands.json $(call cmd,clang_tools) -else -clang-tidy clang-analyzer: - @echo "$@ requires CC=clang" >&2 - @false -endif # Scripts to check various things for consistency # --------------------------------------------------------------------------- -- 2.35.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 0/2] Enable clang tools for kernels build using gcc 2022-06-28 12:27 [PATCH 0/2] Enable clang tools for kernels build using gcc Daniel Thompson 2022-06-28 12:27 ` [PATCH 1/2] clang-tools: Generate clang compatible output even with gcc builds Daniel Thompson 2022-06-28 12:27 ` [PATCH 2/2] Makefile: Enable clang-tidy and clang-analyzer for " Daniel Thompson @ 2022-06-28 13:19 ` Tom Rix 2022-06-28 15:15 ` Daniel Thompson 2 siblings, 1 reply; 6+ messages in thread From: Tom Rix @ 2022-06-28 13:19 UTC (permalink / raw) To: Daniel Thompson, Nathan Chancellor Cc: Masahiro Yamada, Michal Marek, Nick Desaulniers, linux-kbuild, llvm, linux-kernel The limitations of the compilation db are not limited to the kernel. Why shouldn't the change be made in the llvm project ? Tom On 6/28/22 5:27 AM, Daniel Thompson wrote: > This patch set makes it possible to use clang based tools, such as > the clangd completion engine, regardless of our choice of compiler > for the actual kernel build. > > Most of the fine details are included in the patch header for the > first patch (since we'd like them to appear in the kernel history so > they can help future archaeologists). > > Daniel Thompson (2): > clang-tools: Generate clang compatible output even with gcc builds > Makefile: Enable clang-tidy and clang-analyzer for gcc builds > > Makefile | 11 ++-- > scripts/clang-tools/gen_compile_commands.py | 71 ++++++++++++++++++++- > 2 files changed, 74 insertions(+), 8 deletions(-) > > > base-commit: a111daf0c53ae91e71fd2bfe7497862d14132e3e > -- > 2.35.1 > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/2] Enable clang tools for kernels build using gcc 2022-06-28 13:19 ` [PATCH 0/2] Enable clang tools for kernels build using gcc Tom Rix @ 2022-06-28 15:15 ` Daniel Thompson 0 siblings, 0 replies; 6+ messages in thread From: Daniel Thompson @ 2022-06-28 15:15 UTC (permalink / raw) To: Tom Rix Cc: Nathan Chancellor, Masahiro Yamada, Michal Marek, Nick Desaulniers, linux-kbuild, llvm, linux-kernel On Tue, Jun 28, 2022 at 06:19:09AM -0700, Tom Rix wrote: > The limitations of the compilation db are not limited to the kernel. > > Why shouldn't the change be made in the llvm project ? I'd rather answer the question why should it be in the kernel ;-). That's somewhat easier to answer because the kernel does have some unusual properties: 1. It makes heavy use of kernel command line options, especially when compared to most user space components where the default compiler flags are something very simple ("-g -O2" plus include directories). 2. The kernel provides it's own tooling to generate the compilation database. For example, if tools like codechecker added features to their front end to mitigate this problem[1] the kernel wouldn't benefit from them anyway. 3. Rules like `make clang-analyzer` generate *and* consume the compilation database in a single pass. That makes it impossible to run fixup tools between generating the database and consuming it. However going back to the original question: > Why shouldn't the change be made in the llvm project ? clang can (and probably will) learn about some of the newer gcc options over time. However that doesn't actually help much since all that achieves is a game of whack-a-mole where clang has to constantly play catch up or the tooling will break. Some other aspects could be added to the various different projects that use the compilation database that would improve things (better automatic handling of --target for example). That sort of change should still peacefully coexist with an approach based on gen_compile_commands.json though. Daniel. [1] Currently they don't workaround it automatically. They just imply that you may have to "fine tune" compile_commands.json after generating it: https://codechecker.readthedocs.io/en/latest/usage/#step-7-fine-tune-analysis-configuration > On 6/28/22 5:27 AM, Daniel Thompson wrote: > > This patch set makes it possible to use clang based tools, such as > > the clangd completion engine, regardless of our choice of compiler > > for the actual kernel build. > > > > Most of the fine details are included in the patch header for the > > first patch (since we'd like them to appear in the kernel history so > > they can help future archaeologists). > > > > Daniel Thompson (2): > > clang-tools: Generate clang compatible output even with gcc builds > > Makefile: Enable clang-tidy and clang-analyzer for gcc builds > > > > Makefile | 11 ++-- > > scripts/clang-tools/gen_compile_commands.py | 71 ++++++++++++++++++++- > > 2 files changed, 74 insertions(+), 8 deletions(-) > > > > > > base-commit: a111daf0c53ae91e71fd2bfe7497862d14132e3e > > -- > > 2.35.1 > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-06-28 18:35 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-06-28 12:27 [PATCH 0/2] Enable clang tools for kernels build using gcc Daniel Thompson 2022-06-28 12:27 ` [PATCH 1/2] clang-tools: Generate clang compatible output even with gcc builds Daniel Thompson 2022-06-28 18:35 ` Nick Desaulniers 2022-06-28 12:27 ` [PATCH 2/2] Makefile: Enable clang-tidy and clang-analyzer for " Daniel Thompson 2022-06-28 13:19 ` [PATCH 0/2] Enable clang tools for kernels build using gcc Tom Rix 2022-06-28 15:15 ` Daniel Thompson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).