Re: [PATCH] scripts: add a tool to produce a compile_commands.json file

[Tom Roeder <tmroeder@google.com>]

On Tue, Dec 18, 2018 at 11:17:35AM +0900, Masahiro Yamada wrote:
> On Tue, Dec 18, 2018 at 8:21 AM Tom Roeder <tmroeder@google.com> wrote:
> >
> > On Sat, Dec 15, 2018 at 06:37:49PM +0900, Masahiro Yamada wrote:
> > > On Fri, Dec 7, 2018 at 7:24 AM Tom Roeder <tmroeder@google.com> wrote:
> > > >
> > > > The LLVM/Clang project provides many tools for analyzing C source code.
> > > > Many of these tools are based on LibTooling
> > > > (https://clang.llvm.org/docs/LibTooling.html), which depends on a
> > > > database of compiler flags. The standard container for this database is
> > > > compile_commands.json, which consists of a list of JSON objects, each
> > > > with "directory", "file", and "command" fields.
> > > >
> > > > Some build systems, like cmake or bazel, produce this compilation
> > > > information directly. Naturally, Makefiles don't. However, the kernel
> > > > makefiles already create .<target>.o.cmd files that contain all the
> > > > information needed to build a compile_commands.json file.
> > > >
> > > > So, this commit adds scripts/gen_compile_commands.py, which recursively
> > > > searches through a directory for .<target>.o.cmd files and extracts
> > > > appropriate compile commands from them. It writes a
> > > > compile_commands.json file that LibTooling-based tools can use.
> > > >
> > > > By default, gen_compile_commands.py starts its search in its working
> > > > directory and (over)writes compile_commands.json in the working
> > > > directory. However, it also supports --output and --directory flags for
> > > > out-of-tree use.
> > > >
> > > > Note that while gen_compile_commands.py enables the use of clang-based
> > > > tools, it does not require the kernel to be compiled with clang. E.g.,
> > > > the following sequence of commands produces a compile_commands.json file
> > > > that works correctly with LibTooling.
> > > >
> > > > make defconfig
> > > > make
> > > > scripts/gen_compile_commands.py
> > > >
> > > > Also note that this script is written to work correctly in both Python 2
> > > > and Python 3, so it does not specify the Python version in its first
> > > > line.
> > > >
> > > > For an example of the utility of this script: after running
> > > > gen_compile_commands.json on the latest kernel version, I was able to
> > > > use Vim + the YouCompleteMe pluging + clangd to automatically jump to
> > > > definitions and declarations. Obviously, cscope and ctags provide some
> > > > of this functionality; the advantage of supporting LibTooling is that it
> > > > opens the door to many other clang-based tools that understand the code
> > > > directly and do not rely on regular expressions and heuristics.
> > > >
> > > > Tested: Built several recent kernel versions and ran the script against
> > > > them, testing tools like clangd (for editor/LSP support) and clang-check
> > > > (for static analysis). Also extracted some test .cmd files from a kernel
> > > > build and wrote a test script to check that the script behaved correctly
> > > > with all permutations of the --output and --directory flags.
> > > >
> > > > Signed-off-by: Tom Roeder <tmroeder@google.com>
> > >
> > >
> > > I am fine with this,
> > > but I have one question.
> > >
> > > The generated compile_commands.json
> > > contains $(pound)
> >
> > To make sure we're talking about the same thing: the instances that I've
> > seen of "#" occur in macro definitions in the "command" field in some of
> > the JSON objects. For example, I see things like
> > -D\"KBUILD_STR(s)=\\#s\".
> 
> 
> 
> When I ran this tool against the latest kernel
> (specifically, since commit 9564a8cf)
> I saw the following in "command" field.
> 
> -D\"BUILD_STR(s)=$(pound)s\"
> 
> 
> I am not sure whether it is a problem or not.
> 
> I do not care about this tool much.
> I will queue up this patch shortly if it is OK with you.

Fixed now and sent out in a v2 patch. I'm new to the kernel mailing
lists, and I don't know if I'm supposed to link the v2 patch to this
thread. From a glance at the archives, it looked to me like new versions
of patches start their own threads, so that's what I've done here.

> 
> 
> Thanks.
> 
> 
> > >
> > > How is it handled?
> >
> > The Python json module takes care of escaping the output to make a valid
> > JSON string for the "command" field. The gen_compile_commands.py script
> > doesn't take any special action for that or any other character in its
> > output.
> >
> > > Should it be replaced with '\#' ?
> >
> > I don't think it needs to be changed, given my experience with this
> > script and its testing so far: the output seems to work for me. However,
> > are you running into problems due to the presence of this character or
> > inadequate escaping? Please let me know, and I'd be happy to look into
> > it.
> >
> > Tom
> 
> 
> 
> --
> Best Regards
> Masahiro Yamada