[PATCH] gen_compile_commands: include targets not end with .o

[PATCH] gen_compile_commands: include targets not end with .o

[Hu Weiwen]

From: Hu Weiwen <sehuww@mail.scut.edu.cn>

Currently, we only extract commands for targets end with '.o'. But we
also have many standalone executables built in-tree.

Remove this restriction. And to avoid some false matching, exclude
targets end with '.c' or '.h' when directly walking the directory.

Signed-off-by: Hu Weiwen <sehuww@mail.scut.edu.cn>
---
 scripts/clang-tools/gen_compile_commands.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
index 15ba56527acd..6e88c7e166fc 100755
--- a/scripts/clang-tools/gen_compile_commands.py
+++ b/scripts/clang-tools/gen_compile_commands.py
@@ -18,8 +18,8 @@ import sys
 _DEFAULT_OUTPUT = 'compile_commands.json'
 _DEFAULT_LOG_LEVEL = 'WARNING'
 
-_FILENAME_PATTERN = r'^\..*\.cmd$'
-_LINE_PATTERN = r'^savedcmd_[^ ]*\.o := (.* )([^ ]*\.c) *(;|$)'
+_FILENAME_PATTERN = r'^\..*(?<!\.(c|h))\.cmd$'
+_LINE_PATTERN = r'^savedcmd_[^ ]* := (.* )([^ ]*\.c) *(;|$)'
 _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
 # The tools/ directory adopts a different build system, and produces .cmd
 # files in a different format. Do not support it.
-- 
2.25.1

Re: [PATCH] gen_compile_commands: include targets not end with .o

[Nick Desaulniers]

On Sun, May 14, 2023 at 3:19 AM Hu Weiwen <huww98@outlook.com> wrote:
>
> From: Hu Weiwen <sehuww@mail.scut.edu.cn>

Hi Hu,
Sorry, for the delay. I'm usually faster to respond to patches. It was
my mistake this fell through the cracks, and thank you for the patch.

I received this email from <huww98@outlook.com>. Do you mind resending
with your signed off by tag set to the email address from which you
are sending patches?  This patch looks benign to me, but it makes me a
little uneasy when author != sender.

Please also cc
- Masahiro Yamada <masahiroy@kernel.org>
- linux-kbuild@vger.kernel.org

As this would go in via the kbuild tree.

>
> Currently, we only extract commands for targets end with '.o'. But we

s/end/ending/


With this patch applies\d, I get the same word count for
compile_commands.json when running:
$ make LLVM=1 -j128 defconfig compile_commands.json

on my x86_64 host. Is that expected?  Is there a specific arch or set
of configs for which such .c files produce executables directly?

> also have many standalone executables built in-tree.
>
> Remove this restriction. And to avoid some false matching, exclude
> targets end with '.c' or '.h' when directly walking the directory.
>
> Signed-off-by: Hu Weiwen <sehuww@mail.scut.edu.cn>
> ---
>  scripts/clang-tools/gen_compile_commands.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> index 15ba56527acd..6e88c7e166fc 100755
> --- a/scripts/clang-tools/gen_compile_commands.py
> +++ b/scripts/clang-tools/gen_compile_commands.py
> @@ -18,8 +18,8 @@ import sys
>  _DEFAULT_OUTPUT = 'compile_commands.json'
>  _DEFAULT_LOG_LEVEL = 'WARNING'
>
> -_FILENAME_PATTERN = r'^\..*\.cmd$'
> -_LINE_PATTERN = r'^savedcmd_[^ ]*\.o := (.* )([^ ]*\.c) *(;|$)'
> +_FILENAME_PATTERN = r'^\..*(?<!\.(c|h))\.cmd$'
> +_LINE_PATTERN = r'^savedcmd_[^ ]* := (.* )([^ ]*\.c) *(;|$)'
>  _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
>  # The tools/ directory adopts a different build system, and produces .cmd
>  # files in a different format. Do not support it.
> --
> 2.25.1
>
>


-- 
Thanks,
~Nick Desaulniers

Re: [PATCH] gen_compile_commands: include targets not end with .o

[胡玮文]

On Wed, May 24, 2023 at 12:10:29PM -0700, Nick Desaulniers wrote:
> On Sun, May 14, 2023 at 3:19 AM Hu Weiwen <huww98@outlook.com> wrote:
> >
> > From: Hu Weiwen <sehuww@mail.scut.edu.cn>
> 
> Hi Hu,
> Sorry, for the delay. I'm usually faster to respond to patches. It was
> my mistake this fell through the cracks, and thank you for the patch.
> 
> I received this email from <huww98@outlook.com>. Do you mind resending
> with your signed off by tag set to the email address from which you
> are sending patches?  This patch looks benign to me, but it makes me a
> little uneasy when author != sender.
> 
> Please also cc
> - Masahiro Yamada <masahiroy@kernel.org>
> - linux-kbuild@vger.kernel.org
> 
> As this would go in via the kbuild tree.

OK, will resend.

> >
> > Currently, we only extract commands for targets end with '.o'. But we
> 
> s/end/ending/
> 
> 
> With this patch applies\d, I get the same word count for
> compile_commands.json when running:
> $ make LLVM=1 -j128 defconfig compile_commands.json
> 
> on my x86_64 host. Is that expected?  Is there a specific arch or set
> of configs for which such .c files produce executables directly?

`make compile_commands.json' will invoke this script as 
`gen_compile_commands.py ... vmlinux.a modules.order'. So only '.o'
targets are included.  To include .c files that produce executable, run
the script directly after a successful build:
$ python3 scripts/clang-tools/gen_compile_commands.py

Some examples of such file:
* scripts/unifdef.c
* scripts/asn1_compiler.c
* scripts/kallsyms.c
* scripts/basic/fixdep.c
* samples/connector/ucon.c

Thanks,
Hu Weiwen

> > also have many standalone executables built in-tree.
> >
> > Remove this restriction. And to avoid some false matching, exclude
> > targets end with '.c' or '.h' when directly walking the directory.
> >
> > Signed-off-by: Hu Weiwen <sehuww@mail.scut.edu.cn>
> > ---
> >  scripts/clang-tools/gen_compile_commands.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> > index 15ba56527acd..6e88c7e166fc 100755
> > --- a/scripts/clang-tools/gen_compile_commands.py
> > +++ b/scripts/clang-tools/gen_compile_commands.py
> > @@ -18,8 +18,8 @@ import sys
> >  _DEFAULT_OUTPUT = 'compile_commands.json'
> >  _DEFAULT_LOG_LEVEL = 'WARNING'
> >
> > -_FILENAME_PATTERN = r'^\..*\.cmd$'
> > -_LINE_PATTERN = r'^savedcmd_[^ ]*\.o := (.* )([^ ]*\.c) *(;|$)'
> > +_FILENAME_PATTERN = r'^\..*(?<!\.(c|h))\.cmd$'
> > +_LINE_PATTERN = r'^savedcmd_[^ ]* := (.* )([^ ]*\.c) *(;|$)'
> >  _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
> >  # The tools/ directory adopts a different build system, and produces .cmd
> >  # files in a different format. Do not support it.
> > --
> > 2.25.1
> >
> >
> 
> 
> -- 
> Thanks,
> ~Nick Desaulniers

[PATCH v2] gen_compile_commands: include targets not ending with .o

[Hu Weiwen]

Currently, we only extract commands for targets ending with '.o'. But we
also have many standalone executables built in-tree.

Remove this restriction. And to avoid some false matching, exclude
targets end with '.c' or '.h' when directly walking the directory.

To really generate compile_commands.json that includes such target, call
this script directly with no arguments.  `make compile_commands.json`
will not include them.

Signed-off-by: Hu Weiwen <huww98@outlook.com>
---
Since V1: Only commit message is updated.

 scripts/clang-tools/gen_compile_commands.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
index 15ba56527acd..6e88c7e166fc 100755
--- a/scripts/clang-tools/gen_compile_commands.py
+++ b/scripts/clang-tools/gen_compile_commands.py
@@ -18,8 +18,8 @@ import sys
 _DEFAULT_OUTPUT = 'compile_commands.json'
 _DEFAULT_LOG_LEVEL = 'WARNING'
 
-_FILENAME_PATTERN = r'^\..*\.cmd$'
-_LINE_PATTERN = r'^savedcmd_[^ ]*\.o := (.* )([^ ]*\.c) *(;|$)'
+_FILENAME_PATTERN = r'^\..*(?<!\.(c|h))\.cmd$'
+_LINE_PATTERN = r'^savedcmd_[^ ]* := (.* )([^ ]*\.c) *(;|$)'
 _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
 # The tools/ directory adopts a different build system, and produces .cmd
 # files in a different format. Do not support it.
-- 
2.25.1

Re: [PATCH v2] gen_compile_commands: include targets not ending with .o

[Masahiro Yamada]

On Thu, May 25, 2023 at 9:21 PM Hu Weiwen <huww98@outlook.com> wrote:
>
> Currently, we only extract commands for targets ending with '.o'. But we
> also have many standalone executables built in-tree.
>
> Remove this restriction. And to avoid some false matching, exclude
> targets end with '.c' or '.h' when directly walking the directory.


Can you give me some examples of false matching?



Also, this patch would pick up *.i, *.s, *.lst, etc.

Does it make sense to parse cmd files for those?
I guess the answer is no.



>
> To really generate compile_commands.json that includes such target, call
> this script directly with no arguments.  `make compile_commands.json`
> will not include them.
>
> Signed-off-by: Hu Weiwen <huww98@outlook.com>
> ---
> Since V1: Only commit message is updated.
>
>  scripts/clang-tools/gen_compile_commands.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> index 15ba56527acd..6e88c7e166fc 100755
> --- a/scripts/clang-tools/gen_compile_commands.py
> +++ b/scripts/clang-tools/gen_compile_commands.py
> @@ -18,8 +18,8 @@ import sys
>  _DEFAULT_OUTPUT = 'compile_commands.json'
>  _DEFAULT_LOG_LEVEL = 'WARNING'
>
> -_FILENAME_PATTERN = r'^\..*\.cmd$'
> -_LINE_PATTERN = r'^savedcmd_[^ ]*\.o := (.* )([^ ]*\.c) *(;|$)'
> +_FILENAME_PATTERN = r'^\..*(?<!\.(c|h))\.cmd$'
> +_LINE_PATTERN = r'^savedcmd_[^ ]* := (.* )([^ ]*\.c) *(;|$)'
>  _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
>  # The tools/ directory adopts a different build system, and produces .cmd
>  # files in a different format. Do not support it.
> --
> 2.25.1
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2] gen_compile_commands: include targets not ending with .o

[胡玮文]

On Mon, May 29, 2023 at 07:13:46PM +0900, Masahiro Yamada wrote:
> On Thu, May 25, 2023 at 9:21 PM Hu Weiwen <huww98@outlook.com> wrote:
> >
> > Currently, we only extract commands for targets ending with '.o'. But we
> > also have many standalone executables built in-tree.
> >
> > Remove this restriction. And to avoid some false matching, exclude
> > targets end with '.c' or '.h' when directly walking the directory.
> 
> 
> Can you give me some examples of false matching?

Examples:
- drivers/scsi/scsi_devinfo_tbl.c: generated by
  sed -n 's/.*define *BLIST_\\([A-Z0-9_]*\\) *.*/BLIST_FLAG_NAME(\\1),/p' include/scsi/scsi_devinfo.h > drivers/scsi/scsi_devinfo_tbl.c

- arch/x86/entry/vdso/vdso-image-32.c: generated by
  arch/x86/entry/vdso/vdso2c arch/x86/entry/vdso/vdso32.so.dbg arch/x86/entry/vdso/vdso32.so arch/x86/entry/vdso/vdso-image-32.c

I think only .c file targets would have false matching, because the cmd
must ends with '.c' to be matched.

> 
> Also, this patch would pick up *.i, *.s, *.lst, etc.
> 
> Does it make sense to parse cmd files for those?
> I guess the answer is no.

We are already parsing cmd files for those before this patch. I use more
strict `_FILENAME_PATTERN'. But it may be hard to exclude all of these
by filename.

Should we use a more strict `_LINE_PATTERN'? e.g., require the cmd to
begin with a compiler, not `sed' or something else. But we don't know
the compiler name, adding a parameter for compiler name may break the
existing usage.

> >
> > To really generate compile_commands.json that includes such target, call
> > this script directly with no arguments.  `make compile_commands.json`
> > will not include them.
> >
> > Signed-off-by: Hu Weiwen <huww98@outlook.com>
> > ---
> > Since V1: Only commit message is updated.
> >
> >  scripts/clang-tools/gen_compile_commands.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> > index 15ba56527acd..6e88c7e166fc 100755
> > --- a/scripts/clang-tools/gen_compile_commands.py
> > +++ b/scripts/clang-tools/gen_compile_commands.py
> > @@ -18,8 +18,8 @@ import sys
> >  _DEFAULT_OUTPUT = 'compile_commands.json'
> >  _DEFAULT_LOG_LEVEL = 'WARNING'
> >
> > -_FILENAME_PATTERN = r'^\..*\.cmd$'
> > -_LINE_PATTERN = r'^savedcmd_[^ ]*\.o := (.* )([^ ]*\.c) *(;|$)'
> > +_FILENAME_PATTERN = r'^\..*(?<!\.(c|h))\.cmd$'
> > +_LINE_PATTERN = r'^savedcmd_[^ ]* := (.* )([^ ]*\.c) *(;|$)'
> >  _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
> >  # The tools/ directory adopts a different build system, and produces .cmd
> >  # files in a different format. Do not support it.
> > --
> > 2.25.1
> >
> 
> 
> -- 
> Best Regards
> Masahiro Yamada

Re: [PATCH v2] gen_compile_commands: include targets not ending with .o

[Masahiro Yamada]

On Tue, May 30, 2023 at 6:04 PM 胡玮文 <huww98@outlook.com> wrote:
>
> On Mon, May 29, 2023 at 07:13:46PM +0900, Masahiro Yamada wrote:
> > On Thu, May 25, 2023 at 9:21 PM Hu Weiwen <huww98@outlook.com> wrote:
> > >
> > > Currently, we only extract commands for targets ending with '.o'. But we
> > > also have many standalone executables built in-tree.
> > >
> > > Remove this restriction. And to avoid some false matching, exclude
> > > targets end with '.c' or '.h' when directly walking the directory.
> >
> >
> > Can you give me some examples of false matching?
>
> Examples:
> - drivers/scsi/scsi_devinfo_tbl.c: generated by
>   sed -n 's/.*define *BLIST_\\([A-Z0-9_]*\\) *.*/BLIST_FLAG_NAME(\\1),/p' include/scsi/scsi_devinfo.h > drivers/scsi/scsi_devinfo_tbl.c
>
> - arch/x86/entry/vdso/vdso-image-32.c: generated by
>   arch/x86/entry/vdso/vdso2c arch/x86/entry/vdso/vdso32.so.dbg arch/x86/entry/vdso/vdso32.so arch/x86/entry/vdso/vdso-image-32.c
>
> I think only .c file targets would have false matching, because the cmd
> must ends with '.c' to be matched.

I see.


>
> >
> > Also, this patch would pick up *.i, *.s, *.lst, etc.
> >
> > Does it make sense to parse cmd files for those?
> > I guess the answer is no.
>
> We are already parsing cmd files for those before this patch.

True, but they are excluded by _LINE_PATTERN.


> I use more
> strict `_FILENAME_PATTERN'. But it may be hard to exclude all of these
> by filename.

I think so.


> Should we use a more strict `_LINE_PATTERN'? e.g., require the cmd to
> begin with a compiler, not `sed' or something else. But we don't know
> the compiler name, adding a parameter for compiler name may break the
> existing usage.


Standalone executables are host programs.
I do not think it is worthwhile complicating the code.




-- 
Best Regards
Masahiro Yamada