Re: [PATCH] scripts: gtags_files_generator.sh

Re: [PATCH] scripts: gtags_files_generator.sh

[Greg KH]

On Thu, Apr 23, 2020 at 05:53:31PM +0800, xujialu wrote:
> Signed-off-by: xujialu <xujialu@vimux.org>
> ---
>  scripts/gtags_files_generator.sh | 34 ++++++++++++++++++++++++++++++++
>  1 file changed, 34 insertions(+)
>  create mode 100755 scripts/gtags_files_generator.sh

I can't accept any patch without a changelog text.

Re: [PATCH] scripts: gtags_files_generator.sh

[Greg KH]

On Thu, Apr 23, 2020 at 10:36:42PM +0800, xujialu wrote:
> A script for generating gtags.files after compilation.

That's a very short description, why isn't that also in the tool itself,
how is anyone going to know what it is, or how to use it?

And this says what it is, but not _why_ it would be useful at all,
please document that here.

thanks,

greg k-h

Re: [PATCH] scripts: gtags_files_generator.sh

[Jonathan Corbet]

On Thu, 23 Apr 2020 22:36:42 +0800
xujialu <xujialu@vimux.org> wrote:

> A script for generating gtags.files after compilation.
> 
> Signed-off-by: xujialu <xujialu@vimux.org>

Please, try to help out us poor folks who aren't a part of your world and
provide a proper changelog here.  What does this script do, why would one
want a "gtags.files" file, how does one run it, etc?  Additionally, the
script itself should have a comment describing what it does and how to run
it - somebody stumbling across it in 2030 may not just know that stuff.

Thanks,

jon

Re: [PATCH] scripts: gtags_files_generator.sh

[Greg KH]

On Sat, Apr 25, 2020 at 03:31:06PM +0800, xujialu wrote:
> Add a script to generate a gtags.files file.
> 
> For navigating linux sources, it will be more efficient if gtags/cscope just
> collects source files that needed by compilation. The kernel makefiles already
> create *.cmd files that contain the files we needed, then just extracts files
> list from them and into gtags.files cause it's the default name list file for
> gtags.
> 
> make defconfig
> make
> scripts/gtags_files_generator.sh
> gtags [-f gtags.files]
> 
> Enjoy with vim+gtags. :)

What's wrong with just 'make gtags' that we currently have in the kernel
tree?  Shouldn't that be sufficient, and if not, then you need to
explain why it isn't in your changelog.

thanks,

greg k-h

Re: [PATCH] scripts: gtags_files_generator.sh

[Greg KH]

On Sun, Apr 26, 2020 at 10:44:36AM +0800, xujialu wrote:
> Add a script to generate a more precise gtags.files from *.cmd files.
> 
> For navigating linux sources, it will be more efficient if gtags/cscope
> just collects source files that needed for compilation. The kernel
> makefiles already create *.cmd files that contain the files we needed,
> then just extracts files list from them and into gtags.files cause it's
> the the default name list file for gtags.
> 
> make defconfig
> make
> scripts/gtags_files_generator.sh
> gtags [-f gtags.files]
> 
> Enjoy with vim+gtags. :)
> 
> Here is a log for comparison with 'make gtags':
> 
> $ time make ARCH=arm cscope
>   GEN     cscope
> 
> real	1m20.600s
> user	1m36.004s
> sys	0m8.192s
> $ wc -l cscope.files
> 31201 cscope.files		#collected too many files we don't care
> 
> $ time ./scripts/gtags_files_generator.sh
> 
> Succeed, 3716 gtags.files listed!
> 
> real	0m1.593s		#collected files only we care
> user	0m1.704s
> sys	0m0.256s
> 				#collected also dts and dtsi
> $ grep dts gtags.files
> arch/arm/boot/dts/xxxxxx-clocks.dtsi
> arch/arm/boot/dts/xxxxxx.dtsi
> arch/arm/boot/dts/xxxxxxxx.dts
> arch/arm/boot/dts/xxxxxxxx.dtsi
> 
> Signed-off-by: xujialu <xujialu@vimux.org>
> ---
>  scripts/gtags_files_generator.sh | 48 ++++++++++++++++++++++++++++++++
>  1 file changed, 48 insertions(+)
>  create mode 100755 scripts/gtags_files_generator.sh

As this is a new version, you need to properly version your patch, as is
documented, please do that.

Also, you seem to be ignoring my previous questions/comments, which
makes me want to just ignore this patch :(

greg k-h

Re: [PATCH] scripts: gtags_files_generator.sh

[xujialu]

Hi, greg

I'm really sorry for the bad reply i did, i even don't know what was my email problem ..

I mean i'm very new to git-send-email related, very very confused about the email processing..

Please help me understanding where my problem is, I am very serious in organizing this

submission related email, even every sentence written in the email.

Did i filled the wrong msg-id in --in-reply-to? Or i should directly answer your question by following

format, just like your reply:

        > make defconfig
        > make
        > scripts/gtags_files_generator.sh
        > gtags [-f gtags.files]
        >
        > Enjoy with vim+gtags. :)

        What's wrong with just 'make gtags' that we currently have in the kernel
        tree?  Shouldn't that be sufficient, and if not, then you need to
        explain why it isn't in your changelog.

How do you add '>' before this lines, all by hands?

If I did something wrong, please help me to know it, (oh, my poor english ..)

Thanks! :(

This email is not send by git-send-email.


------------------ Original ------------------
From: "gregkh"<gregkh@linuxfoundation.org>;
Date: Sun, Apr 26, 2020 03:09 PM
To: "xujialu"<xujialu@vimux.org>;
Cc: "corbet"<corbet@lwn.net>;"linux-kernel"<linux-kernel@vger.kernel.org>;"masahiroy"<masahiroy@kernel.org>;"akpm"<akpm@linux-foundation.org>;"mchehab+huawei"<mchehab+huawei@kernel.org>;
Subject: Re: [PATCH] scripts: gtags_files_generator.sh



On Sun, Apr 26, 2020 at 10:44:36AM +0800, xujialu wrote:
> Add a script to generate a more precise gtags.files from *.cmd files.
> 
> For navigating linux sources, it will be more efficient if gtags/cscope
> just collects source files that needed for compilation. The kernel
> makefiles already create *.cmd files that contain the files we needed,
> then just extracts files list from them and into gtags.files cause it's
> the the default name list file for gtags.
> 
> make defconfig
> make
> scripts/gtags_files_generator.sh
> gtags [-f gtags.files]
> 
> Enjoy with vim+gtags. :)
> 
> Here is a log for comparison with 'make gtags':
> 
> $ time make ARCH=arm cscope
>   GEN     cscope
> 
> real	1m20.600s
> user	1m36.004s
> sys	0m8.192s
> $ wc -l cscope.files
> 31201 cscope.files		#collected too many files we don't care
> 
> $ time ./scripts/gtags_files_generator.sh
> 
> Succeed, 3716 gtags.files listed!
> 
> real	0m1.593s		#collected files only we care
> user	0m1.704s
> sys	0m0.256s
> 				#collected also dts and dtsi
> $ grep dts gtags.files
> arch/arm/boot/dts/xxxxxx-clocks.dtsi
> arch/arm/boot/dts/xxxxxx.dtsi
> arch/arm/boot/dts/xxxxxxxx.dts
> arch/arm/boot/dts/xxxxxxxx.dtsi
> 
> Signed-off-by: xujialu <xujialu@vimux.org>
> ---
>  scripts/gtags_files_generator.sh | 48 ++++++++++++++++++++++++++++++++
>  1 file changed, 48 insertions(+)
>  create mode 100755 scripts/gtags_files_generator.sh

As this is a new version, you need to properly version your patch, as is
documented, please do that.

Also, you seem to be ignoring my previous questions/comments, which
makes me want to just ignore this patch :(

greg k-h

Re: [PATCH] scripts: gtags_files_generator.sh

[gregkh]

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I include quotations after my reply?

http://daringfireball.net/2007/07/on_top

On Sun, Apr 26, 2020 at 03:46:26PM +0800, xujialu wrote:
> Hi, greg
> 
> I'm really sorry for the bad reply i did, i even don't know what was my email problem ..
> 
> I mean i'm very new to git-send-email related, very very confused about the email processing..
> 
> Please help me understanding where my problem is, I am very serious in organizing this
> 
> submission related email, even every sentence written in the email.
> 
> Did i filled the wrong msg-id in --in-reply-to? Or i should directly answer your question by following
> 
> format, just like your reply:

Yes, please, just do proper email responses, like all the other
conversations on the mailing lists :)

>         > make defconfig
>         > make
>         > scripts/gtags_files_generator.sh
>         > gtags [-f gtags.files]
>         >
>         > Enjoy with vim+gtags. :)
> 
>         What's wrong with just 'make gtags' that we currently have in the kernel
>         tree?  Shouldn't that be sufficient, and if not, then you need to
>         explain why it isn't in your changelog.
> 
> How do you add '>' before this lines, all by hands?

No, my email client does it, as should yours with a simple "reply".
Perhaps you should read the email client documentation in the kernel
tree?

I still don't know what is wrong with 'make gtags'.  If it does not work
properly for you, why not fix that up instead of creating something
totally new that is not even hooked up to the kernel build system?

thanks,

greg k-h

Re: [PATCH] scripts: gtags_files_generator.sh

[Greg KH]

On Tue, Apr 28, 2020 at 01:35:08PM +0800, xujialu wrote:
> > Yes, please, just do proper email responses, like all the other
> > conversations on the mailing lists :)
> 
> Nice of you :)
> 
> > Perhaps you should read the email client documentation in the kernel
> > tree?
> 
> Also watched your interesting video:
>     <HOWTO - apply a Linux kernel patch to the stable tree>
> 
> > I still don't know what is wrong with 'make gtags'.  If it does not work
> > properly for you, why not fix that up instead of creating something
> > totally new that is not even hooked up to the kernel build system?
> 
> As you suggested, i walk through scripts/tags.sh, and suddenly found
> flag COMPILED_SOURCE, so i throw away my lasted stupid script, and tried
> to improve this 'compiled only' feature, seems much better now.
> 
> And about 'make gtags':
> 1. It picks up so many files, do you think 'all sources' is good for us?

When you modify kernel code, yes, you have to fix it up everywhere, you
can not ignore other architectures or drivers that your machine happens
to not use.

An "average" laptop only runs about 2-3 million lines of code or so, a
small percentage of the 25+ million lines that are in the kernel source
tree.

> 2. Seems gtags does not support source files from parrent directories,
> it gives warnning out of tree (global-6.6.3) or just segmentation fault
> (global-5.7.3) when we 'make gtags O=blabla', cause gtags.files is
> generated in dir 'blabla' but source files are out of dir 'blabla'.

Then fix that as a stand-alone patch please, do not mix it in with other
changes.

thanks,

greg k-h

Re: [PATCH v2] scripts: scripts/tags.sh

[Greg KH]

On Tue, Apr 28, 2020 at 01:35:09PM +0800, xujialu wrote:
> Update function all_compiled_sources for only compiled source files.

Your subject line needs a lot of work.  Please read the section entitled
"The canonical patch format" in the kernel file,
Documentation/SubmittingPatches for what is needed in order to properly
describe the change.

> 
> Usage:
>   1) COMPILED_SOURCE=1                           make {cscope,gtags}
>   2) COMPILED_SOURCE=1 KBUILD_ABS_SRCTREE=1      make {cscope,gtags}
>   3) COMPILED_SOURCE=1              ./scripts/tags.sh {cscope,gtags}
>   4) COMPILED_SOURCE=1 ABSPWD=$PWD/ ./scripts/tags.sh {cscope,gtags}

And you did not document this somewhere for people to be able to find
out?  Also not good :(

greg k-h

Re: [PATCH v4 1/2] scripts: Support compiled source, improved precise

[Masahiro Yamada]

On Sat, May 2, 2020 at 2:27 PM xujialu <xujialu@vimux.org> wrote:
>
> Original 'COMPILED_SOURCE=1 make cscope' collects nearly 30000 files
> include too many unused files, this patch precisely collects source
> files from *.cmd, in this case just 3000 files include dts and dtsi.
>
> Usage:
>   1) COMPILED_SOURCE=1                           make {cscope,gtags}
>   2) COMPILED_SOURCE=1 KBUILD_ABS_SRCTREE=1      make {cscope,gtags}
>   3) COMPILED_SOURCE=1              ./scripts/tags.sh {cscope,gtags}
>   4) COMPILED_SOURCE=1 ABSPWD=$PWD/ ./scripts/tags.sh {cscope,gtags}
>
> Signed-off-by: xujialu <xujialu@vimux.org>
> ---
>  scripts/tags.sh | 36 ++++++++++++++++++++++--------------
>  1 file changed, 22 insertions(+), 14 deletions(-)
>
> diff --git a/scripts/tags.sh b/scripts/tags.sh
> index 4e18ae5282a6..941a5c61d343 100755
> --- a/scripts/tags.sh
> +++ b/scripts/tags.sh
> @@ -89,22 +89,30 @@ all_sources()
>         find_other_sources '*.[chS]'
>  }
>
> +# COMPILED_SOURCE=1                           make {cscope,gtags}
> +# COMPILED_SOURCE=1 KBUILD_ABS_SRCTREE=1      make {cscope,gtags}
> +# COMPILED_SOURCE=1              ./scripts/tags.sh {cscope,gtags}
> +# COMPILED_SOURCE=1 ABSPWD=$PWD/ ./scripts/tags.sh {cscope,gtags}


These comment are misleading since this sounds like
is is only usef for cscope, gtags.


Please do not introduce a new variable ABSPWD, which is unneeded.
This is a rare use-case, but if you want to run this script directly,
you must set the variables described at line 9 properly.




> +xtags_juggle_list()
> +{
> +       SRCTREE=$(realpath ${tree}.)
> +
> +       cd $(dirname $(find -name .config -print -quit).)


Why is this needed?

You are already in objtree
when this script is being run.

If you handle the objects built with O= option,
you need to do 'make O=... gtags'.

> +
> +       realpath -e --relative-to=${SRCTREE} $(find -name "*.cmd" -exec \
> +               grep -Poh '(?(?=^source_.* \K).*|(?=^  \K\S).*(?= \\))' {} \+ |
> +               awk '!a[$0]++') include/generated/autoconf.h |
> +       sed -e "/\.\./d" -e "s,^,${ABSPWD}${tree},"
> +}

Why is --relative-to=${SRCTREE} needed?

You are dropping ${SRCTREE} and adding ${ABSPWD}${tree}.
I do not understand what this is doing back-and-forth.



Lastly, the file order is currently carefully crafted
but this patch would make it random-ordered.

I am afraid the following commit would be broken.




commit f81b1be40c44b33b9706d64c117edd29e627ad12 (HEAD)
Author: Guennadi Liakhovetski <g.liakhovetski@gmx.de>
Date:   Mon Feb 8 00:25:59 2010 +0100

    tags: include headers before source files

    Currently looking up a structure definition in TAGS / tags takes one to
    one of multiple "static struct X" definitions in arch sources, which makes
    it for many structs practically impossible to get to the required header.
    This patch changes the order of sources being tagged to first scan
    architecture includes, then the top-level include/ directory, and only
    then the rest. It also takes into account, that many architectures have
    more than one include directory, i.e., not only arch/$ARCH/include, but
    also arch/$ARCH/mach-X/include etc.

    Signed-off-by: Guennadi Liakhovetski <g.liakhovetski@gmx.de>
    Reviewed-by: WANG Cong <xiyou.wangcong@gmail.com>
    [mmarek@suse.cz: fix 'var+=text' bashism]
    Signed-off-by: Michal Marek <mmarek@suse.cz>

> +
>  all_compiled_sources()
>  {
> -       for i in $(all_sources); do
> -               case "$i" in
> -                       *.[cS])
> -                               j=${i/\.[cS]/\.o}
> -                               j="${j#$tree}"
> -                               if [ -e $j ]; then
> -                                       echo $i
> -                               fi
> -                               ;;
> -                       *)
> -                               echo $i
> -                               ;;
> -               esac
> -       done
> +       # Consider 'git ls-files' features:
> +       #   1) sort and uniq target files
> +       #   2) limit target files by index
> +       # git ls-files $(xtags_juggle_list)


How is this related to this ?






> +
> +       xtags_juggle_list | sort -u
>  }
>
>  all_target_sources()
> --
> 2.20.1
>


--
Best Regards
Masahiro Yamada

Re: [PATCH v4 1/2] scripts: Support compiled source, improved precise

[Masahiro Yamada]

On Fri, May 15, 2020 at 2:10 AM xujialu <xujialu@vimux.org> wrote:
>
> Sorry for replying so late.
>
> <snip>
>
> I usually don't run scripts/tags.sh directly. But one day i checked git
> log of scripts/tags.sh, and found this commit c69ef1c87b8c said we may
> run it directly. Then i must took care of that.
>
> Here are some cases that i should write clearly before:
> (I omit COMPILED_SOURCE=1 here just for clear and distinct)
>
> 1) make; make gtags;
> 2) make; ./scripts/tags.sh gtags;
> 3) make O=123; make O=123 gtags;
> 4) make O=123; make gtags;
> 5) make O=123; ./scripts/tags.sh gtags;
> 6) make O=/path/out/of/kernel/; make O=/path/out/of/kernel/ gtags;
> 7) make O=/path/out/of/kernel/; SOMETHING ./scripts/tags.sh gtags;
>
> Assume that we just change directory into kernel root directory and vim
> a source file:
>
> case 1): We have GTAGS generated in current directory, no problem;
>                 In this case: tree=
> case 2): Same as case 1), no problem;
>                 In this case: tree=

... if you set SRCARCH.

If SRCARCH is unset, it will get a warning.

$ ./scripts/tags.sh  gtags
find: ‘arch/*.[chS]/’: No such file or directory




> case 3): GTAGS is generated in directory 123; Here comes the problem,
>          gtags will give error "Segmentation fault" (eg. global-5.7.1)
>          or give warnning "is out of source tree." (eg. global-6.6.3-2)
>          because 'make O=123' changed to directory 123 and our cute
>          source files is in ../ and gtags seems do not like this path
>          begin with '../', actually it's not an subdiretories for 123;
>          If above situation is not persuasive, then consider one may
>          want generate gtags.files contains files without '../' in
>          kernel root directory, so that gtags could be useful;
>          And this is why case 4) exist, if case 4) is really bad idea
>          then we must have another way to do this - case 5);
>                 In this case: tree=../


This is a problem of GNU Global, not of our build system.


"Warning: ... is out of source tree." is listed in
the known bugs:

https://www.gnu.org/software/global/bugs.html


Do not mess up our script.


> case 4): This is not good when we 'make O=123 distclean';
>                 In this case: tree=../



Of course, "make O=123 distclean" cannot clean up
build artifacts created by "make gtags".

So, what problem are you addressing?



> case 5): Find file '.config' in directory 123, then collect files with
>          path just in current directly; No problem;
>                 In this case: tree=../

You are misunderstanding.

See the comment at line 8.

# Uses the following environment variables:
# SUBARCH, SRCARCH, srctree


If you want to run this script directly,
you must set all the mentioned environment variables correctly,
and also run this script in the correct working directory.

It will create tag files in the current working directory.
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^

Do not change the working directory internally.





> case 6): What if KBUILD_OUTPUT is out of kernel directory? Assume that
>          we get a gtags.files in that directly, in the gtags.files, the
>          file path all begin with full path, guess what, gtags will give
>          the error or warnning described above. Why don't we just
>          generate GTAGS with relative path in kernel root directory?
>                 In this case: tree=/path/out/of/kernel/

This is the intended behavior for the other TAGS, tags, cscope.

Again, this is a problem of GNU global.


The general rule is like this:

Tag files should be always output to the separate object tree
if O= is given, then file paths should point to the source
tree with either relative or absolute paths.

This is because the source tree is not always writable.
The source tree might be delivered in a DVD-ROM, read-only mounted nfs,
or located under /usr/src/ which is installed by distro source package.




>
> >
> > +       SRCTREE=$(realpath ${tree}.)
> > +
> > +       cd $(dirname $(find -name .config -print -quit).)
> >
> > Why is this needed?
>
> In case 5), the path of source files collected in .cmd files is some
> begin with '../' and some with full path, eg. /usr/include/stdio.h, we
> must change to directory 123 as 'make O=123 gtags' does so that we could
> use same method (described bellow) in both cases.

No.
scripts/tags.sh must be run in the object tree in this case.

5) make O=123; cd 123; ../scripts/tags.sh srctree=.. SRCARCH=x86 gtags




> > Why is --relative-to=${SRCTREE} needed?
> >
> > You are dropping ${SRCTREE} and adding ${ABSPWD}${tree}.
> > I do not understand what this is doing back-and-forth.
>
> These .cmd files also contain the default include dir (eg.
> /usr/include/stdio.h) and even compiler's header files, try make
> ARCH=arm.


I know. Probably, that would not happen
because the following patch is queued up.
https://patchwork.kernel.org/patch/11505807/




> We should first collect files with path relative to SRCTREE (which is
> kernel root directory), then we kick out the files which path begin with
> '../' by using sed, and then add ${tree} back:
>
>         realpath --relative-to=${SRCTREE} $(find ...) |
>         sed -e "/\.\./d" -e "s,^,${tree},"
>
> Now following code seems good enough:
>
> SRCTREE=$(realpath ${tree}.)
>
> cd $(dirname $(find -name .config -print -quit).)
>
> realpath -e --relative-to=${SRCTREE} $(find -name "*.cmd" -exec \
>         grep -Poh '(?(?=^source_.* \K).*|(?=^  \K\S).*(?= \\))' {} \+ |
>         awk '!a[$0]++') include/generated/autoconf.h |
> sed -e "/\.\./d" -e "s,^,${tree}," | sort -u
>
> What if one want a gtags.files with fullpath:
>
> 'make COMPILED_SOURCE=1 KBUILD_ABS_SRCTREE=1 gtags' is good, and for
> running scripts/tags.sh directly, i add this ABSPWD there.
>
>         COMPILED_SOURCE=1 ABSPWD=$PWD/ ./scripts/tags.sh gtags
>
> <snip>
>
> Let's get back to case 6) and consider case 7), we now should tell
> scripts/tags.sh that where is KBUILD_OUTPUT, then i added another
> OUTPATH for that:
>
>         OUTPATH=/path/to/KBUILD_OUTPUT ./scripts/tags.sh gtags;
>
> Now we have this code:
>
> SRCTREE=$(realpath ${tree}.)
>
> cd $(dirname $(find ${OUTPATH} -name .config -print -quit).)
>
> realpath -e --relative-to=${SRCTREE} $(find -name "*.cmd" -exec \
>         grep -Poh '(?(?=^source_.* \K).*|(?=^  \K\S).*(?= \\))' {} \+ |
>         awk '!a[$0]++') ${OUTPATH}include/generated/autoconf.h |
> sed -e "/\.\./d" -e "s,^,${ABSPWD}${tree}," | sort -u
>
> But this is the real rare use-case, we should forget about it.
>
> > Please do not introduce a new variable ABSPWD, which is unneeded.
> > This is a rare use-case, but if you want to run this script directly,
> > you must set the variables described at line 9 properly.
>
> Usages i wrote has been deleted. ABSPWD has been added at line 9.
> Maybe we could make it a secret to kbuild system cause it's just useful
> for running scripts/tags.sh directly.
>
> > > +       # Consider 'git ls-files' features:
> > > +       #   1) sort and uniq target files
> > > +       #   2) limit target files by index
> > > +       # git ls-files $(xtags_juggle_list)
> >
> > How is this related to this ?
>
> This useless comment has been deleted. I just thought maybe it's useful
> for someone who wants to make these codes better. Actually git ls-files
> takes too much time.
>
> <snip>
>
> Following is something about gtags.files.
>
> > And, the reason is described here:
> > https://bugzilla.kernel.org/show_bug.cgi?id=3D1948
> > So, scope.files has a good reason to exist.
> > gtags.files does not.
>
> I did a 'make cscope' test without cscope.files on Monday, same error as
> the reason described.
>
> Before that, I thought cscope.files is just for convenience when someone
> want to check which files were compiled, do 'vim cscope.files' to get an
> overview of these files and it's easy to go into a source file from vim,
> especially now we could get a percise cscope.files as [patch 1/2] said.
>
> The latest release date of cscope is 2012-08-04, then i tried gtags
> which is growing up, even though cscope works better than gtags in some
> case. For convenience described above, i tried to add a gtags.files.
> But now, let's forget it for a while. :)



I see bad engineering overall, and
it is breaking
f81b1be40c44b33b9706d64c117edd29e627ad12

So, this is NACK.



--
Best Regards
Masahiro Yamada

Re: [PATCH v4 2/2] scripts: Add a intermediate file for 'make gtags'

[Masahiro Yamada]

On Sat, May 2, 2020 at 2:34 PM xujialu <xujialu@vimux.org> wrote:
>
> As 'GTAGS Manual' said: If ´gtags.files´ exists in the current directory
> directory or a file is specified by the -f option, target files are
> limited by it.
>
> So add gtags.files just like cscope.files.
>
> Signed-off-by: xujialu <xujialu@vimux.org>
> ---
>  Makefile        | 2 +-
>  scripts/tags.sh | 3 ++-
>  2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 679f302a8b8b..2c8304ae103f 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1398,7 +1398,7 @@ MRPROPER_FILES += .config .config.old .version \
>
>  # Directories & files removed with 'make distclean'
>  DISTCLEAN_DIRS  +=
> -DISTCLEAN_FILES += tags TAGS cscope* GPATH GTAGS GRTAGS GSYMS
> +DISTCLEAN_FILES += tags TAGS cscope* GPATH GTAGS GRTAGS GSYMS gtags.files
>
>  # clean - Delete most, but leave enough to build external modules
>  #
> diff --git a/scripts/tags.sh b/scripts/tags.sh
> index 941a5c61d343..e2edd0cf67ec 100755
> --- a/scripts/tags.sh
> +++ b/scripts/tags.sh
> @@ -142,7 +142,8 @@ docscope()
>
>  dogtags()
>  {
> -       all_target_sources | gtags -i -f -
> +       all_target_sources > gtags.files
> +       gtags -i -f gtags.files
>  }

My previous reply was not sent to the list.

Here is the summary of my view.



With my basic knowledge of UNIX
such as the concept of pipe, redirect, etc.,
the following two commands should do the same work.


[1] { echo file lists; }  |  gtags  -f  -

[2] { echo file lists; }  > tmpfile
    gtags -f  tmpfile





The gtags manual also says:
-f, --file file
       Give  a  list of candidates of target files.
       Files which are not on the list are ignored.
       The argument file can be set to ´-´ to accept
       a list of files from the standard input.



This patch looks no point to me.




As for the cscope.files, we had a reason to have it separately.


This commit:

https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/commit/?id=3cd681bae4360e2f67ef9b232db28e13319f1c17


And, the bug was elaborated here:

https://bugzilla.kernel.org/show_bug.cgi?id=1948







--
Best Regards
Masahiro Yamada

Re: [PATCH] scripts: gtags_files_generator.sh

[Masahiro Yamada]

On Sun, Apr 26, 2020 at 11:48 AM xujialu <xujialu@vimux.org> wrote:
>
> Add a script to generate a more precise gtags.files from *.cmd files.
>
> For navigating linux sources, it will be more efficient if gtags/cscope
> just collects source files that needed for compilation. The kernel
> makefiles already create *.cmd files that contain the files we needed,
> then just extracts files list from them and into gtags.files cause it's
> the the default name list file for gtags.


This is unneeded.

scripts/tags.sh supports COMPILED_SOURCE=1
to collect tags only from compiled sources.

Please do not reinvent the wheel
to achieve similar things.


$ time make COMPILED_SOURCE=1 gtags
  GEN     gtags

real 0m7.623s
user 0m7.459s
sys 0m1.124s


is fine for me.

If the existing code is problematic,
we need to improve it,
not add the second one.





>
> make defconfig
> make
> scripts/gtags_files_generator.sh
> gtags [-f gtags.files]
>
> Enjoy with vim+gtags. :)
>
> Here is a log for comparison with 'make gtags':
>
> $ time make ARCH=arm cscope
>   GEN     cscope
>
> real    1m20.600s
> user    1m36.004s
> sys     0m8.192s
> $ wc -l cscope.files
> 31201 cscope.files              #collected too many files we don't care
>
> $ time ./scripts/gtags_files_generator.sh
>
> Succeed, 3716 gtags.files listed!
>
> real    0m1.593s                #collected files only we care
> user    0m1.704s
> sys     0m0.256s
>                                 #collected also dts and dtsi


Why do you need to collect dts and dtsi?







> $ grep dts gtags.files
> arch/arm/boot/dts/xxxxxx-clocks.dtsi
> arch/arm/boot/dts/xxxxxx.dtsi
> arch/arm/boot/dts/xxxxxxxx.dts
> arch/arm/boot/dts/xxxxxxxx.dtsi
>
> Signed-off-by: xujialu <xujialu@vimux.org>
> ---
>  scripts/gtags_files_generator.sh | 48 ++++++++++++++++++++++++++++++++
>  1 file changed, 48 insertions(+)
>  create mode 100755 scripts/gtags_files_generator.sh
>
> diff --git a/scripts/gtags_files_generator.sh b/scripts/gtags_files_generator.sh
> new file mode 100755
> index 000000000000..04698d9234a7
> --- /dev/null
> +++ b/scripts/gtags_files_generator.sh
> @@ -0,0 +1,48 @@
> +#!/bin/bash -e
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Author: xujialu <xujialu@vimux.org>
> +#
> +# A script for generating gtags.files after compilation.
> +#
> +# Usage:
> +#      make defconfig
> +#      make
> +#      scripts/gtags_files_generator.sh
> +#      gtags [-f gtags.files]
> +
> +
> +gtags_files_list()
> +{
> +       ls ${AUTOCONF_H} $(realpath -e --relative-to=. \
> +               $(find ${KBUILD_OUTPUT} -name "*$1.cmd" -exec \
> +               grep -Poh '(?(?=^source_.* \K).*|(?=^  \K\S).*(?= \\))' {} \+ |
> +               awk '!a[$0]++' | sed "s,^,${KBUILD_OUTPUT},") |
> +               sed -e '/\.\./d' | awk '!a[$0]++'
> +       )
> +}
> +
> +gtags_files_generator()
> +{
> +       GTAGS_FILES=$PWD/gtags.files
> +
> +       echo && [ $# = 1 ] && cd $1
> +
> +       SYSTEM_MAP=$(find -name System.map -print -quit)
> +
> +       [ -z "${SYSTEM_MAP}" ] && echo "Failed, please compile first!" && exit 1
> +
> +       KBUILD_OUTPUT=${SYSTEM_MAP%/*}/
> +       KBUILD_OUTPUT=${KBUILD_OUTPUT#*/}
> +
> +       AUTOCONF_H=${KBUILD_OUTPUT}include/generated/autoconf.h
> +
> +       grep Configuration ${AUTOCONF_H} | sed 's/ \* //'
> +       gtags_files_list > ${GTAGS_FILES}
> +
> +       echo && [ $# = 1 ] && sed -i "s|^|$1|" ${GTAGS_FILES}
> +
> +       echo "Succeed, $(wc -l ${GTAGS_FILES}) listed!"
> +}
> +
> +gtags_files_generator $1 2>/dev/null
> --
> 2.20.1
>


-- 
Best Regards
Masahiro Yamada