Re: [PATCH 3/3] objtool: move tools/objtool/ to scripts/objtool/

[Masahiro Yamada <yamada.masahiro@socionext.com>]

On Tue, Mar 5, 2019 at 11:46 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Tue, Mar 05, 2019 at 02:48:16PM +0900, Masahiro Yamada wrote:
> > Tools that are globally needed for building the kernel are supposed to
> > be put in the scripts/ directory, but objtool is the exceptional one.
> >
> > People were confused by objtool's Makefile since most of tools run on
> > the target whereas objtool runs on the host machine. Some time ago,
> > Douglas Anderson suggested to move tools/objtool/ to scripts/tools/
> > (https://patchwork.kernel.org/patch/9983535/#20996057), but we did
> > not go as far as submitting a patch at that time.
> >
> > Recently, Robin Meijboom reported that the generated files for objtool
> > were never cleaned up. (https://patchwork.kernel.org/patch/10813797/)
> >
> > So, I looked into this to fix the distortion. Move tools/objtool/ to
> > scripts/objtool/, and rewrite Makefiles in the Kbuild-ish manner.
> >
> > I also cleaned up the top-level Makefile by moving the libelf check
> > scripts/objtool/Makefile.
> >
> > I see some benefits in this.
> >
> > [1] generated files under scripts/ are cleaned up by 'make mrproper'
> >
> > Currently, the generated files under tools/objtool/ are not cleaned up
> > since the build system under tools/ is a complete different world.
> >
> > If we want to clean them up, it should be done by 'make mrproper'
> > instead of 'make clean' since objtool is needed for building external
> > modules.
> >
> > This commit naturally solves the issue since Kbuild cleans under
> > scripts/ by 'make mrproper'.
>
> Yes, but this can be solved instead with a one-line patch.
>
> > [2] build log looks nicer
> >
> > Currently, the build log under tools/objtool/ shows absolute paths, and
> > displays 'DESCEND  objtool' every time for incremental building.
> >
> > Switching over to the standard Kbuild scheme will improve the log.
> >
> >   Before:
> >
> >   DESCEND  objtool
> >   HOSTCC   /home/masahiro/workspace/linux/tools/objtool/fixdep.o
> >   HOSTLD   /home/masahiro/workspace/linux/tools/objtool/fixdep-in.o
> >   LINK     /home/masahiro/workspace/linux/tools/objtool/fixdep
> >   CC       /home/masahiro/workspace/linux/tools/objtool/exec-cmd.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/help.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/pager.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/parse-options.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/run-command.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/sigchain.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/subcmd-config.o
> >   LD       /home/masahiro/workspace/linux/tools/objtool/libsubcmd-in.o
> >   AR       /home/masahiro/workspace/linux/tools/objtool/libsubcmd.a
> >   GEN      /home/masahiro/workspace/linux/tools/objtool/arch/x86/lib/inat-tables.c
> >   CC       /home/masahiro/workspace/linux/tools/objtool/arch/x86/decode.o
> >   LD       /home/masahiro/workspace/linux/tools/objtool/arch/x86/objtool-in.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/builtin-check.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/builtin-orc.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/check.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/orc_gen.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/orc_dump.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/elf.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/special.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/objtool.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/libstring.o
> >   CC       /home/masahiro/workspace/linux/tools/objtool/str_error_r.o
> >   LD       /home/masahiro/workspace/linux/tools/objtool/objtool-in.o
> >   LINK     /home/masahiro/workspace/linux/tools/objtool/objtool
> >
> >   After:
> >
> >   HOSTCC  scripts/objtool/builtin-check.o
> >   HOSTCC  scripts/objtool/builtin-orc.o
> >   HOSTCC  scripts/objtool/check.o
> >   HOSTCC  scripts/objtool/orc_gen.o
> >   HOSTCC  scripts/objtool/orc_dump.o
> >   HOSTCC  scripts/objtool/elf.o
> >   HOSTCC  scripts/objtool/special.o
> >   HOSTCC  scripts/objtool/objtool.o
> >   HOSTCC  scripts/objtool/libstring.o
> >   HOSTCC  scripts/objtool/str_error_r.o
> >   HOSTCC  scripts/objtool/exec-cmd.o
> >   HOSTCC  scripts/objtool/pager.o
> >   HOSTCC  scripts/objtool/parse-options.o
> >   HOSTCC  scripts/objtool/run-command.o
> >   HOSTCC  scripts/objtool/sigchain.o
> >   HOSTCC  scripts/objtool/subcmd-config.o
> >   AWK     scripts/objtool/arch/x86/lib/inat-tables.c
> >   HOSTCC  scripts/objtool/arch/x86/decode.o
> >   HOSTLD  scripts/objtool/objtool
>
> That does look better.
>
> > [3] simplify distro package script
> >
> > The special handling in scripts/package/buildeb is no longer needed
> > since all host programs are copied to linux-headers packages.
>
> I do see the benefits.  In many ways objtool is a more natural fit in
> the scripts dir.  But there are also some downsides to this change:
>
> - This will make it harder to package objtool as a standalone tool.  It
>   has a lot of functionality that could be useful to other non-kernel
>   projects.


If it is really useful for other projects,
I'd like to see it as a real standalone tool,
i.e. split as a separate project.


> - It shares libsubcmd with perf.  Including the subcmd .c files from
>   tools is hacky.

Yes, hacky. We are shifting the ugliness between C and Build system.

But, this hack can be solved; if subcmd library is useful, you can copy
only necessary code into the objtool directory. You do not need all
helpers from libsubcmd.


> - It's disruptive: it will break all the out-of-tree distro kernel
>   packaging scripts which now look for objtool in tools.

All artifacts under scripts/ should be contained in the package.
So, it should work.


> You're right that objtool isn't a natural fit in tools, because it's
> also used as part of the build.  But it's not a natural fit in scripts
> either.  I think we've resolved most of those issues and it seems to be
> working well these days.
>
> So instead of disrupting everything because "make mrproper" doesn't
> work, I think I'd rather just do the following:
>
> diff --git a/Makefile b/Makefile
> index ac5ac28a24e9..7e6696c9b862 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1364,7 +1364,7 @@ PHONY += $(mrproper-dirs) mrproper
>  $(mrproper-dirs):
>         $(Q)$(MAKE) $(clean)=$(patsubst _mrproper_%,%,$@)
>
> -mrproper: clean $(mrproper-dirs)
> +mrproper: clean $(mrproper-dirs) tools/objtool_clean
>         $(call cmd,rmdirs)
>         $(call cmd,rmfiles)
>

Works, and the clean log is also brilliant.

I see the same log even when
there is nothing to clean.


masahiro@grover:~/ref/linux$ make mrproper
  CLEAN   .
  CLEAN   arch/x86/entry/vdso
  CLEAN   usr
  CLEAN   arch/x86/tools
  CLEAN   .tmp_versions
  CLEAN   scripts/basic
  CLEAN   scripts/kconfig
  CLEAN   scripts/mod
  CLEAN   scripts/selinux/genheaders
  CLEAN   scripts/selinux/mdp
  CLEAN   scripts
  DESCEND  objtool
  CLEAN    objtool
  CLEAN   include/config include/generated arch/x86/include/generated
  CLEAN   .config
masahiro@grover:~/ref/linux$ make mrproper
  DESCEND  objtool
  CLEAN    objtool
masahiro@grover:~/ref/linux$ make mrproper
  DESCEND  objtool
  CLEAN    objtool



-- 
Best Regards
Masahiro Yamada