[PATCH RFC v2] tools: Factor Clang, LLC and LLVM utils definitions

[PATCH RFC v2] tools: Factor Clang, LLC and LLVM utils definitions

[Sedat Dilek]

When dealing with BPF/BTF/pahole and DWARF v5 I wanted to build bpftool.

While looking into the source code I found duplicate assignments
in misc tools for the LLVM eco system, e.g. clang and llvm-objcopy.

Move the Clang, LLC and/or LLVM utils definitions to
tools/scripts/Makefile.include file and add missing
includes where needed.
Honestly, I was inspired by commit c8a950d0d3b9
("tools: Factor HOSTCC, HOSTLD, HOSTAR definitions").

I tested with bpftool and perf on Debian/testing AMD64 and
LLVM/Clang v11.1.0-rc1.

Build instructions:

[ make and make-options ]
MAKE="make V=1"
MAKE_OPTS="HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang LD=ld.lld LLVM=1 LLVM_IAS=1"
MAKE_OPTS="$MAKE_OPTS PAHOLE=/opt/pahole/bin/pahole"

[ clean-up ]
$MAKE $MAKE_OPTS -C tools/ clean

[ bpftool ]
$MAKE $MAKE_OPTS -C tools/bpf/bpftool/

[ perf ]
PYTHON=python3 $MAKE $MAKE_OPTS -C tools/perf/

I was careful with respecting the user's wish to override custom compiler,
linker, GNU/binutils and/or LLVM utils settings.

Some personal notes:
1. I have NOT tested with cross-toolchain for other archs (cross compiler/linker etc.).
2. This patch is on top of Linux v5.11-rc4.

I hope to get some feedback from especially Linux-bpf folks.

Acked-by: Jiri Olsa <jolsa@redhat.com> # tools/build and tools/perf
Signed-off-by: Sedat Dilek <sedat.dilek@gmail.com>
---
Changelog RFC v1->v2:
- Add Jiri's ACK
- Adapt to fit Linux v5.11-rc4

 tools/bpf/bpftool/Makefile                  | 2 --
 tools/bpf/runqslower/Makefile               | 3 ---
 tools/build/feature/Makefile                | 4 ++--
 tools/perf/Makefile.perf                    | 1 -
 tools/scripts/Makefile.include              | 7 +++++++
 tools/testing/selftests/bpf/Makefile        | 3 +--
 tools/testing/selftests/tc-testing/Makefile | 3 +--
 7 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
index f897cb5fb12d..71c14efa6e91 100644
--- a/tools/bpf/bpftool/Makefile
+++ b/tools/bpf/bpftool/Makefile
@@ -75,8 +75,6 @@ endif
 
 INSTALL ?= install
 RM ?= rm -f
-CLANG ?= clang
-LLVM_STRIP ?= llvm-strip
 
 FEATURE_USER = .bpftool
 FEATURE_TESTS = libbfd disassembler-four-args reallocarray zlib libcap \
diff --git a/tools/bpf/runqslower/Makefile b/tools/bpf/runqslower/Makefile
index 4d5ca54fcd4c..9d9fb6209be1 100644
--- a/tools/bpf/runqslower/Makefile
+++ b/tools/bpf/runqslower/Makefile
@@ -3,9 +3,6 @@ include ../../scripts/Makefile.include
 
 OUTPUT ?= $(abspath .output)/
 
-CLANG ?= clang
-LLC ?= llc
-LLVM_STRIP ?= llvm-strip
 BPFTOOL_OUTPUT := $(OUTPUT)bpftool/
 DEFAULT_BPFTOOL := $(BPFTOOL_OUTPUT)bpftool
 BPFTOOL ?= $(DEFAULT_BPFTOOL)
diff --git a/tools/build/feature/Makefile b/tools/build/feature/Makefile
index 89ba522e377d..3e55edb3ea54 100644
--- a/tools/build/feature/Makefile
+++ b/tools/build/feature/Makefile
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
+include ../../scripts/Makefile.include
+
 FILES=                                          \
          test-all.bin                           \
          test-backtrace.bin                     \
@@ -76,8 +78,6 @@ FILES=                                          \
 FILES := $(addprefix $(OUTPUT),$(FILES))
 
 PKG_CONFIG ?= $(CROSS_COMPILE)pkg-config
-LLVM_CONFIG ?= llvm-config
-CLANG ?= clang
 
 all: $(FILES)
 
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 62f3deb1d3a8..f4df7534026d 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -176,7 +176,6 @@ endef
 LD += $(EXTRA_LDFLAGS)
 
 PKG_CONFIG = $(CROSS_COMPILE)pkg-config
-LLVM_CONFIG ?= llvm-config
 
 RM      = rm -f
 LN      = ln -f
diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
index 1358e89cdf7d..4255e71f72b7 100644
--- a/tools/scripts/Makefile.include
+++ b/tools/scripts/Makefile.include
@@ -69,6 +69,13 @@ HOSTCC  ?= gcc
 HOSTLD  ?= ld
 endif
 
+# Some tools require Clang, LLC and/or LLVM utils
+CLANG		?= clang
+LLC		?= llc
+LLVM_CONFIG	?= llvm-config
+LLVM_OBJCOPY	?= llvm-objcopy
+LLVM_STRIP	?= llvm-strip
+
 ifeq ($(CC_NO_CLANG), 1)
 EXTRA_WARNINGS += -Wstrict-aliasing=3
 endif
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index c51df6b91bef..952e2bc5f3bc 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 include ../../../../scripts/Kbuild.include
 include ../../../scripts/Makefile.arch
+include ../../../scripts/Makefile.include
 
 CXX ?= $(CROSS_COMPILE)g++
 
@@ -18,8 +19,6 @@ ifneq ($(wildcard $(GENHDR)),)
   GENFLAGS := -DHAVE_GENHDR
 endif
 
-CLANG		?= clang
-LLVM_OBJCOPY	?= llvm-objcopy
 BPF_GCC		?= $(shell command -v bpf-gcc;)
 SAN_CFLAGS	?=
 CFLAGS += -g -rdynamic -Wall -O2 $(GENFLAGS) $(SAN_CFLAGS)		\
diff --git a/tools/testing/selftests/tc-testing/Makefile b/tools/testing/selftests/tc-testing/Makefile
index 91fee5c43274..4d639279f41e 100644
--- a/tools/testing/selftests/tc-testing/Makefile
+++ b/tools/testing/selftests/tc-testing/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
+include ../../../scripts/Makefile.include
 
 top_srcdir = $(abspath ../../../..)
 APIDIR := $(top_scrdir)/include/uapi
@@ -7,8 +8,6 @@ TEST_GEN_FILES = action.o
 KSFT_KHDR_INSTALL := 1
 include ../lib.mk
 
-CLANG ?= clang
-LLC   ?= llc
 PROBE := $(shell $(LLC) -march=bpf -mcpu=probe -filetype=null /dev/null 2>&1)
 
 ifeq ($(PROBE),)
-- 
2.30.0

Re: [PATCH RFC v2] tools: Factor Clang, LLC and LLVM utils definitions

[Andrii Nakryiko]

On Thu, Jan 21, 2021 at 4:32 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> When dealing with BPF/BTF/pahole and DWARF v5 I wanted to build bpftool.
>
> While looking into the source code I found duplicate assignments
> in misc tools for the LLVM eco system, e.g. clang and llvm-objcopy.
>
> Move the Clang, LLC and/or LLVM utils definitions to
> tools/scripts/Makefile.include file and add missing
> includes where needed.
> Honestly, I was inspired by commit c8a950d0d3b9
> ("tools: Factor HOSTCC, HOSTLD, HOSTAR definitions").
>
> I tested with bpftool and perf on Debian/testing AMD64 and
> LLVM/Clang v11.1.0-rc1.
>
> Build instructions:
>
> [ make and make-options ]
> MAKE="make V=1"
> MAKE_OPTS="HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang LD=ld.lld LLVM=1 LLVM_IAS=1"
> MAKE_OPTS="$MAKE_OPTS PAHOLE=/opt/pahole/bin/pahole"
>
> [ clean-up ]
> $MAKE $MAKE_OPTS -C tools/ clean
>
> [ bpftool ]
> $MAKE $MAKE_OPTS -C tools/bpf/bpftool/
>
> [ perf ]
> PYTHON=python3 $MAKE $MAKE_OPTS -C tools/perf/
>
> I was careful with respecting the user's wish to override custom compiler,
> linker, GNU/binutils and/or LLVM utils settings.
>
> Some personal notes:
> 1. I have NOT tested with cross-toolchain for other archs (cross compiler/linker etc.).
> 2. This patch is on top of Linux v5.11-rc4.
>
> I hope to get some feedback from especially Linux-bpf folks.
>
> Acked-by: Jiri Olsa <jolsa@redhat.com> # tools/build and tools/perf
> Signed-off-by: Sedat Dilek <sedat.dilek@gmail.com>
> ---

Hi Sedat,

If no one objects, we'll take this through bpf-next tree. Can you
please re-send this as a non-RFC patch against the bpf-next tree? Feel
free to add my ack. Thanks.

> Changelog RFC v1->v2:
> - Add Jiri's ACK
> - Adapt to fit Linux v5.11-rc4
>

[...]

Re: [PATCH RFC v2] tools: Factor Clang, LLC and LLVM utils definitions

[Sedat Dilek]

On Thu, Jan 28, 2021 at 2:27 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Thu, Jan 21, 2021 at 4:32 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > When dealing with BPF/BTF/pahole and DWARF v5 I wanted to build bpftool.
> >
> > While looking into the source code I found duplicate assignments
> > in misc tools for the LLVM eco system, e.g. clang and llvm-objcopy.
> >
> > Move the Clang, LLC and/or LLVM utils definitions to
> > tools/scripts/Makefile.include file and add missing
> > includes where needed.
> > Honestly, I was inspired by commit c8a950d0d3b9
> > ("tools: Factor HOSTCC, HOSTLD, HOSTAR definitions").
> >
> > I tested with bpftool and perf on Debian/testing AMD64 and
> > LLVM/Clang v11.1.0-rc1.
> >
> > Build instructions:
> >
> > [ make and make-options ]
> > MAKE="make V=1"
> > MAKE_OPTS="HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang LD=ld.lld LLVM=1 LLVM_IAS=1"
> > MAKE_OPTS="$MAKE_OPTS PAHOLE=/opt/pahole/bin/pahole"
> >
> > [ clean-up ]
> > $MAKE $MAKE_OPTS -C tools/ clean
> >
> > [ bpftool ]
> > $MAKE $MAKE_OPTS -C tools/bpf/bpftool/
> >
> > [ perf ]
> > PYTHON=python3 $MAKE $MAKE_OPTS -C tools/perf/
> >
> > I was careful with respecting the user's wish to override custom compiler,
> > linker, GNU/binutils and/or LLVM utils settings.
> >
> > Some personal notes:
> > 1. I have NOT tested with cross-toolchain for other archs (cross compiler/linker etc.).
> > 2. This patch is on top of Linux v5.11-rc4.
> >
> > I hope to get some feedback from especially Linux-bpf folks.
> >
> > Acked-by: Jiri Olsa <jolsa@redhat.com> # tools/build and tools/perf
> > Signed-off-by: Sedat Dilek <sedat.dilek@gmail.com>
> > ---
>
> Hi Sedat,
>
> If no one objects, we'll take this through bpf-next tree. Can you
> please re-send this as a non-RFC patch against the bpf-next tree? Feel
> free to add my ack. Thanks.
>

I am OK with that and will add your ACK.
Is [1] bpf-next Git?

- Sedat -

[1] https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/

> > Changelog RFC v1->v2:
> > - Add Jiri's ACK
> > - Adapt to fit Linux v5.11-rc4
> >
>
> [...]

Re: [PATCH RFC v2] tools: Factor Clang, LLC and LLVM utils definitions

[Andrii Nakryiko]

On Wed, Jan 27, 2021 at 5:30 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Thu, Jan 28, 2021 at 2:27 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Thu, Jan 21, 2021 at 4:32 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > >
> > > When dealing with BPF/BTF/pahole and DWARF v5 I wanted to build bpftool.
> > >
> > > While looking into the source code I found duplicate assignments
> > > in misc tools for the LLVM eco system, e.g. clang and llvm-objcopy.
> > >
> > > Move the Clang, LLC and/or LLVM utils definitions to
> > > tools/scripts/Makefile.include file and add missing
> > > includes where needed.
> > > Honestly, I was inspired by commit c8a950d0d3b9
> > > ("tools: Factor HOSTCC, HOSTLD, HOSTAR definitions").
> > >
> > > I tested with bpftool and perf on Debian/testing AMD64 and
> > > LLVM/Clang v11.1.0-rc1.
> > >
> > > Build instructions:
> > >
> > > [ make and make-options ]
> > > MAKE="make V=1"
> > > MAKE_OPTS="HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang LD=ld.lld LLVM=1 LLVM_IAS=1"
> > > MAKE_OPTS="$MAKE_OPTS PAHOLE=/opt/pahole/bin/pahole"
> > >
> > > [ clean-up ]
> > > $MAKE $MAKE_OPTS -C tools/ clean
> > >
> > > [ bpftool ]
> > > $MAKE $MAKE_OPTS -C tools/bpf/bpftool/
> > >
> > > [ perf ]
> > > PYTHON=python3 $MAKE $MAKE_OPTS -C tools/perf/
> > >
> > > I was careful with respecting the user's wish to override custom compiler,
> > > linker, GNU/binutils and/or LLVM utils settings.
> > >
> > > Some personal notes:
> > > 1. I have NOT tested with cross-toolchain for other archs (cross compiler/linker etc.).
> > > 2. This patch is on top of Linux v5.11-rc4.
> > >
> > > I hope to get some feedback from especially Linux-bpf folks.
> > >
> > > Acked-by: Jiri Olsa <jolsa@redhat.com> # tools/build and tools/perf
> > > Signed-off-by: Sedat Dilek <sedat.dilek@gmail.com>
> > > ---
> >
> > Hi Sedat,
> >
> > If no one objects, we'll take this through bpf-next tree. Can you
> > please re-send this as a non-RFC patch against the bpf-next tree? Feel
> > free to add my ack. Thanks.
> >
>
> I am OK with that and will add your ACK.
> Is [1] bpf-next Git?

Yes, please use [PATCH bpf-next] subject prefix and cc
bpf@vger.kernel.org as well.

>
> - Sedat -
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/
>
> > > Changelog RFC v1->v2:
> > > - Add Jiri's ACK
> > > - Adapt to fit Linux v5.11-rc4
> > >
> >
> > [...]

Re: [PATCH RFC v2] tools: Factor Clang, LLC and LLVM utils definitions

[Sedat Dilek]

On Thu, Jan 28, 2021 at 2:41 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Wed, Jan 27, 2021 at 5:30 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > On Thu, Jan 28, 2021 at 2:27 AM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Thu, Jan 21, 2021 at 4:32 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > >
> > > > When dealing with BPF/BTF/pahole and DWARF v5 I wanted to build bpftool.
> > > >
> > > > While looking into the source code I found duplicate assignments
> > > > in misc tools for the LLVM eco system, e.g. clang and llvm-objcopy.
> > > >
> > > > Move the Clang, LLC and/or LLVM utils definitions to
> > > > tools/scripts/Makefile.include file and add missing
> > > > includes where needed.
> > > > Honestly, I was inspired by commit c8a950d0d3b9
> > > > ("tools: Factor HOSTCC, HOSTLD, HOSTAR definitions").
> > > >
> > > > I tested with bpftool and perf on Debian/testing AMD64 and
> > > > LLVM/Clang v11.1.0-rc1.
> > > >
> > > > Build instructions:
> > > >
> > > > [ make and make-options ]
> > > > MAKE="make V=1"
> > > > MAKE_OPTS="HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang LD=ld.lld LLVM=1 LLVM_IAS=1"
> > > > MAKE_OPTS="$MAKE_OPTS PAHOLE=/opt/pahole/bin/pahole"
> > > >
> > > > [ clean-up ]
> > > > $MAKE $MAKE_OPTS -C tools/ clean
> > > >
> > > > [ bpftool ]
> > > > $MAKE $MAKE_OPTS -C tools/bpf/bpftool/
> > > >
> > > > [ perf ]
> > > > PYTHON=python3 $MAKE $MAKE_OPTS -C tools/perf/
> > > >
> > > > I was careful with respecting the user's wish to override custom compiler,
> > > > linker, GNU/binutils and/or LLVM utils settings.
> > > >
> > > > Some personal notes:
> > > > 1. I have NOT tested with cross-toolchain for other archs (cross compiler/linker etc.).
> > > > 2. This patch is on top of Linux v5.11-rc4.
> > > >
> > > > I hope to get some feedback from especially Linux-bpf folks.
> > > >
> > > > Acked-by: Jiri Olsa <jolsa@redhat.com> # tools/build and tools/perf
> > > > Signed-off-by: Sedat Dilek <sedat.dilek@gmail.com>
> > > > ---
> > >
> > > Hi Sedat,
> > >
> > > If no one objects, we'll take this through bpf-next tree. Can you
> > > please re-send this as a non-RFC patch against the bpf-next tree? Feel
> > > free to add my ack. Thanks.
> > >
> >
> > I am OK with that and will add your ACK.
> > Is [1] bpf-next Git?
>
> Yes, please use [PATCH bpf-next] subject prefix and cc
> bpf@vger.kernel.org as well.
>

Please see link:

https://lore.kernel.org/r/20210128015117.20515-1-sedat.dilek@gmail.com

- Sedat -

> >
> > - Sedat -
> >
> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/
> >
> > > > Changelog RFC v1->v2:
> > > > - Add Jiri's ACK
> > > > - Adapt to fit Linux v5.11-rc4
> > > >
> > >
> > > [...]