[PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Yonghong Song]

To build kernel with clang, people typically use
  make -j60 LLVM=1 LLVM_IAS=1
LLVM_IAS=1 is not required for non-LTO build but
is required for LTO build. In my environment,
I am always having LLVM_IAS=1 regardless of
whether LTO is enabled or not.

After kernel is build with clang, the following command
can be used to build selftests with clang:
  make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1

But currently, using the above command, some compilations
still use gcc and there are also compilation errors and warnings.
This patch set intends to fix these issues.
Patch #1 and #2 fixed the issue so clang/clang++ is
used instead of gcc/g++. Patch #3 fixed a compilation
failure. Patch #4 and #5 fixed various compiler warnings.

Changelog:
  v1 -> v2:
    . add -Wno-unused-command-line-argument and -Wno-format-security
      for clang only as (1). gcc does not exhibit those
      warnings, and (2). -Wno-unused-command-line-argument is
      only supported by clang. (Sedat)

Yonghong Song (5):
  selftests: set CC to clang in lib.mk if LLVM is set
  tools: allow proper CC/CXX/... override with LLVM=1 in
    Makefile.include
  selftests/bpf: fix test_cpp compilation failure with clang
  selftests/bpf: silence clang compilation warnings
  bpftool: fix a clang compilation warning

 tools/bpf/bpftool/net.c              |  2 +-
 tools/scripts/Makefile.include       | 12 ++++++++++--
 tools/testing/selftests/bpf/Makefile |  7 ++++++-
 tools/testing/selftests/lib.mk       |  4 ++++
 4 files changed, 21 insertions(+), 4 deletions(-)

-- 
2.30.2

[PATCH bpf-next v2 1/5] selftests: set CC to clang in lib.mk if LLVM is set

[Yonghong Song]

selftests/bpf/Makefile includes lib.mk. With the following command
  make -j60 LLVM=1 LLVM_IAS=1  <=== compile kernel
  make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1 V=1
some files are still compiled with gcc. This patch
fixed lib.mk issue which sets CC to gcc in all cases.

Cc: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Yonghong Song <yhs@fb.com>
---
 tools/testing/selftests/lib.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index a5ce26d548e4..9a41d8bb9ff1 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -1,6 +1,10 @@
 # This mimics the top-level Makefile. We do it explicitly here so that this
 # Makefile can operate with or without the kbuild infrastructure.
+ifneq ($(LLVM),)
+CC := clang
+else
 CC := $(CROSS_COMPILE)gcc
+endif
 
 ifeq (0,$(MAKELEVEL))
     ifeq ($(OUTPUT),)
-- 
2.30.2

[PATCH bpf-next v2 2/5] tools: allow proper CC/CXX/... override with LLVM=1 in Makefile.include

[Yonghong Song]

selftests/bpf/Makefile includes tools/scripts/Makefile.include.
With the following command
  make -j60 LLVM=1 LLVM_IAS=1  <=== compile kernel
  make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1 V=1
some files are still compiled with gcc. This patch
fixed the case if CC/AR/LD/CXX/STRIP is allowed to be
overridden, it will be written to clang/llvm-ar/..., instead of
gcc binaries. The definition of CC_NO_CLANG is also relocated
to the place after the above CC is defined.

Cc: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Yonghong Song <yhs@fb.com>
---
 tools/scripts/Makefile.include | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
index a402f32a145c..91130648d8e6 100644
--- a/tools/scripts/Makefile.include
+++ b/tools/scripts/Makefile.include
@@ -39,8 +39,6 @@ EXTRA_WARNINGS += -Wundef
 EXTRA_WARNINGS += -Wwrite-strings
 EXTRA_WARNINGS += -Wformat
 
-CC_NO_CLANG := $(shell $(CC) -dM -E -x c /dev/null | grep -Fq "__clang__"; echo $$?)
-
 # Makefiles suck: This macro sets a default value of $(2) for the
 # variable named by $(1), unless the variable has been set by
 # environment or command line. This is necessary for CC and AR
@@ -52,12 +50,22 @@ define allow-override
     $(eval $(1) = $(2)))
 endef
 
+ifneq ($(LLVM),)
+$(call allow-override,CC,clang)
+$(call allow-override,AR,llvm-ar)
+$(call allow-override,LD,ld.lld)
+$(call allow-override,CXX,clang++)
+$(call allow-override,STRIP,llvm-strip)
+else
 # Allow setting various cross-compile vars or setting CROSS_COMPILE as a prefix.
 $(call allow-override,CC,$(CROSS_COMPILE)gcc)
 $(call allow-override,AR,$(CROSS_COMPILE)ar)
 $(call allow-override,LD,$(CROSS_COMPILE)ld)
 $(call allow-override,CXX,$(CROSS_COMPILE)g++)
 $(call allow-override,STRIP,$(CROSS_COMPILE)strip)
+endif
+
+CC_NO_CLANG := $(shell $(CC) -dM -E -x c /dev/null | grep -Fq "__clang__"; echo $$?)
 
 ifneq ($(LLVM),)
 HOSTAR  ?= llvm-ar
-- 
2.30.2

[PATCH bpf-next v2 3/5] selftests/bpf: fix test_cpp compilation failure with clang

[Yonghong Song]

With clang compiler:
  make -j60 LLVM=1 LLVM_IAS=1  <=== compile kernel
  make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
the test_cpp build failed due to the failure:
  warning: treating 'c-header' input as 'c++-header' when in C++ mode, this behavior is deprecated [-Wdeprecated]
  clang-13: error: cannot specify -o when generating multiple output files

test_cpp compilation flag looks like:
  clang++ -g -Og -rdynamic -Wall -I<...> ... \
  -Dbpf_prog_load=bpf_prog_test_load -Dbpf_load_program=bpf_test_load_program \
  test_cpp.cpp <...>/test_core_extern.skel.h <...>/libbpf.a <...>/test_stub.o \
  -lcap -lelf -lz -lrt -lpthread -o <...>/test_cpp

The clang++ compiler complains the header file in the command line and
also failed the compilation due to this.
Let us remove the header file from the command line which is not intended
any way, and this fixed the compilation problem.

Signed-off-by: Yonghong Song <yhs@fb.com>
---
 tools/testing/selftests/bpf/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 6448c626498f..bbd61cc3889b 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -481,7 +481,7 @@ $(OUTPUT)/test_verifier: test_verifier.c verifier/tests.h $(BPFOBJ) | $(OUTPUT)
 # Make sure we are able to include and link libbpf against c++.
 $(OUTPUT)/test_cpp: test_cpp.cpp $(OUTPUT)/test_core_extern.skel.h $(BPFOBJ)
 	$(call msg,CXX,,$@)
-	$(Q)$(CXX) $(CFLAGS) $^ $(LDLIBS) -o $@
+	$(Q)$(CXX) $(CFLAGS) test_cpp.cpp $(BPFOBJ) $(LDLIBS) -o $@
 
 # Benchmark runner
 $(OUTPUT)/bench_%.o: benchs/bench_%.c bench.h
-- 
2.30.2

[PATCH bpf-next v2 4/5] selftests/bpf: silence clang compilation warnings

[Yonghong Song]

With clang compiler:
  make -j60 LLVM=1 LLVM_IAS=1  <=== compile kernel
  make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
Some linker flags are not used/effective for some binaries and
we have warnings like:
  warning: -lelf: 'linker' input unused [-Wunused-command-line-argument]

We also have warnings like:
  .../selftests/bpf/prog_tests/ns_current_pid_tgid.c:74:57: note: treat the string as an argument to avoid this
        if (CHECK(waitpid(cpid, &wstatus, 0) == -1, "waitpid", strerror(errno)))
                                                               ^
                                                               "%s",
  .../selftests/bpf/test_progs.h:129:35: note: expanded from macro 'CHECK'
        _CHECK(condition, tag, duration, format)
                                         ^
  .../selftests/bpf/test_progs.h:108:21: note: expanded from macro '_CHECK'
                fprintf(stdout, ##format);                              \
                                  ^
The first warning can be silenced with clang option -Wno-unused-command-line-argument,
and the second with -Wno-format-security. Further, gcc does not support the option
-Wno-unused-command-line-argument. Since the warning only happens with clang
compiler, these two options are enabled only when clang compiler is used and this
fixed the above warnings.

Signed-off-by: Yonghong Song <yhs@fb.com>
---
 tools/testing/selftests/bpf/Makefile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index bbd61cc3889b..ef7078756c8a 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -28,6 +28,11 @@ CFLAGS += -g -Og -rdynamic -Wall $(GENFLAGS) $(SAN_CFLAGS)		\
 	  -Dbpf_load_program=bpf_test_load_program
 LDLIBS += -lcap -lelf -lz -lrt -lpthread
 
+# Silence some warnings when compiled with clang
+ifneq ($(LLVM),)
+CFLAGS += -Wno-unused-command-line-argument -Wno-format-security
+endif
+
 # Order correspond to 'make run_tests' order
 TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test_progs \
 	test_verifier_log test_dev_cgroup \
-- 
2.30.2

Re: [PATCH bpf-next v2 4/5] selftests/bpf: silence clang compilation warnings

[Andrii Nakryiko]

On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
>
> With clang compiler:
>   make -j60 LLVM=1 LLVM_IAS=1  <=== compile kernel
>   make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
> Some linker flags are not used/effective for some binaries and
> we have warnings like:
>   warning: -lelf: 'linker' input unused [-Wunused-command-line-argument]
>
> We also have warnings like:
>   .../selftests/bpf/prog_tests/ns_current_pid_tgid.c:74:57: note: treat the string as an argument to avoid this
>         if (CHECK(waitpid(cpid, &wstatus, 0) == -1, "waitpid", strerror(errno)))
>                                                                ^
>                                                                "%s",

So why not do what the compiler suggests and use `"err %s",
strerror(errno)` instead of silencing useful warning globally?

>   .../selftests/bpf/test_progs.h:129:35: note: expanded from macro 'CHECK'
>         _CHECK(condition, tag, duration, format)
>                                          ^
>   .../selftests/bpf/test_progs.h:108:21: note: expanded from macro '_CHECK'
>                 fprintf(stdout, ##format);                              \
>                                   ^
> The first warning can be silenced with clang option -Wno-unused-command-line-argument,

this one does seem necessary, otherwise we'll have to adjust per each
.c file the list of libraries needed


> and the second with -Wno-format-security. Further, gcc does not support the option
> -Wno-unused-command-line-argument. Since the warning only happens with clang
> compiler, these two options are enabled only when clang compiler is used and this
> fixed the above warnings.
>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
>  tools/testing/selftests/bpf/Makefile | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
> index bbd61cc3889b..ef7078756c8a 100644
> --- a/tools/testing/selftests/bpf/Makefile
> +++ b/tools/testing/selftests/bpf/Makefile
> @@ -28,6 +28,11 @@ CFLAGS += -g -Og -rdynamic -Wall $(GENFLAGS) $(SAN_CFLAGS)           \
>           -Dbpf_load_program=bpf_test_load_program
>  LDLIBS += -lcap -lelf -lz -lrt -lpthread
>
> +# Silence some warnings when compiled with clang
> +ifneq ($(LLVM),)
> +CFLAGS += -Wno-unused-command-line-argument -Wno-format-security
> +endif
> +
>  # Order correspond to 'make run_tests' order
>  TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test_progs \
>         test_verifier_log test_dev_cgroup \
> --
> 2.30.2
>

[PATCH bpf-next v2 5/5] bpftool: fix a clang compilation warning

[Yonghong Song]

With clang compiler:
  make -j60 LLVM=1 LLVM_IAS=1  <=== compile kernel
  # build selftests/bpf or bpftool
  make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
  make -j60 -C tools/bpf/bpftool LLVM=1 LLVM_IAS=1
the following compilation warning showed up,
  net.c:160:37: warning: comparison of integers of different signs: '__u32' (aka 'unsigned int') and 'int' [-Wsign-compare]
                for (nh = (struct nlmsghdr *)buf; NLMSG_OK(nh, len);
                                                  ^~~~~~~~~~~~~~~~~
  .../tools/include/uapi/linux/netlink.h:99:24: note: expanded from macro 'NLMSG_OK'
                           (nlh)->nlmsg_len <= (len))
                           ~~~~~~~~~~~~~~~~ ^   ~~~

In this particular case, "len" is defined as "int" and (nlh)->nlmsg_len is "unsigned int".
The macro NLMSG_OK is defined as below in uapi/linux/netlink.h.
  #define NLMSG_OK(nlh,len) ((len) >= (int)sizeof(struct nlmsghdr) && \
                             (nlh)->nlmsg_len >= sizeof(struct nlmsghdr) && \
                             (nlh)->nlmsg_len <= (len))

The clang compiler complains the comparision "(nlh)->nlmsg_len <= (len))",
but in bpftool/net.c, it is already ensured that "len > 0" must be true.
So theoretically the compiler could deduce that comparison of
"(nlh)->nlmsg_len" and "len" is okay, but this really depends on compiler
internals. Let us add an explicit type conversion (from "int" to "unsigned int")
for "len" in NLMSG_OK to silence this warning right now.

Signed-off-by: Yonghong Song <yhs@fb.com>
---
 tools/bpf/bpftool/net.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/bpf/bpftool/net.c b/tools/bpf/bpftool/net.c
index ff3aa0cf3997..f836d115d7d6 100644
--- a/tools/bpf/bpftool/net.c
+++ b/tools/bpf/bpftool/net.c
@@ -157,7 +157,7 @@ static int netlink_recv(int sock, __u32 nl_pid, __u32 seq,
 		if (len == 0)
 			break;
 
-		for (nh = (struct nlmsghdr *)buf; NLMSG_OK(nh, len);
+		for (nh = (struct nlmsghdr *)buf; NLMSG_OK(nh, (unsigned int)len);
 		     nh = NLMSG_NEXT(nh, len)) {
 			if (nh->nlmsg_pid != nl_pid) {
 				ret = -LIBBPF_ERRNO__WRNGPID;
-- 
2.30.2

Re: [PATCH bpf-next v2 5/5] bpftool: fix a clang compilation warning

[Andrii Nakryiko]

On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
>
> With clang compiler:
>   make -j60 LLVM=1 LLVM_IAS=1  <=== compile kernel
>   # build selftests/bpf or bpftool
>   make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
>   make -j60 -C tools/bpf/bpftool LLVM=1 LLVM_IAS=1
> the following compilation warning showed up,
>   net.c:160:37: warning: comparison of integers of different signs: '__u32' (aka 'unsigned int') and 'int' [-Wsign-compare]
>                 for (nh = (struct nlmsghdr *)buf; NLMSG_OK(nh, len);
>                                                   ^~~~~~~~~~~~~~~~~
>   .../tools/include/uapi/linux/netlink.h:99:24: note: expanded from macro 'NLMSG_OK'
>                            (nlh)->nlmsg_len <= (len))
>                            ~~~~~~~~~~~~~~~~ ^   ~~~
>
> In this particular case, "len" is defined as "int" and (nlh)->nlmsg_len is "unsigned int".
> The macro NLMSG_OK is defined as below in uapi/linux/netlink.h.
>   #define NLMSG_OK(nlh,len) ((len) >= (int)sizeof(struct nlmsghdr) && \
>                              (nlh)->nlmsg_len >= sizeof(struct nlmsghdr) && \
>                              (nlh)->nlmsg_len <= (len))
>
> The clang compiler complains the comparision "(nlh)->nlmsg_len <= (len))",
> but in bpftool/net.c, it is already ensured that "len > 0" must be true.
> So theoretically the compiler could deduce that comparison of
> "(nlh)->nlmsg_len" and "len" is okay, but this really depends on compiler
> internals. Let us add an explicit type conversion (from "int" to "unsigned int")
> for "len" in NLMSG_OK to silence this warning right now.
>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
>  tools/bpf/bpftool/net.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Acked-by: Andrii Nakryiko <andrii@kernel.org>

>
> diff --git a/tools/bpf/bpftool/net.c b/tools/bpf/bpftool/net.c
> index ff3aa0cf3997..f836d115d7d6 100644
> --- a/tools/bpf/bpftool/net.c
> +++ b/tools/bpf/bpftool/net.c
> @@ -157,7 +157,7 @@ static int netlink_recv(int sock, __u32 nl_pid, __u32 seq,
>                 if (len == 0)
>                         break;
>
> -               for (nh = (struct nlmsghdr *)buf; NLMSG_OK(nh, len);
> +               for (nh = (struct nlmsghdr *)buf; NLMSG_OK(nh, (unsigned int)len);
>                      nh = NLMSG_NEXT(nh, len)) {
>                         if (nh->nlmsg_pid != nl_pid) {
>                                 ret = -LIBBPF_ERRNO__WRNGPID;
> --
> 2.30.2
>

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Nick Desaulniers]

On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
>
> To build kernel with clang, people typically use
>   make -j60 LLVM=1 LLVM_IAS=1
> LLVM_IAS=1 is not required for non-LTO build but
> is required for LTO build. In my environment,
> I am always having LLVM_IAS=1 regardless of
> whether LTO is enabled or not.
>
> After kernel is build with clang, the following command
> can be used to build selftests with clang:
>   make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1

Thank you for the series Yonghong.  When I test the above command with
your series applied, I observe:
/usr/bin/ld: cannot find -lcap
clang-13: error: linker command failed with exit code 1 (use -v to see
invocation)

I need to install libcap-dev, but this also seems to imply that BFD is
being used as the linker, not LLD.  Perhaps if the compiler is being
used as the "driver" to also link executables, `-fuse-ld=lld` is
needed for the compiler flags.

Then there's:
tools/include/tools/libc_compat.h:11:21: error: static declaration of
'reallocarray' follows non-static declaration
static inline void *reallocarray(void *ptr, size_t nmemb, size_t size)
                    ^
/usr/include/stdlib.h:559:14: note: previous declaration is here
extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
             ^
so perhaps the detection of
COMPAT_NEED_REALLOCARRAY/feature-reallocarray is incorrect?


>
> But currently, using the above command, some compilations
> still use gcc and there are also compilation errors and warnings.
> This patch set intends to fix these issues.
> Patch #1 and #2 fixed the issue so clang/clang++ is
> used instead of gcc/g++. Patch #3 fixed a compilation
> failure. Patch #4 and #5 fixed various compiler warnings.
>
> Changelog:
>   v1 -> v2:
>     . add -Wno-unused-command-line-argument and -Wno-format-security
>       for clang only as (1). gcc does not exhibit those
>       warnings, and (2). -Wno-unused-command-line-argument is
>       only supported by clang. (Sedat)
>
> Yonghong Song (5):
>   selftests: set CC to clang in lib.mk if LLVM is set
>   tools: allow proper CC/CXX/... override with LLVM=1 in
>     Makefile.include
>   selftests/bpf: fix test_cpp compilation failure with clang
>   selftests/bpf: silence clang compilation warnings
>   bpftool: fix a clang compilation warning
>
>  tools/bpf/bpftool/net.c              |  2 +-
>  tools/scripts/Makefile.include       | 12 ++++++++++--
>  tools/testing/selftests/bpf/Makefile |  7 ++++++-
>  tools/testing/selftests/lib.mk       |  4 ++++
>  4 files changed, 21 insertions(+), 4 deletions(-)
>
> --
> 2.30.2
>


--
Thanks,
~Nick Desaulniers

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Nick Desaulniers]

On Mon, Apr 12, 2021 at 4:58 PM Nick Desaulniers
<ndesaulniers@google.com> wrote:
>
> On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
> >
> > To build kernel with clang, people typically use
> >   make -j60 LLVM=1 LLVM_IAS=1
> > LLVM_IAS=1 is not required for non-LTO build but
> > is required for LTO build. In my environment,
> > I am always having LLVM_IAS=1 regardless of
> > whether LTO is enabled or not.
> >
> > After kernel is build with clang, the following command
> > can be used to build selftests with clang:
> >   make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
>
> Thank you for the series Yonghong.  When I test the above command with
> your series applied, I observe:
> tools/include/tools/libc_compat.h:11:21: error: static declaration of
> 'reallocarray' follows non-static declaration
> static inline void *reallocarray(void *ptr, size_t nmemb, size_t size)
>                     ^
> /usr/include/stdlib.h:559:14: note: previous declaration is here
> extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
>              ^
> so perhaps the detection of
> COMPAT_NEED_REALLOCARRAY/feature-reallocarray is incorrect?

Is this related to _DEFAULT_SOURCE vs _GNU_SOURCE.  via man 3 reallocarray:
       reallocarray():
           Since glibc 2.29:
               _DEFAULT_SOURCE
           Glibc 2.28 and earlier:
               _GNU_SOURCE

$ cd tools/testing/selftests/bpf
$ grep -rn _DEFAULT_SOURCE | wc -l
0
$ grep -rn _GNU_SOURCE | wc -l
37
$ ldd --version | head -n1
ldd (Debian GLIBC 2.31-9+build1) 2.31
-- 
Thanks,
~Nick Desaulniers

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Yonghong Song]

On 4/12/21 5:02 PM, Nick Desaulniers wrote:
> On Mon, Apr 12, 2021 at 4:58 PM Nick Desaulniers
> <ndesaulniers@google.com> wrote:
>>
>> On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
>>>
>>> To build kernel with clang, people typically use
>>>    make -j60 LLVM=1 LLVM_IAS=1
>>> LLVM_IAS=1 is not required for non-LTO build but
>>> is required for LTO build. In my environment,
>>> I am always having LLVM_IAS=1 regardless of
>>> whether LTO is enabled or not.
>>>
>>> After kernel is build with clang, the following command
>>> can be used to build selftests with clang:
>>>    make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
>>
>> Thank you for the series Yonghong.  When I test the above command with
>> your series applied, I observe:
>> tools/include/tools/libc_compat.h:11:21: error: static declaration of
>> 'reallocarray' follows non-static declaration
>> static inline void *reallocarray(void *ptr, size_t nmemb, size_t size)
>>                      ^
>> /usr/include/stdlib.h:559:14: note: previous declaration is here
>> extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
>>               ^
>> so perhaps the detection of
>> COMPAT_NEED_REALLOCARRAY/feature-reallocarray is incorrect?
> 
> Is this related to _DEFAULT_SOURCE vs _GNU_SOURCE.  via man 3 reallocarray:
>         reallocarray():
>             Since glibc 2.29:
>                 _DEFAULT_SOURCE
>             Glibc 2.28 and earlier:
>                 _GNU_SOURCE
> 

You can try the following patch to see whether it works or not.

diff --git a/tools/build/feature/test-reallocarray.c 
b/tools/build/feature/test-reallocarray.c
index 8f6743e31da7..500cdeca07a7 100644
--- a/tools/build/feature/test-reallocarray.c
+++ b/tools/build/feature/test-reallocarray.c
@@ -1,5 +1,5 @@
  // SPDX-License-Identifier: GPL-2.0
-#define _GNU_SOURCE
+#define _DEFAULT_SOURCE
  #include <stdlib.h>

  int main(void)
@@ -7,4 +7,4 @@ int main(void)
         return !!reallocarray(NULL, 1, 1);
  }

-#undef _GNU_SOURCE
+#undef _DEFAULT_SOURCE
[yhs@devbig003.ftw2 ~/work/bpf-next/tools/build]$

> $ cd tools/testing/selftests/bpf
> $ grep -rn _DEFAULT_SOURCE | wc -l
> 0
> $ grep -rn _GNU_SOURCE | wc -l
> 37
> $ ldd --version | head -n1
> ldd (Debian GLIBC 2.31-9+build1) 2.31
> 

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Nick Desaulniers]

On Mon, Apr 12, 2021 at 5:31 PM Yonghong Song <yhs@fb.com> wrote:
>
>
>
> On 4/12/21 5:02 PM, Nick Desaulniers wrote:
> > On Mon, Apr 12, 2021 at 4:58 PM Nick Desaulniers
> > <ndesaulniers@google.com> wrote:
> >>
> >> On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
> >>>
> >>> To build kernel with clang, people typically use
> >>>    make -j60 LLVM=1 LLVM_IAS=1
> >>> LLVM_IAS=1 is not required for non-LTO build but
> >>> is required for LTO build. In my environment,
> >>> I am always having LLVM_IAS=1 regardless of
> >>> whether LTO is enabled or not.
> >>>
> >>> After kernel is build with clang, the following command
> >>> can be used to build selftests with clang:
> >>>    make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
> >>
> >> Thank you for the series Yonghong.  When I test the above command with
> >> your series applied, I observe:
> >> tools/include/tools/libc_compat.h:11:21: error: static declaration of
> >> 'reallocarray' follows non-static declaration
> >> static inline void *reallocarray(void *ptr, size_t nmemb, size_t size)
> >>                      ^
> >> /usr/include/stdlib.h:559:14: note: previous declaration is here
> >> extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
> >>               ^
> >> so perhaps the detection of
> >> COMPAT_NEED_REALLOCARRAY/feature-reallocarray is incorrect?
> >
> > Is this related to _DEFAULT_SOURCE vs _GNU_SOURCE.  via man 3 reallocarray:
> >         reallocarray():
> >             Since glibc 2.29:
> >                 _DEFAULT_SOURCE
> >             Glibc 2.28 and earlier:
> >                 _GNU_SOURCE
> >
>
> You can try the following patch to see whether it works or not.
>
> diff --git a/tools/build/feature/test-reallocarray.c
> b/tools/build/feature/test-reallocarray.c
> index 8f6743e31da7..500cdeca07a7 100644
> --- a/tools/build/feature/test-reallocarray.c
> +++ b/tools/build/feature/test-reallocarray.c
> @@ -1,5 +1,5 @@
>   // SPDX-License-Identifier: GPL-2.0
> -#define _GNU_SOURCE
> +#define _DEFAULT_SOURCE
>   #include <stdlib.h>
>
>   int main(void)
> @@ -7,4 +7,4 @@ int main(void)
>          return !!reallocarray(NULL, 1, 1);
>   }
>
> -#undef _GNU_SOURCE
> +#undef _DEFAULT_SOURCE

Yeah, I had tried that. No luck though; same error message.  Even:

$ cat foo.c
#define _DEFAULT_SOURCE
#include <stdlib.h>
void *reallocarray(void *ptr, size_t nmemb, size_t size) { return (void*)0; };
$ clang -c foo.c
$ echo $?
0

So I'm not sure precisely what's going on here.  I probably have to go
digging around to understand tools/build/feature/ anyways.  With your
v3 applied, I consistently see:
No zlib found
and yet, I certainly do have zlib on my host.
https://stackoverflow.com/a/54558861

> [yhs@devbig003.ftw2 ~/work/bpf-next/tools/build]$
>
> > $ cd tools/testing/selftests/bpf
> > $ grep -rn _DEFAULT_SOURCE | wc -l
> > 0
> > $ grep -rn _GNU_SOURCE | wc -l
> > 37
> > $ ldd --version | head -n1
> > ldd (Debian GLIBC 2.31-9+build1) 2.31
> >



-- 
Thanks,
~Nick Desaulniers

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Nick Desaulniers]

On Tue, Apr 13, 2021 at 11:46 AM Nick Desaulniers
<ndesaulniers@google.com> wrote:
>
> On Mon, Apr 12, 2021 at 5:31 PM Yonghong Song <yhs@fb.com> wrote:
> >
> >
> >
> > On 4/12/21 5:02 PM, Nick Desaulniers wrote:
> > > On Mon, Apr 12, 2021 at 4:58 PM Nick Desaulniers
> > > <ndesaulniers@google.com> wrote:
> > >>
> > >> On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
> > >>>
> > >>> To build kernel with clang, people typically use
> > >>>    make -j60 LLVM=1 LLVM_IAS=1
> > >>> LLVM_IAS=1 is not required for non-LTO build but
> > >>> is required for LTO build. In my environment,
> > >>> I am always having LLVM_IAS=1 regardless of
> > >>> whether LTO is enabled or not.
> > >>>
> > >>> After kernel is build with clang, the following command
> > >>> can be used to build selftests with clang:
> > >>>    make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
> > >>
> > >> Thank you for the series Yonghong.  When I test the above command with
> > >> your series applied, I observe:
> > >> tools/include/tools/libc_compat.h:11:21: error: static declaration of
> > >> 'reallocarray' follows non-static declaration
> > >> static inline void *reallocarray(void *ptr, size_t nmemb, size_t size)
> > >>                      ^
> > >> /usr/include/stdlib.h:559:14: note: previous declaration is here
> > >> extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
> > >>               ^
> > >> so perhaps the detection of
> > >> COMPAT_NEED_REALLOCARRAY/feature-reallocarray is incorrect?
> > >
> > > Is this related to _DEFAULT_SOURCE vs _GNU_SOURCE.  via man 3 reallocarray:
> > >         reallocarray():
> > >             Since glibc 2.29:
> > >                 _DEFAULT_SOURCE
> > >             Glibc 2.28 and earlier:
> > >                 _GNU_SOURCE
> > >
> >
> > You can try the following patch to see whether it works or not.
> >
> > diff --git a/tools/build/feature/test-reallocarray.c
> > b/tools/build/feature/test-reallocarray.c
> > index 8f6743e31da7..500cdeca07a7 100644
> > --- a/tools/build/feature/test-reallocarray.c
> > +++ b/tools/build/feature/test-reallocarray.c
> > @@ -1,5 +1,5 @@
> >   // SPDX-License-Identifier: GPL-2.0
> > -#define _GNU_SOURCE
> > +#define _DEFAULT_SOURCE
> >   #include <stdlib.h>
> >
> >   int main(void)
> > @@ -7,4 +7,4 @@ int main(void)
> >          return !!reallocarray(NULL, 1, 1);
> >   }
> >
> > -#undef _GNU_SOURCE
> > +#undef _DEFAULT_SOURCE
>
> Yeah, I had tried that. No luck though; same error message.  Even:
>
> $ cat foo.c
> #define _DEFAULT_SOURCE
> #include <stdlib.h>
> void *reallocarray(void *ptr, size_t nmemb, size_t size) { return (void*)0; };
> $ clang -c foo.c
> $ echo $?
> 0
>
> So I'm not sure precisely what's going on here.  I probably have to go
> digging around to understand tools/build/feature/ anyways.  With your
> v3 applied, I consistently see:
> No zlib found
> and yet, I certainly do have zlib on my host.
> https://stackoverflow.com/a/54558861

Jiri, any tips on how to debug feature detection in
tools/build/feature/Makefile?

>
> > [yhs@devbig003.ftw2 ~/work/bpf-next/tools/build]$
> >
> > > $ cd tools/testing/selftests/bpf
> > > $ grep -rn _DEFAULT_SOURCE | wc -l
> > > 0
> > > $ grep -rn _GNU_SOURCE | wc -l
> > > 37
> > > $ ldd --version | head -n1
> > > ldd (Debian GLIBC 2.31-9+build1) 2.31
> > >
>
>
>
> --
> Thanks,
> ~Nick Desaulniers



-- 
Thanks,
~Nick Desaulniers

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Jiri Olsa]

On Tue, Apr 13, 2021 at 11:56:33AM -0700, Nick Desaulniers wrote:
> On Tue, Apr 13, 2021 at 11:46 AM Nick Desaulniers
> <ndesaulniers@google.com> wrote:
> >
> > On Mon, Apr 12, 2021 at 5:31 PM Yonghong Song <yhs@fb.com> wrote:
> > >
> > >
> > >
> > > On 4/12/21 5:02 PM, Nick Desaulniers wrote:
> > > > On Mon, Apr 12, 2021 at 4:58 PM Nick Desaulniers
> > > > <ndesaulniers@google.com> wrote:
> > > >>
> > > >> On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
> > > >>>
> > > >>> To build kernel with clang, people typically use
> > > >>>    make -j60 LLVM=1 LLVM_IAS=1
> > > >>> LLVM_IAS=1 is not required for non-LTO build but
> > > >>> is required for LTO build. In my environment,
> > > >>> I am always having LLVM_IAS=1 regardless of
> > > >>> whether LTO is enabled or not.
> > > >>>
> > > >>> After kernel is build with clang, the following command
> > > >>> can be used to build selftests with clang:
> > > >>>    make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
> > > >>
> > > >> Thank you for the series Yonghong.  When I test the above command with
> > > >> your series applied, I observe:
> > > >> tools/include/tools/libc_compat.h:11:21: error: static declaration of
> > > >> 'reallocarray' follows non-static declaration
> > > >> static inline void *reallocarray(void *ptr, size_t nmemb, size_t size)
> > > >>                      ^
> > > >> /usr/include/stdlib.h:559:14: note: previous declaration is here
> > > >> extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
> > > >>               ^
> > > >> so perhaps the detection of
> > > >> COMPAT_NEED_REALLOCARRAY/feature-reallocarray is incorrect?
> > > >
> > > > Is this related to _DEFAULT_SOURCE vs _GNU_SOURCE.  via man 3 reallocarray:
> > > >         reallocarray():
> > > >             Since glibc 2.29:
> > > >                 _DEFAULT_SOURCE
> > > >             Glibc 2.28 and earlier:
> > > >                 _GNU_SOURCE
> > > >
> > >
> > > You can try the following patch to see whether it works or not.
> > >
> > > diff --git a/tools/build/feature/test-reallocarray.c
> > > b/tools/build/feature/test-reallocarray.c
> > > index 8f6743e31da7..500cdeca07a7 100644
> > > --- a/tools/build/feature/test-reallocarray.c
> > > +++ b/tools/build/feature/test-reallocarray.c
> > > @@ -1,5 +1,5 @@
> > >   // SPDX-License-Identifier: GPL-2.0
> > > -#define _GNU_SOURCE
> > > +#define _DEFAULT_SOURCE
> > >   #include <stdlib.h>
> > >
> > >   int main(void)
> > > @@ -7,4 +7,4 @@ int main(void)
> > >          return !!reallocarray(NULL, 1, 1);
> > >   }
> > >
> > > -#undef _GNU_SOURCE
> > > +#undef _DEFAULT_SOURCE
> >
> > Yeah, I had tried that. No luck though; same error message.  Even:
> >
> > $ cat foo.c
> > #define _DEFAULT_SOURCE
> > #include <stdlib.h>
> > void *reallocarray(void *ptr, size_t nmemb, size_t size) { return (void*)0; };
> > $ clang -c foo.c
> > $ echo $?
> > 0
> >
> > So I'm not sure precisely what's going on here.  I probably have to go
> > digging around to understand tools/build/feature/ anyways.  With your
> > v3 applied, I consistently see:
> > No zlib found
> > and yet, I certainly do have zlib on my host.
> > https://stackoverflow.com/a/54558861
> 
> Jiri, any tips on how to debug feature detection in
> tools/build/feature/Makefile?

for quick check, there's output file for each test, like:

	[jolsa@krava feature]$ ls -l *.make.output
	-rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-all.make.output
	-rw-rw-r--. 1 jolsa jolsa 182 Apr  9 15:52 test-bionic.make.output
	-rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-cplus-demangle.make.output
	-rw-rw-r--. 1 jolsa jolsa 145 Apr  9 15:52 test-jvmti.make.output
	-rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbabeltrace.make.output
	-rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbpf.make.output
	-rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libdebuginfod.make.output
	-rw-rw-r--. 1 jolsa jolsa 193 Apr  9 15:52 test-libunwind-aarch64.make.output
	-rw-rw-r--. 1 jolsa jolsa 177 Apr  9 15:52 test-libunwind-x86.make.output
	[jolsa@krava feature]$ cat test-libunwind-aarch64.make.output
	test-libunwind-aarch64.c:2:10: fatal error: libunwind-aarch64.h: No such file or directory
	    2 | #include <libunwind-aarch64.h>
	      |          ^~~~~~~~~~~~~~~~~~~~~
	compilation terminated.
	[jolsa@krava feature]$ cat test-libunwind-x86.make.output
	test-libunwind-x86.c:2:10: fatal error: libunwind-x86.h: No such file or directory
	    2 | #include <libunwind-x86.h>
	      |          ^~~~~~~~~~~~~~~~~

zlib should be done by:
	[jolsa@krava feature]$ make test-zlib.bin
	gcc  -MD -Wall -Werror -o test-zlib.bin test-zlib.c  > test-zlib.make.output 2>&1 -lz


I can try to recreate, how do you build?

jirka

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Nick Desaulniers]

On Tue, Apr 13, 2021 at 1:36 PM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Tue, Apr 13, 2021 at 11:56:33AM -0700, Nick Desaulniers wrote:
> > On Tue, Apr 13, 2021 at 11:46 AM Nick Desaulniers
> > <ndesaulniers@google.com> wrote:
> > >
> > > On Mon, Apr 12, 2021 at 5:31 PM Yonghong Song <yhs@fb.com> wrote:
> > > >
> > > >
> > > >
> > > > On 4/12/21 5:02 PM, Nick Desaulniers wrote:
> > > > > On Mon, Apr 12, 2021 at 4:58 PM Nick Desaulniers
> > > > > <ndesaulniers@google.com> wrote:
> > > > >>
> > > > >> On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
> > > > >>>
> > > > >>> To build kernel with clang, people typically use
> > > > >>>    make -j60 LLVM=1 LLVM_IAS=1
> > > > >>> LLVM_IAS=1 is not required for non-LTO build but
> > > > >>> is required for LTO build. In my environment,
> > > > >>> I am always having LLVM_IAS=1 regardless of
> > > > >>> whether LTO is enabled or not.
> > > > >>>
> > > > >>> After kernel is build with clang, the following command
> > > > >>> can be used to build selftests with clang:
> > > > >>>    make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1

^ note below

> > > > >>
> > > > >> Thank you for the series Yonghong.  When I test the above command with
> > > > >> your series applied, I observe:
> > > > >> tools/include/tools/libc_compat.h:11:21: error: static declaration of
> > > > >> 'reallocarray' follows non-static declaration
> > > > >> static inline void *reallocarray(void *ptr, size_t nmemb, size_t size)
> > > > >>                      ^
> > > > >> /usr/include/stdlib.h:559:14: note: previous declaration is here
> > > > >> extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
> > > > >>               ^
> > > > >> so perhaps the detection of
> > > > >> COMPAT_NEED_REALLOCARRAY/feature-reallocarray is incorrect?
> > > > >
> > > > > Is this related to _DEFAULT_SOURCE vs _GNU_SOURCE.  via man 3 reallocarray:
> > > > >         reallocarray():
> > > > >             Since glibc 2.29:
> > > > >                 _DEFAULT_SOURCE
> > > > >             Glibc 2.28 and earlier:
> > > > >                 _GNU_SOURCE
> > > > >
> > > >
> > > > You can try the following patch to see whether it works or not.
> > > >
> > > > diff --git a/tools/build/feature/test-reallocarray.c
> > > > b/tools/build/feature/test-reallocarray.c
> > > > index 8f6743e31da7..500cdeca07a7 100644
> > > > --- a/tools/build/feature/test-reallocarray.c
> > > > +++ b/tools/build/feature/test-reallocarray.c
> > > > @@ -1,5 +1,5 @@
> > > >   // SPDX-License-Identifier: GPL-2.0
> > > > -#define _GNU_SOURCE
> > > > +#define _DEFAULT_SOURCE
> > > >   #include <stdlib.h>
> > > >
> > > >   int main(void)
> > > > @@ -7,4 +7,4 @@ int main(void)
> > > >          return !!reallocarray(NULL, 1, 1);
> > > >   }
> > > >
> > > > -#undef _GNU_SOURCE
> > > > +#undef _DEFAULT_SOURCE
> > >
> > > Yeah, I had tried that. No luck though; same error message.  Even:
> > >
> > > $ cat foo.c
> > > #define _DEFAULT_SOURCE
> > > #include <stdlib.h>
> > > void *reallocarray(void *ptr, size_t nmemb, size_t size) { return (void*)0; };
> > > $ clang -c foo.c
> > > $ echo $?
> > > 0
> > >
> > > So I'm not sure precisely what's going on here.  I probably have to go
> > > digging around to understand tools/build/feature/ anyways.  With your
> > > v3 applied, I consistently see:
> > > No zlib found
> > > and yet, I certainly do have zlib on my host.
> > > https://stackoverflow.com/a/54558861
> >
> > Jiri, any tips on how to debug feature detection in
> > tools/build/feature/Makefile?
>
> for quick check, there's output file for each test, like:
>
>         [jolsa@krava feature]$ ls -l *.make.output
>         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-all.make.output
>         -rw-rw-r--. 1 jolsa jolsa 182 Apr  9 15:52 test-bionic.make.output
>         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-cplus-demangle.make.output
>         -rw-rw-r--. 1 jolsa jolsa 145 Apr  9 15:52 test-jvmti.make.output
>         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbabeltrace.make.output
>         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbpf.make.output
>         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libdebuginfod.make.output
>         -rw-rw-r--. 1 jolsa jolsa 193 Apr  9 15:52 test-libunwind-aarch64.make.output
>         -rw-rw-r--. 1 jolsa jolsa 177 Apr  9 15:52 test-libunwind-x86.make.output
>         [jolsa@krava feature]$ cat test-libunwind-aarch64.make.output
>         test-libunwind-aarch64.c:2:10: fatal error: libunwind-aarch64.h: No such file or directory
>             2 | #include <libunwind-aarch64.h>
>               |          ^~~~~~~~~~~~~~~~~~~~~
>         compilation terminated.
>         [jolsa@krava feature]$ cat test-libunwind-x86.make.output
>         test-libunwind-x86.c:2:10: fatal error: libunwind-x86.h: No such file or directory
>             2 | #include <libunwind-x86.h>
>               |          ^~~~~~~~~~~~~~~~~
>
> zlib should be done by:
>         [jolsa@krava feature]$ make test-zlib.bin
>         gcc  -MD -Wall -Werror -o test-zlib.bin test-zlib.c  > test-zlib.make.output 2>&1 -lz
>
>
> I can try to recreate, how do you build?

See note above, I'm similarly running precisely:
$ make LLVM=1 LLVM_IAS=1 -j72 defconfig
$ make LLVM=1 LLVM_IAS=1 -j72 clean
$ make LLVM=1 LLVM_IAS=1 -j72 -C tools/testing/selftests/bpf

-- 
Thanks,
~Nick Desaulniers

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Jiri Olsa]

On Tue, Apr 13, 2021 at 01:45:39PM -0700, Nick Desaulniers wrote:

SNIP

> > > >
> > > > So I'm not sure precisely what's going on here.  I probably have to go
> > > > digging around to understand tools/build/feature/ anyways.  With your
> > > > v3 applied, I consistently see:
> > > > No zlib found
> > > > and yet, I certainly do have zlib on my host.
> > > > https://stackoverflow.com/a/54558861
> > >
> > > Jiri, any tips on how to debug feature detection in
> > > tools/build/feature/Makefile?
> >
> > for quick check, there's output file for each test, like:
> >
> >         [jolsa@krava feature]$ ls -l *.make.output
> >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-all.make.output
> >         -rw-rw-r--. 1 jolsa jolsa 182 Apr  9 15:52 test-bionic.make.output
> >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-cplus-demangle.make.output
> >         -rw-rw-r--. 1 jolsa jolsa 145 Apr  9 15:52 test-jvmti.make.output
> >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbabeltrace.make.output
> >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbpf.make.output
> >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libdebuginfod.make.output
> >         -rw-rw-r--. 1 jolsa jolsa 193 Apr  9 15:52 test-libunwind-aarch64.make.output
> >         -rw-rw-r--. 1 jolsa jolsa 177 Apr  9 15:52 test-libunwind-x86.make.output
> >         [jolsa@krava feature]$ cat test-libunwind-aarch64.make.output
> >         test-libunwind-aarch64.c:2:10: fatal error: libunwind-aarch64.h: No such file or directory
> >             2 | #include <libunwind-aarch64.h>
> >               |          ^~~~~~~~~~~~~~~~~~~~~
> >         compilation terminated.
> >         [jolsa@krava feature]$ cat test-libunwind-x86.make.output
> >         test-libunwind-x86.c:2:10: fatal error: libunwind-x86.h: No such file or directory
> >             2 | #include <libunwind-x86.h>
> >               |          ^~~~~~~~~~~~~~~~~
> >
> > zlib should be done by:
> >         [jolsa@krava feature]$ make test-zlib.bin
> >         gcc  -MD -Wall -Werror -o test-zlib.bin test-zlib.c  > test-zlib.make.output 2>&1 -lz
> >
> >
> > I can try to recreate, how do you build?
> 
> See note above, I'm similarly running precisely:
> $ make LLVM=1 LLVM_IAS=1 -j72 defconfig
> $ make LLVM=1 LLVM_IAS=1 -j72 clean
> $ make LLVM=1 LLVM_IAS=1 -j72 -C tools/testing/selftests/bpf

for some reason I'm stuck with this error on latest bpf-next/master

$ make LLVM=1 LLVM_IAS=1 -C tools/testing/selftests/bpf

make[1]: Nothing to be done for 'docs'.
  CLNG-BPF [test_maps] test_lwt_ip_encap.o
  CLNG-BPF [test_maps] test_tc_edt.o
  CLNG-BPF [test_maps] local_storage.o
progs/local_storage.c:41:15: error: use of undeclared identifier 'BPF_MAP_TYPE_TASK_STORAGE'; did you mean 'BPF_MAP_TYPE_SK_STORAGE'?
        __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~
                     BPF_MAP_TYPE_SK_STORAGE
/home/jolsa/linux/tools/testing/selftests/bpf/tools/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint'
#define __uint(name, val) int (*name)[val]
                                      ^
/home/jolsa/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h:10317:2: note: 'BPF_MAP_TYPE_SK_STORAGE' declared here
        BPF_MAP_TYPE_SK_STORAGE = 24,
        ^
1 error generated.
make: *** [Makefile:448: /home/jolsa/linux/tools/testing/selftests/bpf/local_storage.o] Error 1
make: Leaving directory '/home/jolsa/linux/tools/testing/selftests/bpf'


jirka

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Andrii Nakryiko]

On Wed, Apr 14, 2021 at 6:18 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Tue, Apr 13, 2021 at 01:45:39PM -0700, Nick Desaulniers wrote:
>
> SNIP
>
> > > > >
> > > > > So I'm not sure precisely what's going on here.  I probably have to go
> > > > > digging around to understand tools/build/feature/ anyways.  With your
> > > > > v3 applied, I consistently see:
> > > > > No zlib found
> > > > > and yet, I certainly do have zlib on my host.
> > > > > https://stackoverflow.com/a/54558861
> > > >
> > > > Jiri, any tips on how to debug feature detection in
> > > > tools/build/feature/Makefile?
> > >
> > > for quick check, there's output file for each test, like:
> > >
> > >         [jolsa@krava feature]$ ls -l *.make.output
> > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-all.make.output
> > >         -rw-rw-r--. 1 jolsa jolsa 182 Apr  9 15:52 test-bionic.make.output
> > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-cplus-demangle.make.output
> > >         -rw-rw-r--. 1 jolsa jolsa 145 Apr  9 15:52 test-jvmti.make.output
> > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbabeltrace.make.output
> > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbpf.make.output
> > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libdebuginfod.make.output
> > >         -rw-rw-r--. 1 jolsa jolsa 193 Apr  9 15:52 test-libunwind-aarch64.make.output
> > >         -rw-rw-r--. 1 jolsa jolsa 177 Apr  9 15:52 test-libunwind-x86.make.output
> > >         [jolsa@krava feature]$ cat test-libunwind-aarch64.make.output
> > >         test-libunwind-aarch64.c:2:10: fatal error: libunwind-aarch64.h: No such file or directory
> > >             2 | #include <libunwind-aarch64.h>
> > >               |          ^~~~~~~~~~~~~~~~~~~~~
> > >         compilation terminated.
> > >         [jolsa@krava feature]$ cat test-libunwind-x86.make.output
> > >         test-libunwind-x86.c:2:10: fatal error: libunwind-x86.h: No such file or directory
> > >             2 | #include <libunwind-x86.h>
> > >               |          ^~~~~~~~~~~~~~~~~
> > >
> > > zlib should be done by:
> > >         [jolsa@krava feature]$ make test-zlib.bin
> > >         gcc  -MD -Wall -Werror -o test-zlib.bin test-zlib.c  > test-zlib.make.output 2>&1 -lz
> > >
> > >
> > > I can try to recreate, how do you build?
> >
> > See note above, I'm similarly running precisely:
> > $ make LLVM=1 LLVM_IAS=1 -j72 defconfig
> > $ make LLVM=1 LLVM_IAS=1 -j72 clean
> > $ make LLVM=1 LLVM_IAS=1 -j72 -C tools/testing/selftests/bpf
>
> for some reason I'm stuck with this error on latest bpf-next/master

did you build vmlinux image before building selftests? those enums
should come through vmlinux.h from up-to-date vmlinux

>
> $ make LLVM=1 LLVM_IAS=1 -C tools/testing/selftests/bpf
>
> make[1]: Nothing to be done for 'docs'.
>   CLNG-BPF [test_maps] test_lwt_ip_encap.o
>   CLNG-BPF [test_maps] test_tc_edt.o
>   CLNG-BPF [test_maps] local_storage.o
> progs/local_storage.c:41:15: error: use of undeclared identifier 'BPF_MAP_TYPE_TASK_STORAGE'; did you mean 'BPF_MAP_TYPE_SK_STORAGE'?
>         __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
>                      ^~~~~~~~~~~~~~~~~~~~~~~~~
>                      BPF_MAP_TYPE_SK_STORAGE
> /home/jolsa/linux/tools/testing/selftests/bpf/tools/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint'
> #define __uint(name, val) int (*name)[val]
>                                       ^
> /home/jolsa/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h:10317:2: note: 'BPF_MAP_TYPE_SK_STORAGE' declared here
>         BPF_MAP_TYPE_SK_STORAGE = 24,
>         ^
> 1 error generated.
> make: *** [Makefile:448: /home/jolsa/linux/tools/testing/selftests/bpf/local_storage.o] Error 1
> make: Leaving directory '/home/jolsa/linux/tools/testing/selftests/bpf'
>
>
> jirka
>

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Jiri Olsa]

On Wed, Apr 14, 2021 at 05:16:01PM -0700, Andrii Nakryiko wrote:
> On Wed, Apr 14, 2021 at 6:18 AM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > On Tue, Apr 13, 2021 at 01:45:39PM -0700, Nick Desaulniers wrote:
> >
> > SNIP
> >
> > > > > >
> > > > > > So I'm not sure precisely what's going on here.  I probably have to go
> > > > > > digging around to understand tools/build/feature/ anyways.  With your
> > > > > > v3 applied, I consistently see:
> > > > > > No zlib found
> > > > > > and yet, I certainly do have zlib on my host.
> > > > > > https://stackoverflow.com/a/54558861
> > > > >
> > > > > Jiri, any tips on how to debug feature detection in
> > > > > tools/build/feature/Makefile?
> > > >
> > > > for quick check, there's output file for each test, like:
> > > >
> > > >         [jolsa@krava feature]$ ls -l *.make.output
> > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-all.make.output
> > > >         -rw-rw-r--. 1 jolsa jolsa 182 Apr  9 15:52 test-bionic.make.output
> > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-cplus-demangle.make.output
> > > >         -rw-rw-r--. 1 jolsa jolsa 145 Apr  9 15:52 test-jvmti.make.output
> > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbabeltrace.make.output
> > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbpf.make.output
> > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libdebuginfod.make.output
> > > >         -rw-rw-r--. 1 jolsa jolsa 193 Apr  9 15:52 test-libunwind-aarch64.make.output
> > > >         -rw-rw-r--. 1 jolsa jolsa 177 Apr  9 15:52 test-libunwind-x86.make.output
> > > >         [jolsa@krava feature]$ cat test-libunwind-aarch64.make.output
> > > >         test-libunwind-aarch64.c:2:10: fatal error: libunwind-aarch64.h: No such file or directory
> > > >             2 | #include <libunwind-aarch64.h>
> > > >               |          ^~~~~~~~~~~~~~~~~~~~~
> > > >         compilation terminated.
> > > >         [jolsa@krava feature]$ cat test-libunwind-x86.make.output
> > > >         test-libunwind-x86.c:2:10: fatal error: libunwind-x86.h: No such file or directory
> > > >             2 | #include <libunwind-x86.h>
> > > >               |          ^~~~~~~~~~~~~~~~~
> > > >
> > > > zlib should be done by:
> > > >         [jolsa@krava feature]$ make test-zlib.bin
> > > >         gcc  -MD -Wall -Werror -o test-zlib.bin test-zlib.c  > test-zlib.make.output 2>&1 -lz
> > > >
> > > >
> > > > I can try to recreate, how do you build?
> > >
> > > See note above, I'm similarly running precisely:
> > > $ make LLVM=1 LLVM_IAS=1 -j72 defconfig
> > > $ make LLVM=1 LLVM_IAS=1 -j72 clean
> > > $ make LLVM=1 LLVM_IAS=1 -j72 -C tools/testing/selftests/bpf
> >
> > for some reason I'm stuck with this error on latest bpf-next/master
> 
> did you build vmlinux image before building selftests? those enums
> should come through vmlinux.h from up-to-date vmlinux

it was there.. but I found the clang/lld I compiled can't link properly,
which is probably unrelated to the error below, but I need to solve it
first ;-)

jirka

> 
> >
> > $ make LLVM=1 LLVM_IAS=1 -C tools/testing/selftests/bpf
> >
> > make[1]: Nothing to be done for 'docs'.
> >   CLNG-BPF [test_maps] test_lwt_ip_encap.o
> >   CLNG-BPF [test_maps] test_tc_edt.o
> >   CLNG-BPF [test_maps] local_storage.o
> > progs/local_storage.c:41:15: error: use of undeclared identifier 'BPF_MAP_TYPE_TASK_STORAGE'; did you mean 'BPF_MAP_TYPE_SK_STORAGE'?
> >         __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
> >                      ^~~~~~~~~~~~~~~~~~~~~~~~~
> >                      BPF_MAP_TYPE_SK_STORAGE
> > /home/jolsa/linux/tools/testing/selftests/bpf/tools/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint'
> > #define __uint(name, val) int (*name)[val]
> >                                       ^
> > /home/jolsa/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h:10317:2: note: 'BPF_MAP_TYPE_SK_STORAGE' declared here
> >         BPF_MAP_TYPE_SK_STORAGE = 24,
> >         ^
> > 1 error generated.
> > make: *** [Makefile:448: /home/jolsa/linux/tools/testing/selftests/bpf/local_storage.o] Error 1
> > make: Leaving directory '/home/jolsa/linux/tools/testing/selftests/bpf'
> >
> >
> > jirka
> >
> 

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Nick Desaulniers]

On Thu, Apr 15, 2021 at 6:23 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Wed, Apr 14, 2021 at 05:16:01PM -0700, Andrii Nakryiko wrote:
> > On Wed, Apr 14, 2021 at 6:18 AM Jiri Olsa <jolsa@redhat.com> wrote:
> > >
> > > On Tue, Apr 13, 2021 at 01:45:39PM -0700, Nick Desaulniers wrote:
> > >
> > > SNIP
> > >
> > > > > > >
> > > > > > > So I'm not sure precisely what's going on here.  I probably have to go
> > > > > > > digging around to understand tools/build/feature/ anyways.  With your
> > > > > > > v3 applied, I consistently see:
> > > > > > > No zlib found
> > > > > > > and yet, I certainly do have zlib on my host.
> > > > > > > https://stackoverflow.com/a/54558861
> > > > > >
> > > > > > Jiri, any tips on how to debug feature detection in
> > > > > > tools/build/feature/Makefile?
> > > > >
> > > > > for quick check, there's output file for each test, like:
> > > > >
> > > > >         [jolsa@krava feature]$ ls -l *.make.output
> > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-all.make.output
> > > > >         -rw-rw-r--. 1 jolsa jolsa 182 Apr  9 15:52 test-bionic.make.output
> > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-cplus-demangle.make.output
> > > > >         -rw-rw-r--. 1 jolsa jolsa 145 Apr  9 15:52 test-jvmti.make.output
> > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbabeltrace.make.output
> > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbpf.make.output
> > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libdebuginfod.make.output
> > > > >         -rw-rw-r--. 1 jolsa jolsa 193 Apr  9 15:52 test-libunwind-aarch64.make.output
> > > > >         -rw-rw-r--. 1 jolsa jolsa 177 Apr  9 15:52 test-libunwind-x86.make.output
> > > > >         [jolsa@krava feature]$ cat test-libunwind-aarch64.make.output
> > > > >         test-libunwind-aarch64.c:2:10: fatal error: libunwind-aarch64.h: No such file or directory
> > > > >             2 | #include <libunwind-aarch64.h>
> > > > >               |          ^~~~~~~~~~~~~~~~~~~~~
> > > > >         compilation terminated.
> > > > >         [jolsa@krava feature]$ cat test-libunwind-x86.make.output
> > > > >         test-libunwind-x86.c:2:10: fatal error: libunwind-x86.h: No such file or directory
> > > > >             2 | #include <libunwind-x86.h>
> > > > >               |          ^~~~~~~~~~~~~~~~~
> > > > >
> > > > > zlib should be done by:
> > > > >         [jolsa@krava feature]$ make test-zlib.bin
> > > > >         gcc  -MD -Wall -Werror -o test-zlib.bin test-zlib.c  > test-zlib.make.output 2>&1 -lz
> > > > >
> > > > >
> > > > > I can try to recreate, how do you build?
> > > >
> > > > See note above, I'm similarly running precisely:
> > > > $ make LLVM=1 LLVM_IAS=1 -j72 defconfig
> > > > $ make LLVM=1 LLVM_IAS=1 -j72 clean
> > > > $ make LLVM=1 LLVM_IAS=1 -j72 -C tools/testing/selftests/bpf
> > >
> > > for some reason I'm stuck with this error on latest bpf-next/master
> >
> > did you build vmlinux image before building selftests? those enums
> > should come through vmlinux.h from up-to-date vmlinux
>
> it was there.. but I found the clang/lld I compiled can't link properly,
> which is probably unrelated to the error below, but I need to solve it
> first ;-)

Do you have more info about what command you're running, or what error
you're observing?  The kernel itself is not linking for you?


>
> jirka
>
> >
> > >
> > > $ make LLVM=1 LLVM_IAS=1 -C tools/testing/selftests/bpf
> > >
> > > make[1]: Nothing to be done for 'docs'.
> > >   CLNG-BPF [test_maps] test_lwt_ip_encap.o
> > >   CLNG-BPF [test_maps] test_tc_edt.o
> > >   CLNG-BPF [test_maps] local_storage.o
> > > progs/local_storage.c:41:15: error: use of undeclared identifier 'BPF_MAP_TYPE_TASK_STORAGE'; did you mean 'BPF_MAP_TYPE_SK_STORAGE'?
> > >         __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
> > >                      ^~~~~~~~~~~~~~~~~~~~~~~~~
> > >                      BPF_MAP_TYPE_SK_STORAGE
> > > /home/jolsa/linux/tools/testing/selftests/bpf/tools/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint'
> > > #define __uint(name, val) int (*name)[val]
> > >                                       ^
> > > /home/jolsa/linux/tools/testing/selftests/bpf/tools/include/vmlinux.h:10317:2: note: 'BPF_MAP_TYPE_SK_STORAGE' declared here
> > >         BPF_MAP_TYPE_SK_STORAGE = 24,
> > >         ^
> > > 1 error generated.
> > > make: *** [Makefile:448: /home/jolsa/linux/tools/testing/selftests/bpf/local_storage.o] Error 1
> > > make: Leaving directory '/home/jolsa/linux/tools/testing/selftests/bpf'
> > >
> > >
> > > jirka
> > >
> >
>


-- 
Thanks,
~Nick Desaulniers

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Jiri Olsa]

On Thu, Apr 15, 2021 at 09:55:50AM -0700, Nick Desaulniers wrote:
> On Thu, Apr 15, 2021 at 6:23 AM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > On Wed, Apr 14, 2021 at 05:16:01PM -0700, Andrii Nakryiko wrote:
> > > On Wed, Apr 14, 2021 at 6:18 AM Jiri Olsa <jolsa@redhat.com> wrote:
> > > >
> > > > On Tue, Apr 13, 2021 at 01:45:39PM -0700, Nick Desaulniers wrote:
> > > >
> > > > SNIP
> > > >
> > > > > > > >
> > > > > > > > So I'm not sure precisely what's going on here.  I probably have to go
> > > > > > > > digging around to understand tools/build/feature/ anyways.  With your
> > > > > > > > v3 applied, I consistently see:
> > > > > > > > No zlib found
> > > > > > > > and yet, I certainly do have zlib on my host.
> > > > > > > > https://stackoverflow.com/a/54558861
> > > > > > >
> > > > > > > Jiri, any tips on how to debug feature detection in
> > > > > > > tools/build/feature/Makefile?
> > > > > >
> > > > > > for quick check, there's output file for each test, like:
> > > > > >
> > > > > >         [jolsa@krava feature]$ ls -l *.make.output
> > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-all.make.output
> > > > > >         -rw-rw-r--. 1 jolsa jolsa 182 Apr  9 15:52 test-bionic.make.output
> > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-cplus-demangle.make.output
> > > > > >         -rw-rw-r--. 1 jolsa jolsa 145 Apr  9 15:52 test-jvmti.make.output
> > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbabeltrace.make.output
> > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbpf.make.output
> > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libdebuginfod.make.output
> > > > > >         -rw-rw-r--. 1 jolsa jolsa 193 Apr  9 15:52 test-libunwind-aarch64.make.output
> > > > > >         -rw-rw-r--. 1 jolsa jolsa 177 Apr  9 15:52 test-libunwind-x86.make.output
> > > > > >         [jolsa@krava feature]$ cat test-libunwind-aarch64.make.output
> > > > > >         test-libunwind-aarch64.c:2:10: fatal error: libunwind-aarch64.h: No such file or directory
> > > > > >             2 | #include <libunwind-aarch64.h>
> > > > > >               |          ^~~~~~~~~~~~~~~~~~~~~
> > > > > >         compilation terminated.
> > > > > >         [jolsa@krava feature]$ cat test-libunwind-x86.make.output
> > > > > >         test-libunwind-x86.c:2:10: fatal error: libunwind-x86.h: No such file or directory
> > > > > >             2 | #include <libunwind-x86.h>
> > > > > >               |          ^~~~~~~~~~~~~~~~~
> > > > > >
> > > > > > zlib should be done by:
> > > > > >         [jolsa@krava feature]$ make test-zlib.bin
> > > > > >         gcc  -MD -Wall -Werror -o test-zlib.bin test-zlib.c  > test-zlib.make.output 2>&1 -lz
> > > > > >
> > > > > >
> > > > > > I can try to recreate, how do you build?
> > > > >
> > > > > See note above, I'm similarly running precisely:
> > > > > $ make LLVM=1 LLVM_IAS=1 -j72 defconfig
> > > > > $ make LLVM=1 LLVM_IAS=1 -j72 clean
> > > > > $ make LLVM=1 LLVM_IAS=1 -j72 -C tools/testing/selftests/bpf
> > > >
> > > > for some reason I'm stuck with this error on latest bpf-next/master
> > >
> > > did you build vmlinux image before building selftests? those enums
> > > should come through vmlinux.h from up-to-date vmlinux
> >
> > it was there.. but I found the clang/lld I compiled can't link properly,
> > which is probably unrelated to the error below, but I need to solve it
> > first ;-)
> 
> Do you have more info about what command you're running, or what error
> you're observing?  The kernel itself is not linking for you?

I did the commands you sent, but I might have but llvm/clang build,
I'm rebuilding now, it takes forever.. 

jirka

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Jiri Olsa]

On Thu, Apr 15, 2021 at 07:17:35PM +0200, Jiri Olsa wrote:
> On Thu, Apr 15, 2021 at 09:55:50AM -0700, Nick Desaulniers wrote:
> > On Thu, Apr 15, 2021 at 6:23 AM Jiri Olsa <jolsa@redhat.com> wrote:
> > >
> > > On Wed, Apr 14, 2021 at 05:16:01PM -0700, Andrii Nakryiko wrote:
> > > > On Wed, Apr 14, 2021 at 6:18 AM Jiri Olsa <jolsa@redhat.com> wrote:
> > > > >
> > > > > On Tue, Apr 13, 2021 at 01:45:39PM -0700, Nick Desaulniers wrote:
> > > > >
> > > > > SNIP
> > > > >
> > > > > > > > >
> > > > > > > > > So I'm not sure precisely what's going on here.  I probably have to go
> > > > > > > > > digging around to understand tools/build/feature/ anyways.  With your
> > > > > > > > > v3 applied, I consistently see:
> > > > > > > > > No zlib found
> > > > > > > > > and yet, I certainly do have zlib on my host.
> > > > > > > > > https://stackoverflow.com/a/54558861
> > > > > > > >
> > > > > > > > Jiri, any tips on how to debug feature detection in
> > > > > > > > tools/build/feature/Makefile?
> > > > > > >
> > > > > > > for quick check, there's output file for each test, like:
> > > > > > >
> > > > > > >         [jolsa@krava feature]$ ls -l *.make.output
> > > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-all.make.output
> > > > > > >         -rw-rw-r--. 1 jolsa jolsa 182 Apr  9 15:52 test-bionic.make.output
> > > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-cplus-demangle.make.output
> > > > > > >         -rw-rw-r--. 1 jolsa jolsa 145 Apr  9 15:52 test-jvmti.make.output
> > > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbabeltrace.make.output
> > > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libbpf.make.output
> > > > > > >         -rw-rw-r--. 1 jolsa jolsa   0 Apr  8 20:25 test-libdebuginfod.make.output
> > > > > > >         -rw-rw-r--. 1 jolsa jolsa 193 Apr  9 15:52 test-libunwind-aarch64.make.output
> > > > > > >         -rw-rw-r--. 1 jolsa jolsa 177 Apr  9 15:52 test-libunwind-x86.make.output
> > > > > > >         [jolsa@krava feature]$ cat test-libunwind-aarch64.make.output
> > > > > > >         test-libunwind-aarch64.c:2:10: fatal error: libunwind-aarch64.h: No such file or directory
> > > > > > >             2 | #include <libunwind-aarch64.h>
> > > > > > >               |          ^~~~~~~~~~~~~~~~~~~~~
> > > > > > >         compilation terminated.
> > > > > > >         [jolsa@krava feature]$ cat test-libunwind-x86.make.output
> > > > > > >         test-libunwind-x86.c:2:10: fatal error: libunwind-x86.h: No such file or directory
> > > > > > >             2 | #include <libunwind-x86.h>
> > > > > > >               |          ^~~~~~~~~~~~~~~~~
> > > > > > >
> > > > > > > zlib should be done by:
> > > > > > >         [jolsa@krava feature]$ make test-zlib.bin
> > > > > > >         gcc  -MD -Wall -Werror -o test-zlib.bin test-zlib.c  > test-zlib.make.output 2>&1 -lz
> > > > > > >
> > > > > > >
> > > > > > > I can try to recreate, how do you build?
> > > > > >
> > > > > > See note above, I'm similarly running precisely:
> > > > > > $ make LLVM=1 LLVM_IAS=1 -j72 defconfig
> > > > > > $ make LLVM=1 LLVM_IAS=1 -j72 clean
> > > > > > $ make LLVM=1 LLVM_IAS=1 -j72 -C tools/testing/selftests/bpf
> > > > >
> > > > > for some reason I'm stuck with this error on latest bpf-next/master
> > > >
> > > > did you build vmlinux image before building selftests? those enums
> > > > should come through vmlinux.h from up-to-date vmlinux
> > >
> > > it was there.. but I found the clang/lld I compiled can't link properly,
> > > which is probably unrelated to the error below, but I need to solve it
> > > first ;-)
> > 
> > Do you have more info about what command you're running, or what error
> > you're observing?  The kernel itself is not linking for you?
> 
> I did the commands you sent, but I might have but llvm/clang build,
> I'm rebuilding now, it takes forever.. 

ok, finished all commands finaly.. did not reproduce the issue :-\
any luck with those build make.output files for zlib?

jirka

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Yonghong Song]

On 4/12/21 4:58 PM, Nick Desaulniers wrote:
> On Mon, Apr 12, 2021 at 7:29 AM Yonghong Song <yhs@fb.com> wrote:
>>
>> To build kernel with clang, people typically use
>>    make -j60 LLVM=1 LLVM_IAS=1
>> LLVM_IAS=1 is not required for non-LTO build but
>> is required for LTO build. In my environment,
>> I am always having LLVM_IAS=1 regardless of
>> whether LTO is enabled or not.
>>
>> After kernel is build with clang, the following command
>> can be used to build selftests with clang:
>>    make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
> 
> Thank you for the series Yonghong.  When I test the above command with
> your series applied, I observe:
> /usr/bin/ld: cannot find -lcap
> clang-13: error: linker command failed with exit code 1 (use -v to see
> invocation)
> 
> I need to install libcap-dev, but this also seems to imply that BFD is
> being used as the linker, not LLD.  Perhaps if the compiler is being
> used as the "driver" to also link executables, `-fuse-ld=lld` is
> needed for the compiler flags.

Yes, bfd is needed to build selftests/bpf. This is due to a dependency
on bpftool which needs uses libbfd to disassemble the jited code.

> 
> Then there's:
> tools/include/tools/libc_compat.h:11:21: error: static declaration of
> 'reallocarray' follows non-static declaration
> static inline void *reallocarray(void *ptr, size_t nmemb, size_t size)
>                      ^
> /usr/include/stdlib.h:559:14: note: previous declaration is here
> extern void *reallocarray (void *__ptr, size_t __nmemb, size_t __size)
>               ^
> so perhaps the detection of
> COMPAT_NEED_REALLOCARRAY/feature-reallocarray is incorrect?

libbpf already stopped to use system reallocarray(), but
bpftool still uses it.

In bpftool makefile, we have

ifeq ($(feature-reallocarray), 0)
CFLAGS += -DCOMPAT_NEED_REALLOCARRAY
endif

I guess probably detection of feature-reallocarray is
not correct? Could you take a look at your system?
My system supports reallocarray, so the above
-DCOMPAT_NEED_REALLOCARRAY is not added to
compilation flags.

> 
> 
>>
>> But currently, using the above command, some compilations
>> still use gcc and there are also compilation errors and warnings.
>> This patch set intends to fix these issues.
>> Patch #1 and #2 fixed the issue so clang/clang++ is
>> used instead of gcc/g++. Patch #3 fixed a compilation
>> failure. Patch #4 and #5 fixed various compiler warnings.
>>
>> Changelog:
>>    v1 -> v2:
>>      . add -Wno-unused-command-line-argument and -Wno-format-security
>>        for clang only as (1). gcc does not exhibit those
>>        warnings, and (2). -Wno-unused-command-line-argument is
>>        only supported by clang. (Sedat)
>>
>> Yonghong Song (5):
>>    selftests: set CC to clang in lib.mk if LLVM is set
>>    tools: allow proper CC/CXX/... override with LLVM=1 in
>>      Makefile.include
>>    selftests/bpf: fix test_cpp compilation failure with clang
>>    selftests/bpf: silence clang compilation warnings
>>    bpftool: fix a clang compilation warning
>>
>>   tools/bpf/bpftool/net.c              |  2 +-
>>   tools/scripts/Makefile.include       | 12 ++++++++++--
>>   tools/testing/selftests/bpf/Makefile |  7 ++++++-
>>   tools/testing/selftests/lib.mk       |  4 ++++
>>   4 files changed, 21 insertions(+), 4 deletions(-)
>>
>> --
>> 2.30.2
>>
> 
> 
> --
> Thanks,
> ~Nick Desaulniers
> 

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Sedat Dilek]

On Mon, Apr 12, 2021 at 4:29 PM Yonghong Song <yhs@fb.com> wrote:
>
> To build kernel with clang, people typically use
>   make -j60 LLVM=1 LLVM_IAS=1
> LLVM_IAS=1 is not required for non-LTO build but
> is required for LTO build. In my environment,
> I am always having LLVM_IAS=1 regardless of
> whether LTO is enabled or not.
>
> After kernel is build with clang, the following command
> can be used to build selftests with clang:
>   make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
>
> But currently, using the above command, some compilations
> still use gcc and there are also compilation errors and warnings.
> This patch set intends to fix these issues.
> Patch #1 and #2 fixed the issue so clang/clang++ is
> used instead of gcc/g++. Patch #3 fixed a compilation
> failure. Patch #4 and #5 fixed various compiler warnings.
>

Might be good to add some hints like when the build stops or errors:

Like in my case when I had no CONFIG_DEBUG_INFO_BTF set.

Of course in combination with Clang-LTO a pointer "you need pahole
version 1.21." and CONFIG_DEBUG_INFO_BTF=y.

Finally, a hint for missing xxx-dev(el) packages (see Nick's report).

The tools directory has its own build rules thus I cannot say how to
check for specific Kconfigs.
I have not looked.

NOTE: I have not checked without setting CONFIG_DEBUG_INFO.

- Sedat -

> Changelog:
>   v1 -> v2:
>     . add -Wno-unused-command-line-argument and -Wno-format-security
>       for clang only as (1). gcc does not exhibit those
>       warnings, and (2). -Wno-unused-command-line-argument is
>       only supported by clang. (Sedat)
>
> Yonghong Song (5):
>   selftests: set CC to clang in lib.mk if LLVM is set
>   tools: allow proper CC/CXX/... override with LLVM=1 in
>     Makefile.include
>   selftests/bpf: fix test_cpp compilation failure with clang
>   selftests/bpf: silence clang compilation warnings
>   bpftool: fix a clang compilation warning
>
>  tools/bpf/bpftool/net.c              |  2 +-
>  tools/scripts/Makefile.include       | 12 ++++++++++--
>  tools/testing/selftests/bpf/Makefile |  7 ++++++-
>  tools/testing/selftests/lib.mk       |  4 ++++
>  4 files changed, 21 insertions(+), 4 deletions(-)
>
> --
> 2.30.2
>

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Yonghong Song]

On 4/12/21 6:44 PM, Sedat Dilek wrote:
> On Mon, Apr 12, 2021 at 4:29 PM Yonghong Song <yhs@fb.com> wrote:
>>
>> To build kernel with clang, people typically use
>>    make -j60 LLVM=1 LLVM_IAS=1
>> LLVM_IAS=1 is not required for non-LTO build but
>> is required for LTO build. In my environment,
>> I am always having LLVM_IAS=1 regardless of
>> whether LTO is enabled or not.
>>
>> After kernel is build with clang, the following command
>> can be used to build selftests with clang:
>>    make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
>>
>> But currently, using the above command, some compilations
>> still use gcc and there are also compilation errors and warnings.
>> This patch set intends to fix these issues.
>> Patch #1 and #2 fixed the issue so clang/clang++ is
>> used instead of gcc/g++. Patch #3 fixed a compilation
>> failure. Patch #4 and #5 fixed various compiler warnings.
>>
> 
> Might be good to add some hints like when the build stops or errors:
> 
> Like in my case when I had no CONFIG_DEBUG_INFO_BTF set.
> 
> Of course in combination with Clang-LTO a pointer "you need pahole
> version 1.21." and CONFIG_DEBUG_INFO_BTF=y.
> 
> Finally, a hint for missing xxx-dev(el) packages (see Nick's report).
> 
> The tools directory has its own build rules thus I cannot say how to
> check for specific Kconfigs.
> I have not looked.

I will add some more information about how I have tested in the
cover letter.

> 
> NOTE: I have not checked without setting CONFIG_DEBUG_INFO.
> 
> - Sedat -
> 
>> Changelog:
>>    v1 -> v2:
>>      . add -Wno-unused-command-line-argument and -Wno-format-security
>>        for clang only as (1). gcc does not exhibit those
>>        warnings, and (2). -Wno-unused-command-line-argument is
>>        only supported by clang. (Sedat)
>>
>> Yonghong Song (5):
>>    selftests: set CC to clang in lib.mk if LLVM is set
>>    tools: allow proper CC/CXX/... override with LLVM=1 in
>>      Makefile.include
>>    selftests/bpf: fix test_cpp compilation failure with clang
>>    selftests/bpf: silence clang compilation warnings
>>    bpftool: fix a clang compilation warning
>>
>>   tools/bpf/bpftool/net.c              |  2 +-
>>   tools/scripts/Makefile.include       | 12 ++++++++++--
>>   tools/testing/selftests/bpf/Makefile |  7 ++++++-
>>   tools/testing/selftests/lib.mk       |  4 ++++
>>   4 files changed, 21 insertions(+), 4 deletions(-)
>>
>> --
>> 2.30.2
>>

Re: [PATCH bpf-next v2 0/5] bpf: tools: support build selftests/bpf with clang

[Sedat Dilek]

On Mon, Apr 12, 2021 at 4:29 PM Yonghong Song <yhs@fb.com> wrote:
>
> To build kernel with clang, people typically use
>   make -j60 LLVM=1 LLVM_IAS=1

Just as a side note:

When building with a high parallel-make-job like above "-j60" I saw
that "test_core_extern.skel.h" was not generated which is a pre-req
header file for building stuff like test_cpp.cpp.

I have in my build-script:

MAXCPUS="$(($(getconf _NPROCESSORS_ONLN)))"
MAKE_JOBS="${MAXCPUS}"

Thanks.

- Sedat -


> LLVM_IAS=1 is not required for non-LTO build but
> is required for LTO build. In my environment,
> I am always having LLVM_IAS=1 regardless of
> whether LTO is enabled or not.
>
> After kernel is build with clang, the following command
> can be used to build selftests with clang:
>   make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
>
> But currently, using the above command, some compilations
> still use gcc and there are also compilation errors and warnings.
> This patch set intends to fix these issues.
> Patch #1 and #2 fixed the issue so clang/clang++ is
> used instead of gcc/g++. Patch #3 fixed a compilation
> failure. Patch #4 and #5 fixed various compiler warnings.
>
> Changelog:
>   v1 -> v2:
>     . add -Wno-unused-command-line-argument and -Wno-format-security
>       for clang only as (1). gcc does not exhibit those
>       warnings, and (2). -Wno-unused-command-line-argument is
>       only supported by clang. (Sedat)
>
> Yonghong Song (5):
>   selftests: set CC to clang in lib.mk if LLVM is set
>   tools: allow proper CC/CXX/... override with LLVM=1 in
>     Makefile.include
>   selftests/bpf: fix test_cpp compilation failure with clang
>   selftests/bpf: silence clang compilation warnings
>   bpftool: fix a clang compilation warning
>
>  tools/bpf/bpftool/net.c              |  2 +-
>  tools/scripts/Makefile.include       | 12 ++++++++++--
>  tools/testing/selftests/bpf/Makefile |  7 ++++++-
>  tools/testing/selftests/lib.mk       |  4 ++++
>  4 files changed, 21 insertions(+), 4 deletions(-)
>
> --
> 2.30.2
>