Re: [PATCH bpf-next 3/5] selftests/bpf: fix test_cpp compilation failure with clang

[Yonghong Song <yhs@fb.com>]

On 4/11/21 10:31 AM, Sedat Dilek wrote:
> On Sun, Apr 11, 2021 at 7:20 PM Yonghong Song <yhs@fb.com> wrote:
>>
>>
>>
>> On 4/11/21 3:47 AM, Sedat Dilek wrote:
>>> On Sat, Apr 10, 2021 at 6:49 PM Yonghong Song <yhs@fb.com> wrote:
>>>>
>>>> With clang compiler:
>>>>     make -j60 LLVM=1 LLVM_IAS=1  <=== compile kernel
>>>>     make -j60 -C tools/testing/selftests/bpf LLVM=1 LLVM_IAS=1
>>>> the test_cpp build failed due to the failure:
>>>>     warning: treating 'c-header' input as 'c++-header' when in C++ mode, this behavior is deprecated [-Wdeprecated]
>>>>     clang-13: warning: cannot specify -o when generating multiple output files
>>>>
>>>> test_cpp compilation flag looks like:
>>>>     clang++ -g -Og -rdynamic -Wall -I<...> ... \
>>>>     -Dbpf_prog_load=bpf_prog_test_load -Dbpf_load_program=bpf_test_load_program \
>>>>     test_cpp.cpp <...>/test_core_extern.skel.h <...>/libbpf.a <...>/test_stub.o \
>>>>     -lcap -lelf -lz -lrt -lpthread -o <...>/test_cpp
>>>>
>>>> The clang++ compiler complains the header file in the command line.
>>>> Let us remove the header file from the command line which is not intended
>>>> any way, and this fixed the problem.
>>>>
>>>> Signed-off-by: Yonghong Song <yhs@fb.com>
>>>> ---
>>>>    tools/testing/selftests/bpf/Makefile | 2 +-
>>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
>>>> index 6448c626498f..bbd61cc3889b 100644
>>>> --- a/tools/testing/selftests/bpf/Makefile
>>>> +++ b/tools/testing/selftests/bpf/Makefile
>>>> @@ -481,7 +481,7 @@ $(OUTPUT)/test_verifier: test_verifier.c verifier/tests.h $(BPFOBJ) | $(OUTPUT)
>>>>    # Make sure we are able to include and link libbpf against c++.
>>>>    $(OUTPUT)/test_cpp: test_cpp.cpp $(OUTPUT)/test_core_extern.skel.h $(BPFOBJ)
>>>>           $(call msg,CXX,,$@)
>>>> -       $(Q)$(CXX) $(CFLAGS) $^ $(LDLIBS) -o $@
>>>> +       $(Q)$(CXX) $(CFLAGS) test_cpp.cpp $(BPFOBJ) $(LDLIBS) -o $@
>>>>
>>>>    # Benchmark runner
>>>>    $(OUTPUT)/bench_%.o: benchs/bench_%.c bench.h
>>>> --
>>>> 2.30.2
>>>>
>>>
>>> NOTE: bpf-next might be different from my build-environment.
>>>
>>> Yonghong, can you please re-test by adding explicitly CXX=g++ to your make line?
>>>
>>> Here I have:
>>>
>>> $ grep test_cpp make-log_tools-testing-selftests-bpf_clang_clang++.txt
>>> 1907:clang++ -g -rdynamic -Wall -O2 -DHAVE_GENHDR
>>> -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf
>>> -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/include
>>> -I/home/dileks/src/linux-kernel/git/include/generated
>>> -I/home/dileks/src/linux-kernel/git/tools/lib
>>> -I/home/dileks/src/linux-kernel/git/tools/include
>>> -I/home/dileks/src/linux-kernel/git/tools/include/uapi
>>> -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf
>>> -Wno-unused-command-line-argument -Wno-format-security
>>> -Dbpf_prog_load=bpf_prog_test_load
>>> -Dbpf_load_program=bpf_test_load_program test_cpp.cpp
>>> /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.a
>>> -lcap -lelf -lz -lrt -lpthread -o
>>> /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/test_cpp
>>>
>>> This clang++ line does not include <...>/test_core_extern.skel.h ^^^
>>>
>>> $ grep test_core_extern.skel.h
>>> make-log_tools-testing-selftests-bpf_clang_clang++.txt
>>> 704:/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/sbin/bpftool
>>> gen skeleton /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/test_co
>>> re_extern.o > /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/test_core_extern.skel.h
>>> 1592:/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/sbin/bpftool
>>> gen skeleton /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/no_alu
>>> 32/test_core_extern.o >
>>> /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/no_alu32/test_core_extern.skel.h
>>>
>>> Checking test_cpp:
>>>
>>> $ /opt/llvm-toolchain/bin/llvm-objdump-12 -Dr test_cpp | grep extern
>>> 0000000000417e50 <cmp_externs>:
>>>    417e54: 75 22                         jne     0x417e78 <cmp_externs+0x28>
>>>    417e59: 75 10                         jne     0x417e6b <cmp_externs+0x1b>
>>>    417e61: 75 21                         jne     0x417e84 <cmp_externs+0x34>
>>>    417e69: 75 1e                         jne     0x417e89 <cmp_externs+0x39>
>>>    417e87: eb f2                         jmp     0x417e7b <cmp_externs+0x2b>
>>>    417e8c: eb ed                         jmp     0x417e7b <cmp_externs+0x2b>
>>>
>>> $ /opt/llvm-toolchain/bin/llvm-objdump-12 -Dr test_cpp | grep core_extern
>>> [ EMPTY ]
>>>
>>> When compiled with g++ in an earlier setup this contained "core_extern".
>>>
>>> With this version of your patchser it breaks *here* when using CXX=g++
>>> (and uses /usr/bin/ld as linker):
>>>
>>> g++ -g -rdynamic -Wall -O2 -DHAVE_GENHDR
>>> -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf
>>> -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/b
>>> pf/tools/include -I/home/dileks/src/linux-kernel/git/include/generated
>>> -I/home/dileks/src/linux-kernel/git/tools/lib
>>> -I/home/dileks/src/linux-kernel/git/tools/include
>>> -I/home/dileks/src/linux-kernel/git/tools/include/uapi
>>> -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf
>>> -Wno-unused-command-line-argument -Wno-format-security
>>> -Dbpf_prog_load=bpf_prog_test_load
>>> -Dbpf_load_program=bpf_test_load_program test_cpp.cpp
>>> /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.a
>>> -lcap -lelf -lz -lrt -lpthread -o
>>> /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/test_cpp
>>>
>>> /usr/bin/ld: /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.a(libbpf-in.o):
>>> relocation R_X86_64_32 against `.rodata.str1.1' ca
>>> n not be used when making a PIE object; recompile with -fPIE
>>> collect2: error: ld returned 1 exit status
>>> make: *** [Makefile:457:
>>> /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/test_cpp]
>>> Error 1
>>
>> I cannot reproduce the issue with g++ with bpf-next, my command line is
>>
>> g++ -g -Og -rdynamic -Wall
>> -I/home/yhs/work/bpf-next/tools/testing/selftests/bpf
>> -I/home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/include
>> -I/home/yhs/work/bpf-next/include/generated
>> -I/home/yhs/work/bpf-next/tools/lib
>> -I/home/yhs/work/bpf-next/tools/include
>> -I/home/yhs/work/bpf-next/tools/include/uapi
>> -I/home/yhs/work/bpf-next/tools/testing/selftests/bpf
>> -Wno-unused-command-line-argument -Wno-format-security
>> -Dbpf_prog_load=bpf_prog_test_load
>> -Dbpf_load_program=bpf_test_load_program test_cpp.cpp
>> /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.a
>> -lcap -lelf -lz -lrt -lpthread -o
>> /home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_cpp
>>
>> I modified to
>> g++ -g -rdynamic -Wall -O2 -DHAVE_GENHDR
>> -I/home/yhs/work/bpf-next/tools/testing/selftests/bpf ...
>> and cannot reproduce the issue.
>> The macro HAVE_GENHDR is only effect for test_verifier.
>>
>>
>> Could you try to run the above g++ command by adding
>> test_core_extern.skel.h back, something like
>>
>>   > g++ -g -rdynamic -Wall -O2 -DHAVE_GENHDR
>>   > -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf
>>   > -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/b
>>   > pf/tools/include -I/home/dileks/src/linux-kernel/git/include/generated
>>   > -I/home/dileks/src/linux-kernel/git/tools/lib
>>   > -I/home/dileks/src/linux-kernel/git/tools/include
>>   > -I/home/dileks/src/linux-kernel/git/tools/include/uapi
>>   > -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf
>>   > -Wno-unused-command-line-argument -Wno-format-security
>>   > -Dbpf_prog_load=bpf_prog_test_load
>>   > -Dbpf_load_program=bpf_test_load_program test_cpp.cpp
>>   > test_core_extern.skel.h
>>   >
>> /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.a
>>   > -lcap -lelf -lz -lrt -lpthread -o
>>   > /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/test_cpp
>>
>> The issue could be somewhere else?
>>
> 
> I have seen all that *skel* was reworked in bpf-next, so this is an issue here.
> 
> Adding test_core_extern.skel.h:
> 
> $ cd /tools/testing/selftests/bpf/
> 
> $ file test_core_extern.skel.h
> test_core_extern.skel.h: C source, ASCII text
> 
> $ g++ -g -rdynamic -Wall -O2 -DHAVE_GENHDR
> -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf
> -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/include
> -I/home/dileks/src/linux-kernel/git/include/generated
> -I/home/dileks/src/linux-kernel/git/tools/lib
> -I/home/dileks/src/linux-kernel/git/tools/include
> -I/home/dileks/src/linux-kernel/git/tools/include/uapi
> -I/home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf
> -Wno-unused-command-line-argument -Wno-format-security
> -Dbpf_prog_load=bpf_prog_test_load
> -Dbpf_load_program=bpf_test_load_program test_cpp.cpp
> test_core_extern.skel.h
> /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.a
> -lcap -lelf -lz -lrt -lpthread -o
> /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/test_cpp
> /usr/bin/ld: /home/dileks/src/linux-kernel/git/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.a(libbpf-in.o):
> relocation R_X86_64_32 against `.rodata.str1.1' ca
> n not be used when making a PIE object; recompile with -fPIE
> collect2: error: ld returned 1 exit status

So the issue is not related to this patch since adding 
test_core_extern.skel.h the failure is still there.

So looks like this is g++ compilation issue. So you should be
able to reproduce the issue without this patch set, right?

My gcc compiled libbpf.a also has a bunch of R_X86_64_32 relations
against the .rodata.str1.1 section:

$ readelf -r libbpf.a | less
File: libbpf.a(libbpf-in.o)

Relocation section '.rela.text' at offset 0x11a4b8 contains 3152 entries:
   Offset          Info           Type           Sym. Value    Sym. Name 
+ Addend
000000000136  00020000000b R_X86_64_32S      0000000000000000 .rodata + 0
00000000013b  00030000000a R_X86_64_32       0000000000000000 
.rodata.str1.1 + 6e
000000000141  00030000000a R_X86_64_32       0000000000000000 
.rodata.str1.1 + c
000000000147  00030000000a R_X86_64_32       0000000000000000 
.rodata.str1.1 + 10
00000000014d  00030000000a R_X86_64_32       0000000000000000 
.rodata.str1.1 + 16
000000000153  00030000000a R_X86_64_32       0000000000000000 
.rodata.str1.1 + 1d
000000000159  00030000000a R_X86_64_32       0000000000000000 
.rodata.str1.1 + 23
00000000015f  00030000000a R_X86_64_32       0000000000000000 
.rodata.str1.1 + 28
...

and I did not see any issues.
Adding '-###' to the compilation flags you can find the flags for
each subcommand, for linker, I have

  /usr/libexec/gcc/x86_64-redhat-linux/8/collect2 -plugin 
/usr/libexec/gcc/x86_64-redhat-linux/8/liblto_plugin.so 
"-plugin-opt=/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper" 
"-plugin-opt=-fresolution=/tmp/ccjktcVg.res" 
"-plugin-opt=-pass-through=-lgcc_s" "-plugin-opt=-pass-through=-lgcc" 
"-plugin-opt=-pass-through=-lc" "-plugin-opt=-pass-through=-lgcc_s" 
"-plugin-opt=-pass-through=-lgcc" --build-id --no-add-needed 
--eh-frame-hdr "--hash-style=gnu" -m elf_x86_64 -export-dynamic 
-dynamic-linker /lib64/ld-linux-x86-64.so.2 -o 
/home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_cpp 
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crt1.o 
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crti.o 
/usr/lib/gcc/x86_64-redhat-linux/8/crtbegin.o 
-L/usr/lib/gcc/x86_64-redhat-linux/8 
-L/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64 -L/lib/../lib64 
-L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/8/../../.. 
/tmp/cc0xqkPi.o 
/home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.a 
-lcap -lelf -lz -lrt -lpthread "-lstdc++" -lm -lgcc_s -lgcc -lc -lgcc_s 
-lgcc /usr/lib/gcc/x86_64-redhat-linux/8/crtend.o 
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crtn.o
COLLECT_GCC_OPTIONS='-g' '-rdynamic' '-Wall' '-O2' '-D' 'HAVE_GENHDR' 
'-I' '/home/yhs/work/bpf-next/tools/testing/selftests/bpf' '-I' 
'/home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/include' '-I' 
'/home/yhs/work/bpf-next/include/generated' '-I' 
'/home/yhs/work/bpf-next/tools/lib' '-I' 
'/home/yhs/work/bpf-next/tools/include' '-I' 
'/home/yhs/work/bpf-next/tools/include/uapi' '-I' 
'/home/yhs/work/bpf-next/tools/testing/selftests/bpf' 
'-Wno-uuunused-command-line-argument' '-Wno-format-security' '-D' 
'bpf_prog_load=bpf_prog_test_load' '-D' 
'bpf_load_program=bpf_test_load_program' '-o' 
'/home/yhs/work/bpf-next/tools/testing/selftests/bpf/test_cpp' 
'-shared-libgcc' '-mtune=generic' '-march=x86-64'

Not sure whether you will see some different COLLECT_GCC_OPTIONS flags 
or not.

There is not much I can do here since I cannot reproduce the issue.
Maybe you can help to narrow down the reason of the failure a little
bit more.

> 
> Write that test_cpp.cpp in C :-)?

test_cpp.cpp is intentionally to test libbpf used by a c++ application.

> 
> BTW, did you check (llvm-)objdump output?
> 
> $ /opt/llvm-toolchain/bin/llvm-objdump-12 -Dr test_cpp | grep core_extern

This is what I got with g++ compiled test_cpp:

$ llvm-objdump -Dr test_cpp | grep core_extern
   406a80: e8 5b 01 00 00                callq   0x406be0 
<_ZL25test_core_extern__destroyP16test_core_extern>
   406ab9: e8 22 01 00 00                callq   0x406be0 
<_ZL25test_core_extern__destroyP16test_core_extern>
0000000000406be0 <_ZL25test_core_extern__destroyP16test_core_extern>:
   406be3: 74 1a                         je      0x406bff 
<_ZL25test_core_extern__destroyP16test_core_extern+0x1f>
   406bef: 74 05                         je      0x406bf6 
<_ZL25test_core_extern__destroyP16test_core_extern+0x16>

> 
> - Sedat -
> 
>>>
>>> $ grep test_cpp ../make-log_tools-testing-selftests-bpf_clang_g++.txt
>>> | grep test_core_extern.skel.h
>>> [ EMPTY ]
>>>
>>> As said I do NOT use bpf-next.
>>>
>>> - Sedat -
>>>
>>>
>>>
>>> - Sedat -
>>>