From mboxrd@z Thu Jan 1 00:00:00 1970 From: Arnaldo Carvalho de Melo Subject: Re: Help with the BPF verifier Date: Sat, 3 Nov 2018 08:29:34 -0300 Message-ID: <20181103112934.GH20495@kernel.org> References: <20181101185217.GA20495@kernel.org> <20181102150239.GG20495@kernel.org> <51197ebf-a3d8-a3ae-0389-d7e4dae3e833@solarflare.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Yonghong Song , Daniel Borkmann , Jiri Olsa , Martin Lau , Alexei Starovoitov , Linux Networking Development Mailing List To: Edward Cree Return-path: Received: from mail.kernel.org ([198.145.29.99]:45968 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727537AbeKCUkj (ORCPT ); Sat, 3 Nov 2018 16:40:39 -0400 Content-Disposition: inline In-Reply-To: <51197ebf-a3d8-a3ae-0389-d7e4dae3e833@solarflare.com> Sender: netdev-owner@vger.kernel.org List-ID: Em Fri, Nov 02, 2018 at 03:42:49PM +0000, Edward Cree escreveu: > On 02/11/18 15:02, Arnaldo Carvalho de Melo wrote: > > Yeah, didn't work as well:=20 >=20 > > And the -vv in 'perf trace' didn't seem to map to further details in the > > output of the verifier debug: > Yeah for log_level 2 you probably need to make source-level changes to ei= ther > =C2=A0perf or libbpf (I think the latter).=C2=A0 It's annoying that essen= tially no tools > =C2=A0plumb through an option for that, someone should fix them ;-) >=20 > > libbpf: -- BEGIN DUMP LOG --- > > libbpf:=20 > > 0: (bf) r6 =3D r1 > > 1: (bf) r1 =3D r10 > > 2: (07) r1 +=3D -328 > > 3: (b7) r7 =3D 64 > > 4: (b7) r2 =3D 64 > > 5: (bf) r3 =3D r6 > > 6: (85) call bpf_probe_read#4 > > 7: (79) r1 =3D *(u64 *)(r10 -320) > > 8: (15) if r1 =3D=3D 0x101 goto pc+4 > > R0=3Dinv(id=3D0) R1=3Dinv(id=3D0) R6=3Dctx(id=3D0,off=3D0,imm=3D0) R7= =3Dinv64 R10=3Dfp0,call_-1 > > 9: (55) if r1 !=3D 0x2 goto pc+22 > > R0=3Dinv(id=3D0) R1=3Dinv2 R6=3Dctx(id=3D0,off=3D0,imm=3D0) R7=3Dinv64= R10=3Dfp0,call_-1 > > 10: (bf) r1 =3D r6 > > 11: (07) r1 +=3D 16 > > 12: (05) goto pc+2 > > 15: (79) r3 =3D *(u64 *)(r1 +0) > > dereference of modified ctx ptr R1 off=3D16 disallowed > Aha, we at least got a different error message this time. > And indeed llvm has done that optimisation, rather than the more obvious > 11: r3 =3D *(u64 *)(r1 +16) > =C2=A0because it wants to have lots of reads share a single insn.=C2=A0 Y= ou may be able > =C2=A0to defeat that optimisation by adding compiler barriers, idk.=C2=A0= Maybe someone > =C2=A0with llvm knowledge can figure out how to stop it (ideally, llvm wo= uld know > =C2=A0when it's generating for bpf backend and not do that).=C2=A0 -O0?= =C2=A0 =C2=AF\_(=E3=83=84)_/=C2=AF set env: NR_CPUS=3D4 set env: LINUX_VERSION_CODE=3D0x41300 set env: CLANG_EXEC=3D/usr/local/bin/clang unset env: CLANG_OPTIONS set env: KERNEL_INC_OPTIONS=3D -nostdinc -isystem /usr/lib/gcc/x86_64-redha= t-linux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/incl= ude/generated -I/home/acme/git/linux/include -I./include -I/home/acme/git/= linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acm= e/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git= /linux/include/linux/kconfig.h=20 set env: PERF_BPF_INC_OPTIONS=3D-I/home/acme/lib/perf/include/bpf set env: WORKING_DIR=3D/lib/modules/4.19.0-rc8-00014-gc0cff31be705/build set env: CLANG_SOURCE=3D/home/acme/git/perf/tools/perf/examples/bpf/augment= ed_raw_syscalls.c llvm compiling command template: $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=3D$= NR_CPUS -DLINUX_VERSION_CODE=3D$LINUX_VERSION_CODE $CLANG_OPTIONS $PERF_BPF= _INC_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign -worki= ng-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf $CLANG_EMIT_LLVM -= O2 -o - $LLVM_OPTIONS_PIPE llvm compiling command : /usr/local/bin/clang -D__KERNEL__ -D__NR_CPUS__=3D= 4 -DLINUX_VERSION_CODE=3D0x41300 -I/home/acme/lib/perf/include/bpf -nostd= inc -isystem /usr/lib/gcc/x86_64-redhat-linux/7/include -I/home/acme/git/li= nux/arch/x86/include -I./arch/x86/include/generated -I/home/acme/git/linux= /include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/= x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/= generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h -Wno-= unused-value -Wno-pointer-sign -working-directory /lib/modules/4.19.0-rc8-0= 0014-gc0cff31be705/build -c /home/acme/git/perf/tools/perf/examples/bpf/aug= mented_raw_syscalls.c -target bpf -O2 -o - So it is using -O2, lets try with -O0... So I added this to my ~/.perfconfig, i.e. the default clang command line template replacing -O2 with -O0. [root@seventh perf]# cat ~/.perfconfig=20 [llvm] clang-bpf-cmd-template =3D "$CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=3D$NR_C= PUS -DLINUX_VERSION_CODE=3D$LINUX_VERSION_CODE $CLANG_OPTIONS $PERF_BPF_INC= _OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign -working-d= irectory $WORKING_DIR -c \"$CLANG_SOURCE\" -target bpf $CLANG_EMIT_LLVM -O0= -o - $LLVM_OPTIONS_PIPE" # dump-obj =3D true [root@seventh perf]#=20 And got an explosion: # trace -vv -e tools/perf/examples/bpf/augmented_raw_syscalls.c sleep 1 bpf: builtin compilation failed: -95, try external compiler Kernel build dir is set to /lib/modules/4.19.0-rc8-00014-gc0cff31be705/build set env: KBUILD_DIR=3D/lib/modules/4.19.0-rc8-00014-gc0cff31be705/build unset env: KBUILD_OPTS include option is set to -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-lin= ux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/g= enerated -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux= /arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git= /linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linu= x/include/linux/kconfig.h=20 set env: NR_CPUS=3D4 set env: LINUX_VERSION_CODE=3D0x41300 set env: CLANG_EXEC=3D/usr/local/bin/clang unset env: CLANG_OPTIONS set env: KERNEL_INC_OPTIONS=3D -nostdinc -isystem /usr/lib/gcc/x86_64-redha= t-linux/7/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/incl= ude/generated -I/home/acme/git/linux/include -I./include -I/home/acme/git/= linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acm= e/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git= /linux/include/linux/kconfig.h=20 set env: PERF_BPF_INC_OPTIONS=3D-I/home/acme/lib/perf/include/bpf set env: WORKING_DIR=3D/lib/modules/4.19.0-rc8-00014-gc0cff31be705/build set env: CLANG_SOURCE=3D/home/acme/git/perf/tools/perf/examples/bpf/augment= ed_raw_syscalls.c llvm compiling command template: $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=3D$= NR_CPUS -DLINUX_VERSION_CODE=3D$LINUX_VERSION_CODE $CLANG_OPTIONS $PERF_BPF= _INC_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign -worki= ng-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf $CLANG_EMIT_LLVM -= O0 -o - $LLVM_OPTIONS_PIPE llvm compiling command : /usr/local/bin/clang -D__KERNEL__ -D__NR_CPUS__=3D= 4 -DLINUX_VERSION_CODE=3D0x41300 -I/home/acme/lib/perf/include/bpf -nostd= inc -isystem /usr/lib/gcc/x86_64-redhat-linux/7/include -I/home/acme/git/li= nux/arch/x86/include -I./arch/x86/include/generated -I/home/acme/git/linux= /include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/= x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/= generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h -Wno-= unused-value -Wno-pointer-sign -working-directory /lib/modules/4.19.0-rc8-0= 0014-gc0cff31be705/build -c /home/acme/git/perf/tools/perf/examples/bpf/aug= mented_raw_syscalls.c -target bpf -O0 -o -=20 clang-7: /home/acme/git/llvm/lib/Target/BPF/MCTargetDesc/BPFAsmBackend.cpp:= 74: virtual void {anonymous}::BPFAsmBackend::applyFixup(const llvm::MCAssem= bler&, const llvm::MCFixup&, const llvm::MCValue&, llvm::MutableArrayRef, uint64_t, bool, const llvm::MCSubtargetInfo*) const: Assertion `Value = =3D=3D 0' failed. Stack dump: 0. Program arguments: /usr/local/bin/clang-7 -cc1 -triple bpf -emit-obj -mr= elax-all -disable-free -main-file-name augmented_raw_syscalls.c -mrelocatio= n-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-ve= rbose -mconstructor-aliases -dwarf-column-info -debugger-tuning=3Dgdb -cove= rage-notes-file /home/acme/git/perf/-.gcno -nostdsysteminc -nobuiltininc -r= esource-dir /usr/local/lib/clang/8.0.0 -working-directory /lib/modules/4.19= =2E0-rc8-00014-gc0cff31be705/build -isystem /usr/lib/gcc/x86_64-redhat-linu= x/7/include -include /home/acme/git/linux/include/linux/kconfig.h -D __KERN= EL__ -D __NR_CPUS__=3D4 -D LINUX_VERSION_CODE=3D0x41300 -I /home/acme/lib/p= erf/include/bpf -I /home/acme/git/linux/arch/x86/include -I ./arch/x86/incl= ude/generated -I /home/acme/git/linux/include -I ./include -I /home/acme/gi= t/linux/arch/x86/include/uapi -I ./arch/x86/include/generated/uapi -I /home= /acme/git/linux/include/uapi -I ./include/generated/uapi -O0 -Wno-unused-va= lue -Wno-pointer-sign -fdebug-compilation-dir /home/acme/git/perf -ferror-l= imit 19 -fmessage-length 203 -fobjc-runtime=3Dgcc -fdiagnostics-show-option= -fcolor-diagnostics -o - -x c /home/acme/git/perf/tools/perf/examples/bpf/= augmented_raw_syscalls.c -faddrsig=20 1. parser at end of file 2. Code generation #0 0x000000000408ff98 llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/usr/= local/bin/clang-7+0x408ff98) #1 0x000000000409002b (/usr/local/bin/clang-7+0x409002b) #2 0x000000000408e059 llvm::sys::RunSignalHandlers() (/usr/local/bin/clang-= 7+0x408e059) #3 0x000000000408fa5b (/usr/local/bin/clang-7+0x408fa5b) #4 0x00007fcd33ab81c0 __restore_rt (/lib64/libpthread.so.0+0x121c0) #5 0x00007fcd32817750 __GI_raise (/lib64/libc.so.6+0x34750) #6 0x00007fcd32818d31 __GI_abort (/lib64/libc.so.6+0x35d31) #7 0x00007fcd3281005a __assert_fail_base (/lib64/libc.so.6+0x2d05a) #8 0x00007fcd328100d2 (/lib64/libc.so.6+0x2d0d2) #9 0x0000000002602d13 (/usr/local/bin/clang-7+0x2602d13) #10 0x0000000003bf6da7 llvm::MCAssembler::layout(llvm::MCAsmLayout&) (/usr/= local/bin/clang-7+0x3bf6da7) #11 0x0000000003bf6e2c llvm::MCAssembler::Finish() (/usr/local/bin/clang-7+= 0x3bf6e2c) #12 0x0000000003c524e8 llvm::MCObjectStreamer::FinishImpl() (/usr/local/bin= /clang-7+0x3c524e8) #13 0x0000000003c2db4b llvm::MCELFStreamer::FinishImpl() (/usr/local/bin/cl= ang-7+0x3c2db4b) #14 0x0000000003c5ba43 llvm::MCStreamer::Finish() (/usr/local/bin/clang-7+0= x3c5ba43) #15 0x0000000004c281b9 llvm::AsmPrinter::doFinalization(llvm::Module&) (/us= r/local/bin/clang-7+0x4c281b9) #16 0x00000000038d637b llvm::FPPassManager::doFinalization(llvm::Module&) (= /usr/local/bin/clang-7+0x38d637b) #17 0x00000000038d683d (/usr/local/bin/clang-7+0x38d683d) #18 0x00000000038d6d6b llvm::legacy::PassManagerImpl::run(llvm::Module&) (/= usr/local/bin/clang-7+0x38d6d6b) #19 0x00000000038d6f9b llvm::legacy::PassManager::run(llvm::Module&) (/usr/= local/bin/clang-7+0x38d6f9b) #20 0x000000000436b091 (/usr/local/bin/clang-7+0x436b091) #21 0x000000000436df07 clang::EmitBackendOutput(clang::DiagnosticsEngine&, = clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::Tar= getOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm= ::Module*, clang::BackendAction, std::unique_ptr >) (/usr/local/bin/clang-7+0x43= 6df07) #22 0x0000000004f6ee11 (/usr/local/bin/clang-7+0x4f6ee11) #23 0x00000000062af3a3 clang::ParseAST(clang::Sema&, bool, bool) (/usr/loca= l/bin/clang-7+0x62af3a3) #24 0x0000000004a73873 clang::ASTFrontendAction::ExecuteAction() (/usr/loca= l/bin/clang-7+0x4a73873) #25 0x0000000004f6ced8 clang::CodeGenAction::ExecuteAction() (/usr/local/bi= n/clang-7+0x4f6ced8) #26 0x0000000004a7329d clang::FrontendAction::Execute() (/usr/local/bin/cla= ng-7+0x4a7329d) #27 0x0000000004a04ccf clang::CompilerInstance::ExecuteAction(clang::Fronte= ndAction&) (/usr/local/bin/clang-7+0x4a04ccf) #28 0x0000000004bc512e clang::ExecuteCompilerInvocation(clang::CompilerInst= ance*) (/usr/local/bin/clang-7+0x4bc512e) #29 0x0000000001d88a6c cc1_main(llvm::ArrayRef, char const*, v= oid*) (/usr/local/bin/clang-7+0x1d88a6c) #30 0x0000000001d7e31e (/usr/local/bin/clang-7+0x1d7e31e) #31 0x0000000001d7ef94 main (/usr/local/bin/clang-7+0x1d7ef94) #32 0x00007fcd32803fea __libc_start_main (/lib64/libc.so.6+0x20fea) #33 0x0000000001d7beea _start (/usr/local/bin/clang-7+0x1d7beea) clang-7: error: unable to execute command: Aborted (core dumped) clang-7: error: clang frontend command failed due to signal (use -v to see = invocation) clang version 8.0.0 (http://llvm.org/git/clang.git 8587270a739ee30c926a76d5= 657e65e85b560f6e) (http://llvm.org/git/llvm.git 0566eefef9c3777bd780ec4cbb9= efa764633b76c) Target: bpf Thread model: posix InstalledDir: /usr/local/bin clang-7: note: diagnostic msg: PLEASE submit a bug report to https://bugs.l= lvm.org/ and include the crash backtrace, preprocessed source, and associat= ed run script. clang-7: note: diagnostic msg:=20 ******************** PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT: Preprocessed source(s) and associated run script(s) are located at: clang-7: note: diagnostic msg: /tmp/augmented_raw_syscalls-7444d9.c clang-7: note: diagnostic msg: /tmp/augmented_raw_syscalls-7444d9.sh clang-7: note: diagnostic msg:=20 ******************** ERROR: unable to compile tools/perf/examples/bpf/augmented_raw_syscalls.c Hint: Check error message shown above. Hint: You can also pre-compile it into .o using: clang -target bpf -O2 -c tools/perf/examples/bpf/augmented_raw_sysca= lls.c with proper -I and -D options. event syntax error: 'tools/perf/examples/bpf/augmented_raw_syscalls.c' \___ Failed to load tools/perf/examples/bpf/augmented_= raw_syscalls.c from source: Error when compiling BPF scriptlet (add -v to see detail) Run 'perf list' for a list of valid events Usage: perf trace [] [] or: perf trace [] -- [] or: perf trace record [] [] or: perf trace record [] -- [] -e, --event event/syscall selector. use 'perf list' to list a= vailable events [root@seventh perf]#=20 Trying with -O1... > Alternatively, your prog looks short enough that maybe you could kick the= C > =C2=A0habit and write it directly in eBPF asm, that way no-one is optimis= ing things > =C2=A0behind your back.=C2=A0 (I realise this option won't appeal to ever= yone ;-) Nah, if that is what it takes... Would be better with simple looking C code tho :-) > The reason the verifier disallows this, iirc, is because it needs to be a= ble > =C2=A0to rewrite the offsets on ctx accesses (see convert_ctx_accesses())= in case > =C2=A0underlying kernel struct doesn't match the layout of the ctx ABI.= =C2=A0 To do this > =C2=A0it needs the ctx offset to live entirely in the insn doing the acce= ss, > =C2=A0otherwise different paths could lead to the same insn accessing dif= ferent ctx > =C2=A0offsets with different fixups needed =E2=80=94 can't be done. >=20 > -Ed