Re: libbpf error: unknown register name 'r0' in asm

[Andrii Nakryiko <andrii.nakryiko@gmail.com>]

On Fri, Oct 9, 2020 at 1:24 PM Yaniv Agman <yanivagman@gmail.com> wrote:
>
> ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-22:53 מאת ‪Andrii Nakryiko‬‏
> <‪andrii.nakryiko@gmail.com‬‏>:‬
> >
> > On Fri, Oct 9, 2020 at 12:32 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> > >
> > > ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-21:39 מאת ‪Andrii Nakryiko‬‏
> > > <‪andrii.nakryiko@gmail.com‬‏>:‬
> > > >
> > > > On Fri, Oct 9, 2020 at 11:33 AM Yaniv Agman <yanivagman@gmail.com> wrote:
> > > > >
> > > > > ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-21:21 מאת ‪Daniel Borkmann‬‏
> > > > > <‪daniel@iogearbox.net‬‏>:‬
> > > > > >
> > > > > > On 10/9/20 8:09 PM, Yaniv Agman wrote:
> > > > > > > ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-20:39 מאת ‪Daniel Borkmann‬‏
> > > > > > > <‪daniel@iogearbox.net‬‏>:‬
> > > > > > >>
> > > > > > >> On 10/9/20 6:56 PM, Yaniv Agman wrote:
> > > > > > >>> ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-19:27 מאת ‪Daniel Borkmann‬‏
> > > > > > >>> <‪daniel@iogearbox.net‬‏>:‬
> > > > > > >>>>
> > > > > > >>>> [ Cc +Yonghong ]
> > > > > > >>>>
> > > > > > >>>> On 10/9/20 6:05 PM, Yaniv Agman wrote:
> > > > > > >>>>> Pulling the latest changes of libbpf and compiling my application with it,
> > > > > > >>>>> I see the following error:
> > > > > > >>>>>
> > > > > > >>>>> ../libbpf/src//root/usr/include/bpf/bpf_helpers.h:99:10: error:
> > > > > > >>>>> unknown register name 'r0' in asm
> > > > > > >>>>>                         : "r0", "r1", "r2", "r3", "r4", "r5");
> > > > > > >>>>>
> > > > > > >>>>> The commit which introduced this change is:
> > > > > > >>>>> 80c7838600d39891f274e2f7508b95a75e4227c1
> > > > > > >>>>>
> > > > > > >>>>> I'm not sure if I'm doing something wrong (missing include?), or this
> > > > > > >>>>> is a genuine error
> > > > > > >>>>
> > > > > > >>>> Seems like your clang/llvm version might be too old.
> > > > > > >>>
> > > > > > >>> I'm using clang 10.0.1
> > > > > > >>
> > > > > > >> Ah, okay, I see. Would this diff do the trick for you?
> > > > > > >
> > > > > > > Yes! Now it compiles without any problems!
> > > > > >
> > > > > > Great, thx, I'll cook proper fix and check with clang6 as Yonghong mentioned.
> > > > > >
> > > > >
> > > > > Thanks!
> > > > > Does this happen because I'm first compiling using "emit-llvm" and
> > > > > then using llc?
> > > >
> > > > So this must be the reason, but I'll wait for Yonghong to confirm.
> > > >
> > > > > I wish I could use bpf target directly, but I'm then having problems
> > > > > with includes of asm code (like pt_regs and other stuff)
> > > >
> > > > Are you developing for a 32-bit platform? Or what exactly is the
> > > > problem? I've been trying to solve problems for 32-bit arches recently
> > > > by making libbpf smarter, that relies on CO-RE though. Is CO-RE an
> > > > option for you?
> > > >
> > >
> > > Examples for the errors I'm getting:
> > > /lib/modules/4.14.199-1-MANJARO/build/arch/x86/include/asm/atomic.h:177:9:
> > > error: invalid output constraint '+q' in asm
> > >         return xadd(&v->counter, i);
> > >                ^
> > > /lib/modules/4.14.199-1-MANJARO/build/arch/x86/include/asm/cmpxchg.h:234:25:
> > > note: expanded from macro 'xadd'
> > > #define xadd(ptr, inc)          __xadd((ptr), (inc), LOCK_PREFIX)
> > > ...
> > >
> > > From What I understood, this is a known issue for tracing programs
> > > (like the one I'm developing)
> >
> > We do have a bunch of selftests that use pt_regs and include, say,
> > linux/ptrace.h header. I wonder why we are not seeing these problems.
> > Selftests, btw, are also built with -emit-llvm and then piping output
> > to llc.
> >
> > So.. there must be something else going on. It's hard to guess like
> > this without seeing the code, but maybe -D__TARGET_ARCH_$(SRCARCH)
> > during compilation could help, just as an idea.
> >
>
> Adding -D__TARGET_ARCH_$(SRCARCH) doesn't seem to help.
> If you are interested in having a look at the code,
> The code (event_monitor_ebpf.c) including the makefile is here:
> https://github.com/yanivagman/tracee/tree/move_to_libbpf/tracee

You are including a lot of kernel headers in that code. If any of them
pulls in some x86-specific stuff, you would get this problem. Try to
minimize the amount of includes you have there, maybe the issue will
go away.

If not, one other way would be to generate vmlinux.h for your specific
kernel build. It will have memory layout of all the structs identical
to what you get from kernel headers. Then you can replace all those
includes with a single #include "vmlinux.h". You can disable CO-RE
part with #define BPF_NO_PRESERVE_ACCESS_INDEX before vmlinux.h is
included.

But I'm a bit puzzled how your approach is going to work without CO-RE
if you ever intend to run your program on more than one build of the
kernel. Different kernel builds/versions might have different memory
layouts of the structs you are relying on, so you'd need to compile
them for each kernel build separately. So it's either CO-RE or
on-the-host compilation (what BCC is doing). And if you are moving
away from BCC, but not adopting CO-RE, I'm not sure how you are going
to deal with that issue.


>
> This is still WIP (the move to libbpf), and libbpf should be added as
> a submodule to the project root
>
> > > Unfortunately, CO-RE is not (yet) an option.
> > > I'm currently making the move from bcc to libbpf, and our application
> > > needs to support kernel 4.14, and work on all environments.
> >
> > Kernel version is not a big problem, it's vmlinux BTF availability
> > that could be a problem. vmlinux BTF can be added into any version of
> > kernel with pahole -J, post factum, but that assumes you have some
> > control over how kernels are built and distributed, of course.
> >
> > >
> > > > [...]