Re: libbpf error: unknown register name 'r0' in asm

[Yaniv Agman <yanivagman@gmail.com>]

‫בתאריך יום ב׳, 12 באוק׳ 2020 ב-23:03 מאת ‪Andrii Nakryiko‬‏
<‪andrii.nakryiko@gmail.com‬‏>:‬
>
> On Fri, Oct 9, 2020 at 1:24 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> >
> > ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-22:53 מאת ‪Andrii Nakryiko‬‏
> > <‪andrii.nakryiko@gmail.com‬‏>:‬
> > >
> > > On Fri, Oct 9, 2020 at 12:32 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> > > >
> > > > ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-21:39 מאת ‪Andrii Nakryiko‬‏
> > > > <‪andrii.nakryiko@gmail.com‬‏>:‬
> > > > >
> > > > > On Fri, Oct 9, 2020 at 11:33 AM Yaniv Agman <yanivagman@gmail.com> wrote:
> > > > > >
> > > > > > ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-21:21 מאת ‪Daniel Borkmann‬‏
> > > > > > <‪daniel@iogearbox.net‬‏>:‬
> > > > > > >
> > > > > > > On 10/9/20 8:09 PM, Yaniv Agman wrote:
> > > > > > > > ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-20:39 מאת ‪Daniel Borkmann‬‏
> > > > > > > > <‪daniel@iogearbox.net‬‏>:‬
> > > > > > > >>
> > > > > > > >> On 10/9/20 6:56 PM, Yaniv Agman wrote:
> > > > > > > >>> ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-19:27 מאת ‪Daniel Borkmann‬‏
> > > > > > > >>> <‪daniel@iogearbox.net‬‏>:‬
> > > > > > > >>>>
> > > > > > > >>>> [ Cc +Yonghong ]
> > > > > > > >>>>
> > > > > > > >>>> On 10/9/20 6:05 PM, Yaniv Agman wrote:
> > > > > > > >>>>> Pulling the latest changes of libbpf and compiling my application with it,
> > > > > > > >>>>> I see the following error:
> > > > > > > >>>>>
> > > > > > > >>>>> ../libbpf/src//root/usr/include/bpf/bpf_helpers.h:99:10: error:
> > > > > > > >>>>> unknown register name 'r0' in asm
> > > > > > > >>>>>                         : "r0", "r1", "r2", "r3", "r4", "r5");
> > > > > > > >>>>>
> > > > > > > >>>>> The commit which introduced this change is:
> > > > > > > >>>>> 80c7838600d39891f274e2f7508b95a75e4227c1
> > > > > > > >>>>>
> > > > > > > >>>>> I'm not sure if I'm doing something wrong (missing include?), or this
> > > > > > > >>>>> is a genuine error
> > > > > > > >>>>
> > > > > > > >>>> Seems like your clang/llvm version might be too old.
> > > > > > > >>>
> > > > > > > >>> I'm using clang 10.0.1
> > > > > > > >>
> > > > > > > >> Ah, okay, I see. Would this diff do the trick for you?
> > > > > > > >
> > > > > > > > Yes! Now it compiles without any problems!
> > > > > > >
> > > > > > > Great, thx, I'll cook proper fix and check with clang6 as Yonghong mentioned.
> > > > > > >
> > > > > >
> > > > > > Thanks!
> > > > > > Does this happen because I'm first compiling using "emit-llvm" and
> > > > > > then using llc?
> > > > >
> > > > > So this must be the reason, but I'll wait for Yonghong to confirm.
> > > > >
> > > > > > I wish I could use bpf target directly, but I'm then having problems
> > > > > > with includes of asm code (like pt_regs and other stuff)
> > > > >
> > > > > Are you developing for a 32-bit platform? Or what exactly is the
> > > > > problem? I've been trying to solve problems for 32-bit arches recently
> > > > > by making libbpf smarter, that relies on CO-RE though. Is CO-RE an
> > > > > option for you?
> > > > >
> > > >
> > > > Examples for the errors I'm getting:
> > > > /lib/modules/4.14.199-1-MANJARO/build/arch/x86/include/asm/atomic.h:177:9:
> > > > error: invalid output constraint '+q' in asm
> > > >         return xadd(&v->counter, i);
> > > >                ^
> > > > /lib/modules/4.14.199-1-MANJARO/build/arch/x86/include/asm/cmpxchg.h:234:25:
> > > > note: expanded from macro 'xadd'
> > > > #define xadd(ptr, inc)          __xadd((ptr), (inc), LOCK_PREFIX)
> > > > ...
> > > >
> > > > From What I understood, this is a known issue for tracing programs
> > > > (like the one I'm developing)
> > >
> > > We do have a bunch of selftests that use pt_regs and include, say,
> > > linux/ptrace.h header. I wonder why we are not seeing these problems.
> > > Selftests, btw, are also built with -emit-llvm and then piping output
> > > to llc.
> > >
> > > So.. there must be something else going on. It's hard to guess like
> > > this without seeing the code, but maybe -D__TARGET_ARCH_$(SRCARCH)
> > > during compilation could help, just as an idea.
> > >
> >
> > Adding -D__TARGET_ARCH_$(SRCARCH) doesn't seem to help.
> > If you are interested in having a look at the code,
> > The code (event_monitor_ebpf.c) including the makefile is here:
> > https://github.com/yanivagman/tracee/tree/move_to_libbpf/tracee
>
> You are including a lot of kernel headers in that code. If any of them
> pulls in some x86-specific stuff, you would get this problem. Try to
> minimize the amount of includes you have there, maybe the issue will
> go away.
>

I only added an include when I needed it, so I'm not sure I can remove
any of them...

> If not, one other way would be to generate vmlinux.h for your specific
> kernel build. It will have memory layout of all the structs identical
> to what you get from kernel headers. Then you can replace all those
> includes with a single #include "vmlinux.h". You can disable CO-RE
> part with #define BPF_NO_PRESERVE_ACCESS_INDEX before vmlinux.h is
> included.
>

My application is supposed to run on every linux environment with kernel > 4.14.
The only requirement from the user is having clang and kernel headers.
Is there a way to generate vmlinux.h from the given kernel headers?
From my understanding, vmlinux.h can only be generated from vmlinux
using bpftool,
but I don't want to require the users to supply vmlinux as well.

> But I'm a bit puzzled how your approach is going to work without CO-RE
> if you ever intend to run your program on more than one build of the
> kernel. Different kernel builds/versions might have different memory
> layouts of the structs you are relying on, so you'd need to compile
> them for each kernel build separately. So it's either CO-RE or
> on-the-host compilation (what BCC is doing). And if you are moving
> away from BCC, but not adopting CO-RE, I'm not sure how you are going
> to deal with that issue.
>

The idea is to compile the code once on the target node, then use the
created object file in subsequent runs.
Unlike bcc, which compiles the bpf code on every run,
with this approach, we "install" the application once, then the
program should run immediately without any further required
compilation
The end goal is to support CO-RE, after most distros will support it
out of the box, and the user won't have to supply vmlinux for the
program to work

>
> >
> > This is still WIP (the move to libbpf), and libbpf should be added as
> > a submodule to the project root
> >
> > > > Unfortunately, CO-RE is not (yet) an option.
> > > > I'm currently making the move from bcc to libbpf, and our application
> > > > needs to support kernel 4.14, and work on all environments.
> > >
> > > Kernel version is not a big problem, it's vmlinux BTF availability
> > > that could be a problem. vmlinux BTF can be added into any version of
> > > kernel with pahole -J, post factum, but that assumes you have some
> > > control over how kernels are built and distributed, of course.
> > >
> > > >
> > > > > [...]