Re: libbpf error: unknown register name 'r0' in asm

From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
To: Yaniv Agman <yanivagman@gmail.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>, bpf <bpf@vger.kernel.org>,
	Yonghong Song <yhs@fb.com>
Subject: Re: libbpf error: unknown register name 'r0' in asm
Date: Fri, 9 Oct 2020 12:53:46 -0700	[thread overview]
Message-ID: <CAEf4Bzbm7D+ygkoCCoTy8OR0krVWosS_o13Gv4Xd2jhOSC5a7Q@mail.gmail.com> (raw)
In-Reply-To: <CAMy7=ZWhAzJP5m3QW0gHe4rVFoETT=zhCcyVeKBuTcO=ttC=MA@mail.gmail.com>

On Fri, Oct 9, 2020 at 12:32 PM Yaniv Agman <yanivagman@gmail.com> wrote:
>
> ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-21:39 מאת ‪Andrii Nakryiko‬‏
> <‪andrii.nakryiko@gmail.com‬‏>:‬
> >
> > On Fri, Oct 9, 2020 at 11:33 AM Yaniv Agman <yanivagman@gmail.com> wrote:
> > >
> > > ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-21:21 מאת ‪Daniel Borkmann‬‏
> > > <‪daniel@iogearbox.net‬‏>:‬
> > > >
> > > > On 10/9/20 8:09 PM, Yaniv Agman wrote:
> > > > > ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-20:39 מאת ‪Daniel Borkmann‬‏
> > > > > <‪daniel@iogearbox.net‬‏>:‬
> > > > >>
> > > > >> On 10/9/20 6:56 PM, Yaniv Agman wrote:
> > > > >>> ‫בתאריך יום ו׳, 9 באוק׳ 2020 ב-19:27 מאת ‪Daniel Borkmann‬‏
> > > > >>> <‪daniel@iogearbox.net‬‏>:‬
> > > > >>>>
> > > > >>>> [ Cc +Yonghong ]
> > > > >>>>
> > > > >>>> On 10/9/20 6:05 PM, Yaniv Agman wrote:
> > > > >>>>> Pulling the latest changes of libbpf and compiling my application with it,
> > > > >>>>> I see the following error:
> > > > >>>>>
> > > > >>>>> ../libbpf/src//root/usr/include/bpf/bpf_helpers.h:99:10: error:
> > > > >>>>> unknown register name 'r0' in asm
> > > > >>>>>                         : "r0", "r1", "r2", "r3", "r4", "r5");
> > > > >>>>>
> > > > >>>>> The commit which introduced this change is:
> > > > >>>>> 80c7838600d39891f274e2f7508b95a75e4227c1
> > > > >>>>>
> > > > >>>>> I'm not sure if I'm doing something wrong (missing include?), or this
> > > > >>>>> is a genuine error
> > > > >>>>
> > > > >>>> Seems like your clang/llvm version might be too old.
> > > > >>>
> > > > >>> I'm using clang 10.0.1
> > > > >>
> > > > >> Ah, okay, I see. Would this diff do the trick for you?
> > > > >
> > > > > Yes! Now it compiles without any problems!
> > > >
> > > > Great, thx, I'll cook proper fix and check with clang6 as Yonghong mentioned.
> > > >
> > >
> > > Thanks!
> > > Does this happen because I'm first compiling using "emit-llvm" and
> > > then using llc?
> >
> > So this must be the reason, but I'll wait for Yonghong to confirm.
> >
> > > I wish I could use bpf target directly, but I'm then having problems
> > > with includes of asm code (like pt_regs and other stuff)
> >
> > Are you developing for a 32-bit platform? Or what exactly is the
> > problem? I've been trying to solve problems for 32-bit arches recently
> > by making libbpf smarter, that relies on CO-RE though. Is CO-RE an
> > option for you?
> >
>
> Examples for the errors I'm getting:
> /lib/modules/4.14.199-1-MANJARO/build/arch/x86/include/asm/atomic.h:177:9:
> error: invalid output constraint '+q' in asm
>         return xadd(&v->counter, i);
>                ^
> /lib/modules/4.14.199-1-MANJARO/build/arch/x86/include/asm/cmpxchg.h:234:25:
> note: expanded from macro 'xadd'
> #define xadd(ptr, inc)          __xadd((ptr), (inc), LOCK_PREFIX)
> ...
>
> From What I understood, this is a known issue for tracing programs
> (like the one I'm developing)

We do have a bunch of selftests that use pt_regs and include, say,
linux/ptrace.h header. I wonder why we are not seeing these problems.
Selftests, btw, are also built with -emit-llvm and then piping output
to llc.

So.. there must be something else going on. It's hard to guess like
this without seeing the code, but maybe -D__TARGET_ARCH_$(SRCARCH)
during compilation could help, just as an idea.

> Unfortunately, CO-RE is not (yet) an option.
> I'm currently making the move from bcc to libbpf, and our application
> needs to support kernel 4.14, and work on all environments.

Kernel version is not a big problem, it's vmlinux BTF availability
that could be a problem. vmlinux BTF can be added into any version of
kernel with pahole -J, post factum, but that assumes you have some
control over how kernels are built and distributed, of course.

>
> > [...]