Re: [PATCH 3/3] bpftool: Allow to link libbpf dynamically

[Quentin Monnet <quentin.monnet@netronome.com>]

2019-11-27 15:15 UTC+0100 ~ Jiri Olsa <jolsa@redhat.com>
> On Wed, Nov 27, 2019 at 01:38:55PM +0000, Quentin Monnet wrote:
>> 2019-11-27 10:48 UTC+0100 ~ Jiri Olsa <jolsa@kernel.org>
>>> Currently we support only static linking with kernel's libbpf
>>> (tools/lib/bpf). This patch adds LIBBPF_DYNAMIC compile variable
>>> that triggers libbpf detection and bpf dynamic linking:
>>>
>>>   $ make -C tools/bpf/bpftool make LIBBPF_DYNAMIC=1
>>>
>>> If libbpf is not installed, build (with LIBBPF_DYNAMIC=1) stops with:
>>>
>>>   $ make -C tools/bpf/bpftool LIBBPF_DYNAMIC=1
>>>     Auto-detecting system features:
>>>     ...                        libbfd: [ on  ]
>>>     ...        disassembler-four-args: [ on  ]
>>>     ...                          zlib: [ on  ]
>>>     ...                        libbpf: [ OFF ]
>>>
>>>   Makefile:102: *** Error: libbpf-devel is missing, please install it.  Stop.
>>>
>>> Adding specific bpftool's libbpf check for libbpf_netlink_open (LIBBPF_0.0.6)
>>> which is the latest we need for bpftool at the moment.
>>>
>>> Adding LIBBPF_DIR compile variable to allow linking with
>>> libbpf installed into specific directory:
>>>
>>>   $ make -C tools/lib/bpf/ prefix=/tmp/libbpf/ install_lib install_headers
>>>   $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/
>>>
>>> It might be needed to clean build tree first because features
>>> framework does not detect the change properly:
>>>
>>>   $ make -C tools/build/feature clean
>>>   $ make -C tools/bpf/bpftool/ clean
>>>
>>> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
>>> ---
>>>  tools/bpf/bpftool/Makefile        | 40 ++++++++++++++++++++++++++++++-
>>>  tools/build/feature/test-libbpf.c |  9 +++++++
>>>  2 files changed, 48 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
>>> index 39bc6f0f4f0b..2b6ed08cb31e 100644
>>> --- a/tools/bpf/bpftool/Makefile
>>> +++ b/tools/bpf/bpftool/Makefile
>>
>>> @@ -55,7 +64,7 @@ ifneq ($(EXTRA_LDFLAGS),)
>>>  LDFLAGS += $(EXTRA_LDFLAGS)
>>>  endif
>>>  
>>> -LIBS = $(LIBBPF) -lelf -lz
>>> +LIBS = -lelf -lz
>>
>> Hi Jiri,
>>
>> This change seems to be breaking the build with the static library for
>> me. I know you add back $(LIBBPF) later in the Makefile, see at the end
>> of this email...
>>
>>>  
>>>  INSTALL ?= install
>>>  RM ?= rm -f
>>> @@ -64,6 +73,23 @@ FEATURE_USER = .bpftool
>>>  FEATURE_TESTS = libbfd disassembler-four-args reallocarray zlib
>>>  FEATURE_DISPLAY = libbfd disassembler-four-args zlib
>>>  
>>> +ifdef LIBBPF_DYNAMIC
>>> +  # Add libbpf check with the flags to ensure bpftool
>>> +  # specific version is detected.
>>
>> Nit: We do not check for a specific bpftool version, we check for a
>> recent enough libbpf version?
> 
> hi,
> we check for a version that has the latest exported function
> that bpftool needs, which is currently libbpf_netlink_open
> 
> please check the '#ifdef BPFTOOL' in feature/test-libbpf.c
> 
> it's like that because there's currently no support to check
> for particular library version in the build/features framework
> 
> I'll make that comment more clear

Thanks!
>>> +  FEATURE_CHECK_CFLAGS-libbpf := -DBPFTOOL
>>> +  FEATURE_TESTS   += libbpf
>>> +  FEATURE_DISPLAY += libbpf
>>> +
>>> +  # for linking with debug library run:
>>> +  # make LIBBPF_DYNAMIC=1 LIBBPF_DIR=/opt/libbpf
>>> +  ifdef LIBBPF_DIR
>>> +    LIBBPF_CFLAGS  := -I$(LIBBPF_DIR)/include
>>> +    LIBBPF_LDFLAGS := -L$(LIBBPF_DIR)/$(libdir_relative)
>>> +    FEATURE_CHECK_CFLAGS-libbpf  := $(LIBBPF_CFLAGS)
>>> +    FEATURE_CHECK_LDFLAGS-libbpf := $(LIBBPF_LDFLAGS)
>>> +  endif
>>> +endif
>>> +
>>>  check_feat := 1
>>>  NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall
>>>  ifdef MAKECMDGOALS
>>> @@ -88,6 +114,18 @@ ifeq ($(feature-reallocarray), 0)
>>>  CFLAGS += -DCOMPAT_NEED_REALLOCARRAY
>>>  endif
>>>  
>>> +ifdef LIBBPF_DYNAMIC
>>> +  ifeq ($(feature-libbpf), 1)
>>> +    LIBS    += -lbpf
>>> +    CFLAGS  += $(LIBBPF_CFLAGS)
>>> +    LDFLAGS += $(LIBBPF_LDFLAGS)
>>> +  else
>>> +    dummy := $(error Error: No libbpf devel library found, please install libbpf-devel)
>>
>> libbpf-devel sounds like a RH/Fedora package name, but other
>> distributions might have different names (Debian/Ubuntu would go by
>> libbpf-dev I suppose, although I don't believe such package exists at
>> the moment). Maybe use a more generic message?
> 
> sure, actually in perf we use both package names like:
> 
>   Error: No libbpf devel library found, please install libbpf-devel or libbpf-dev.
> 
> or we can go with generic message:
> 
>   Error: No libbpf devel library found, please install.

Either is fine with me, thanks!

>>> +  endif
>>> +else
>>> +  LIBS += $(LIBBPF)
>>
>> ... I believe the order of the libraries is relevant, and it seems the
>> static libbpf should be passed before the dynamic libs. Here I could fix
>> the build with the static library on my setup by prepending the library
>> path instead, like this:
>>
>> 	LIBS := $(LIBBPF) $(LIBS)
> 
> could you please paste the build error? I don't see any on Fedora,

Sure, here it is. On top of your branch (on Ubuntu 18.04):

----------------

$ make -C tools/bpf/bpftool/
make: Entering directory '/root/linux/bpf-next/tools/bpf/bpftool'

Auto-detecting system features:
...                        libbfd: [ on  ]
...        disassembler-four-args: [ on  ]
...                          zlib: [ on  ]

  CC       map_perf_ring.o
  CC       xlated_dumper.o
  CC       btf.o
  CC       tracelog.o
  CC       perf.o
  CC       cfg.o
  CC       btf_dumper.o
  CC       net.o
  CC       netlink_dumper.o
  CC       common.o
  CC       cgroup.o
  CC       main.o
  CC       json_writer.o
  CC       prog.o
  CC       map.o
  CC       feature.o
  CC       jit_disasm.o
  CC       disasm.o
make[1]: Entering directory '/root/linux/bpf-next/tools/lib/bpf'

Auto-detecting system features:
...                        libelf: [ on  ]
...                           bpf: [ on  ]

Parsed description of 117 helper function(s)
  MKDIR    staticobjs/
  CC       staticobjs/libbpf.o
  CC       staticobjs/bpf.o
  CC       staticobjs/nlattr.o
  CC       staticobjs/btf.o
  CC       staticobjs/libbpf_errno.o
  CC       staticobjs/str_error.o
  CC       staticobjs/netlink.o
  CC       staticobjs/bpf_prog_linfo.o
  CC       staticobjs/libbpf_probes.o
  CC       staticobjs/xsk.o
  CC       staticobjs/hashmap.o
  CC       staticobjs/btf_dump.o
  LD       staticobjs/libbpf-in.o
  LINK     libbpf.a
make[1]: Leaving directory '/root/linux/bpf-next/tools/lib/bpf'
  LINK     bpftool
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__init_prog_names':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:466: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:473: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__elf_finish':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:570: undefined reference to `elf_end'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__elf_init':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:600: undefined reference to `elf_memory'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:613: undefined reference to `elf_begin'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:623: undefined reference to `gelf_getehdr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object_search_section_size':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:714: undefined reference to `gelf_getshdr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:720: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:730: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:708: undefined reference to `elf_nextscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__variable_offset':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:784: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:790: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__init_user_maps':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:937: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:939: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:957: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:981: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:990: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__init_user_btf_maps':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1359: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1361: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `section_have_execinstr':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1428: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1432: undefined reference to `gelf_getshdr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_object__elf_collect':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1608: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1608: undefined reference to `elf_rawdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1619: undefined reference to `gelf_getshdr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1625: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1632: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1613: undefined reference to `elf_nextscn'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `bpf_program__collect_reloc':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1928: undefined reference to `gelf_getrel'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1932: undefined reference to `gelf_getsym'
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:1941: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `__bpf_object__open':
/root/linux/bpf-next/tools/lib/bpf/libbpf.c:3934: undefined reference to `elf_version'
/root/linux/bpf-next/tools/lib/bpf/libbpf.a(libbpf-in.o): In function `btf__parse_elf':
/root/linux/bpf-next/tools/lib/bpf/btf.c:413: undefined reference to `elf_version'
/root/linux/bpf-next/tools/lib/bpf/btf.c:427: undefined reference to `elf_begin'
/root/linux/bpf-next/tools/lib/bpf/btf.c:432: undefined reference to `gelf_getehdr'
/root/linux/bpf-next/tools/lib/bpf/btf.c:440: undefined reference to `elf_getscn'
/root/linux/bpf-next/tools/lib/bpf/btf.c:440: undefined reference to `elf_rawdata'
/root/linux/bpf-next/tools/lib/bpf/btf.c:450: undefined reference to `gelf_getshdr'
/root/linux/bpf-next/tools/lib/bpf/btf.c:455: undefined reference to `elf_strptr'
/root/linux/bpf-next/tools/lib/bpf/btf.c:462: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/btf.c:470: undefined reference to `elf_getdata'
/root/linux/bpf-next/tools/lib/bpf/btf.c:445: undefined reference to `elf_nextscn'
/root/linux/bpf-next/tools/lib/bpf/btf.c:500: undefined reference to `elf_end'
collect2: error: ld returned 1 exit status
Makefile:158: recipe for target 'bpftool' failed
make: *** [bpftool] Error 1
make: Leaving directory '/root/linux/bpf-next/tools/bpf/bpftool'

----------------

Adding the V=1 flag shows the compilation is attempted with the
following command:

gcc -O2 -W -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wbad-function-cast -Wdeclaration-after-statement -Wformat-security -Wformat-y2k -Winit-self -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wno-system-headers -Wold-style-definition -Wpacked -Wredundant-decls -Wstrict-prototypes -Wswitch-default -Wundef -Wwrite-strings -Wformat -Wstrict-aliasing=3 -Wshadow -DPACKAGE='"bpftool"' -D__EXPORTED_HEADERS__ -I/root/linux/bpf-next/kernel/bpf/ -I/root/linux/bpf-next/tools/include -I/root/linux/bpf-next/tools/include/uapi -I/root/linux/bpf-next/tools/lib/bpf -I/root/linux/bpf-next/tools/perf -DBPFTOOL_VERSION='"5.4.0"' -DDISASM_FOUR_ARGS_SIGNATURE -DHAVE_LIBBFD_SUPPORT  -o bpftool map_perf_ring.o perf.o net.o cgroup.o tracelog.o btf_dumper.o xlated_dumper.o json_writer.o prog.o map.o common.o btf.o cfg.o main.o netlink_dumper.o feature.o jit_disasm.o disasm.o -lelf -lz /root/linux/bpf-next/tools/lib/bpf/libbpf.a -lbfd -ldl -lopcodes

If I move -lelf and -lz after <path>/libbpf.a, then it builds fine.

Hope this helps,
Quentin