From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 191B1C433E0 for ; Thu, 7 Jan 2021 23:33:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C21AB2360D for ; Thu, 7 Jan 2021 23:33:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727669AbhAGXdG (ORCPT ); Thu, 7 Jan 2021 18:33:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726854AbhAGXdF (ORCPT ); Thu, 7 Jan 2021 18:33:05 -0500 Received: from mail-yb1-xb33.google.com (mail-yb1-xb33.google.com [IPv6:2607:f8b0:4864:20::b33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C0E1C0612F4 for ; Thu, 7 Jan 2021 15:32:25 -0800 (PST) Received: by mail-yb1-xb33.google.com with SMTP id u203so7750551ybb.2 for ; Thu, 07 Jan 2021 15:32:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=RZKa1nQBEg9RWEcShkWYidMfQv9KIwWyqXw+S/3w7Nc=; b=glsKFwTBt0aA0D8uVHL+OfY9C0OlAvT03Hy1KjptBQ6sj2HkZk6LNdvdDxIUwtkp5h vet8uUMbigwAC+ZH/Xy1GPqnLqLhSwukSIraJ88T8ewjETcRZKFfQw23905SiyVJfhAL pwn8i+Xaw5zfCPt3xKN/MRL9qnf0FSo10Sys2QSzwyJ7YMOyQn8yJXvziSrXnTW2kqRa y7mf6fPiseExIPZ66p8LoGtnKuqUV1fyaUtnrAo53DjgzAVqGoczAiLRABJ5yH4XGNMD r33oPW0t2eZpjYjWpJx9fBaPA/TksUpbSTqw9yBgfR36KIPLQX2//kqmQbjSx7q27LgT xJ5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=RZKa1nQBEg9RWEcShkWYidMfQv9KIwWyqXw+S/3w7Nc=; b=Mju3GdIJz2xU9GxJpMYHSdX8en1DyKJfivN6FHUqzVluz66Pso1RHoXVuI26kJ2aIB SZPSA2FQ4Pc3dUcu5aecsVuEgOPQtGfuWyBODXgm3tvz9ZT5rfp91aqoFbfjSnY+Ju1W yOG+iO0Uif8HhAjgY89g9h/ydU9elZJIJzJ31ad5utIdkxHIyy5UhW4x5aKDOMyrn4ZU Gpy47gPEGrDsq3VA/me8dnAJBDIidjQ66NzlZ/ttgarrOzOPxSsVsGsPdCNqBfVxnEXv mm2UHoH8+AR441k50rI+m/BVW0unmBKiWd4C8Kry0ZAVWVFOMlmqjcYWWSDFgTMBM1cY V8fw== X-Gm-Message-State: AOAM533GJAAknKwbfz4muKV3WXQ1uXDAf8+C0FlwskWAhxar0yLQpfO6 LfEjehWF7zD3fApc+0mVHB3kKzpG1hiqcaC3C1vrG71H X-Google-Smtp-Source: ABdhPJz3iSNMl+2srqBJMfMTTJHrjAqU9kTixOS5+Bsw1vZ7nAZgpowLZqrBPcdfkQzjs7VwY4YyzwX6g7HsP1OT9bM= X-Received: by 2002:a25:854a:: with SMTP id f10mr1736758ybn.510.1610062344721; Thu, 07 Jan 2021 15:32:24 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Andrii Nakryiko Date: Thu, 7 Jan 2021 15:32:13 -0800 Message-ID: Subject: Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. To: Vamsi Kodavanty Cc: bpf Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org On Thu, Jan 7, 2021 at 2:45 PM Vamsi Kodavanty wrote: > > Andrii, > Thank you so much for the quick response. You were right. I was > trying to CO-RE the `tcpconnect.py` program. It had some `.rodata` > which I removed. Now the program goes much further. It now finishes > the relocations successfully. But, fails at the next step > `bpf_object__load_progs`. It's discouraged to use top posting on the mailing list. So for the future please reply inline. > > === > libbpf: prog 'tcp_v6_connect_ret': relo #3: patched insn #50 > (ALU/ALU64) imm 56 -> 56 > libbpf: failed to open '/sys/bus/event_source/devices/kprobe/type': No > such file or directory > libbpf: failed to determine kprobe perf type: No such file or directory > libbpf: prog 'tcp_v4_connect': failed to create kprobe > 'tcp_v4_connect' perf event: No such file or directory > libbpf: failed to auto-attach program 'tcp_v4_connect': -2 > === > > This host has a 4.14 kernel AmazonLinux2 and does not have the above > file. It instead has this > > === > $ sudo cat /sys/kernel/debug/tracing/kprobe_events > p:kprobes/p_sys_execve_bcc_2566 sys_execve > === > Right. Libbpf only supports a newer and safer way to attach to kprobes. For your experiments, try to stick to tracepoints and you'll have a better time. But it's another thing I've been meaning to add to libbpf for supporting older kernels. I even have code written to do legacy kprobe attachment, just need to find time to send a patch to add it as a fallback for kernels that don't support new kprobe interface. > I am guessing this is a backward compatibility issue?. I will try to > look at an earlier version of libbpf to see how this was handled. it was never supported by libbpf. BCC support both old and new APIs, but old APIs is more dangerous, as it's easy to leave attached kprobe BPF program active in the kernel unintentionally. E.g., if the process crashes. But with bpf_link and its expected use of bpf_link__destroy(), we can support the legacy API (it won't be any safer, but still). > Meanwhile, if you have further comments they are appreciated. > > And again I can't thank you enough for how helpful you have been and > your time. Cheers!. > > > > > On Thu, Jan 7, 2021 at 10:52 AM Andrii Nakryiko > wrote: > > > > On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty > > wrote: > > > > > > First of all thank you very much for your quick response. And helpful pointers. > > > It seems like you also think what I am attempting to do should work. > > > > > > Please see inline [VAMSI-2]. > > > > > > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko > > > wrote: > > > > > > > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty > > > > wrote: > > > > > > > > > > Had a few questions on CO-RE dependencies and usage. From what I read > > > > > CO-RE needs a supported kernel version and be compiled with > > > > > `CONFIG_DEBUG_INFO_BTF=y`. > > > > > > > > > > I also understand there are three pieces to enable CO-RE > > > > > functionality. (1) The BTF format. For efficient/compressed kernel > > > > > symbol table. (2) clang changes to emit the BTF relocations. (3) > > > > > > > > BTF is not really a symbol table, rather a type information. Like > > > > simpler and more compact DWARF. > > > > > > > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once > > > > > these 3 steps are done the resulting byte code is no different from > > > > > non-CO-RE byte code. > > > > > > > > > > Given this I am hoping the knowledgeable folks on this mailer correct > > > > > and guide me if I am stating something incorrectly. > > > > > > > > > > (1) Is the kernel support requirement ONLY for the purposes of > > > > > generating and exposing the BTF file information on > > > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications > > > > > `libbpf` can find the BTF information at a standard location?. > > > > > > > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also > > > > try to search for vmlinux image (and BTF info within it) in a few > > > > standard locations, see [0]. Early versions of in-kernel BTF didn't > > > > even expose /sys/kernel/btf/vmlinux. > > > > > > > > [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580 > > > > > > > > > > > > > > (2) If the answer to the above question is YES. Could the below > > > > > mechanism be used so that it works on all kernels whether they support > > > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?. > > > > > (a) Extract BTF generation process outside of the kernel build. > > > > > Use this to generate the equivalent BTF file for it. > > > > > > > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF > > > > info, but it's also possible to just embed BTF manually with a direct > > > > invocation of pahole -J, see [1] on how it's done for > > > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no > > > > matter the version, and it will work with CO-RE relocations. > > > > > > > > [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170 > > > > > > > > > > [VAMSI-2] Yes, this is exactly what I did. I extracted out the > > > `gen_btf` from the > > > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF > > > file for the > > > 4.14.0 kernel. > > > > > > > > (b) Make changes to `libbpf` to look for BTF not only at the > > > > > standard locations but also at a user specified location. The BTF file > > > > > generated in (a) can be presented here. > > > > > > > > You can already do that, actually, though it's not very obvious. You > > > > can specify (or override) kernel BTF location by using > > > > bpf_object__load_xattr() and passing target_btf_path pointing to your > > > > BTF location (see [2]). I've been meaning to add it instead to a > > > > bpf_object_open_opts, btw, to make its use possible with a BPF > > > > skeleton. Also keep in mind that currently libbpf expects that custom > > > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But > > > > we can improve that, of course. > > > > > > > > [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141 > > > > > > [VAMSI-2] I took a look at this and what you suggested above does not > > > work as is. > > > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like > > > `bpf_object__load_vmlinux_btf` is not yet modified to use the > > > `target_btf_path` attribute. > > > > Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but > > since then added a bunch more use cases. So some libbpf changes are > > needed to make this work. But it should still work for CO-RE to have a > > custom BTF. > > > > I'm not sure about making bpf_object__load_vmlinux_btf() load custom > > BTF as the real kernel BTF, because that will never work for > > fentry/fexit, struct_ops, etc. I think it is better to teach > > bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF > > if we need it only for CO-RE relocations *and* we have it overloaded > > with target_btf_path > > > > > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As > > > you suggested enabling > > > use from the BPF skeleton seems useful and I can possibly help with that. > > > > Yeah, adding something like core_btf_path option to > > bpf_object_open_opts would go nicely with this change. > > > > > > > > For now, just for proof of concept I modified the search options in > > > `libbpf_find_kernel_btf` to > > > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures. > > > > > > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0 > > > > so here you successfully loaded custom BTF, which is good. > > > > > libbpf: Kernel doesn't support BTF, skipping uploading it. > > > > this just means that your BPF object's BTF won't be loaded into the > > kernel. That's no big deal, ignore this. > > > > > libbpf: kernel doesn't support global data > > > > But this means that your BPF programs rely on global variables, which > > are not supported by the kernel. So you need to change the code to not > > use global variables to make this work on very old kernels. > > > > > > > libbpf: failed to load object 'tcpconnect_bpf' > > > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95 > > > failed to load BPF object: -95 > > > > This is probably OPNOTSUPP from the global data above > > > > > > > > This is the reason I had posted on the mailer. If the CO-RE executable > > > has relocations > > > resolved by the time of the BPF load. Why do we need to check for > > > kernel support?. Also, > > > does this mean what I am attempting to do will not work?. > > > > > > > it will work with minimal libbpf logic changes. Nothing in principle > > prevents this. > > > > > > > Best Regards. And again thanks a lot for your precious time. > > > - Vamsi. > > > > > > > > > > > > > This should provide us a way to enable CO-RE functionality on older > > > > > kernel versions as well. I tried to make the above changes and tried > > > > > against a 4.14 kernel and it did not work. Either I am not doing > > > > > something right or my assumptions are wrong. > > > > > > > > > > Thanks in advance for your time. And I hope someone here can guide me > > > > > in the right direction. > > > > > > > > > > Regards > > > > > Vamsi.