[PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[Yonghong Song]

LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them
llvm linker lld friendly. The scope of
existing relocations R_BPF_64_{64,32} is narrowed
and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
are introduced.

Let us add some documentation about llvm bpf
relocations so people can understand how to resolve
them properly in their respective tools.

Cc: John Fastabend <john.fastabend@gmail.com>
Cc: Lorenz Bauer <lmb@cloudflare.com>
Signed-off-by: Yonghong Song <yhs@fb.com>
---
 Documentation/bpf/index.rst      |   1 +
 Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
 2 files changed, 169 insertions(+)
 create mode 100644 Documentation/bpf/llvm_reloc.rst

diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
index a702f67dd45f..93e8cf12a6d4 100644
--- a/Documentation/bpf/index.rst
+++ b/Documentation/bpf/index.rst
@@ -84,6 +84,7 @@ Other
    :maxdepth: 1
 
    ringbuf
+   llvm_reloc
 
 .. Links:
 .. _networking-filter: ../networking/filter.rst
diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
new file mode 100644
index 000000000000..bc62bce591b1
--- /dev/null
+++ b/Documentation/bpf/llvm_reloc.rst
@@ -0,0 +1,168 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+====================
+BPF LLVM Relocations
+====================
+
+This document describes LLVM BPF backend relocation types.
+
+Relocation Record
+=================
+
+LLVM BPF backend records each relocation with the following 16-byte
+ELF structure::
+
+  typedef struct
+  {
+    Elf64_Addr    r_offset;  // Offset from the beginning of section.
+    Elf64_Xword   r_info;    // Relocation type and symbol index.
+  } Elf64_Rel;
+
+For static function/variable references, the symbol often refers to
+the section itself which has a value of 0. To identify actual static
+function/variable, its section offset or some computation result
+based on section offset is written to the original insn/data buffer,
+which is called ``IA`` (implicit addend) below.  For global
+function/variables, the symbol refers to actual global and the implicit
+addend is 0.
+
+Different Relocation Types
+==========================
+
+Six relocation types are supported. The following is an overview and
+``S`` represents the value of the symbol in the symbol table::
+
+  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
+  0     R_BPF_NONE         None
+  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA
+  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
+  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
+  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
+  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
+
+For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
+The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
+data bitsize is 32 (4 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
+The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
+bitsize is 64 (8 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
+But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
+``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
+is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
+to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
+become unusable by bcc and kernel.
+
+Type ``R_BPF_64_32`` is used for call instruction. The call target section
+offset is stored at ``r_offset + 4`` (32bit) and calculated as
+``(S + IA) / 8 - 1``.
+
+Examples
+========
+
+Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``
+and ``call`` instructions. For example::
+
+  __attribute__((noinline)) __attribute__((section("sec1")))
+  int gfunc(int a, int b) {
+    return a * b;
+  }
+  static __attribute__((noinline)) __attribute__((section("sec1")))
+  int lfunc(int a, int b) {
+    return a + b;
+  }
+  int global __attribute__((section("sec2")));
+  int test(int a, int b) {
+    return gfunc(a, b) +  lfunc(a, b) + global;
+  }
+
+Compiled with ``clang -target bpf -O2 -c test.c``, we will have
+following code with `llvm-objdump -d test.o``::
+
+  Disassembly of section .text:
+
+  0000000000000000 <test>:
+         0:       bf 26 00 00 00 00 00 00 r6 = r2
+         1:       bf 17 00 00 00 00 00 00 r7 = r1
+         2:       85 10 00 00 ff ff ff ff call -1
+         3:       bf 08 00 00 00 00 00 00 r8 = r0
+         4:       bf 71 00 00 00 00 00 00 r1 = r7
+         5:       bf 62 00 00 00 00 00 00 r2 = r6
+         6:       85 10 00 00 02 00 00 00 call 2
+         7:       0f 80 00 00 00 00 00 00 r0 += r8
+         8:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
+        10:       61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
+        11:       0f 10 00 00 00 00 00 00 r0 += r1
+        12:       95 00 00 00 00 00 00 00 exit
+
+  Disassembly of section sec1:
+
+  0000000000000000 <gfunc>:
+         0:       bf 20 00 00 00 00 00 00 r0 = r2
+         1:       2f 10 00 00 00 00 00 00 r0 *= r1
+         2:       95 00 00 00 00 00 00 00 exit
+
+  0000000000000018 <lfunc>:
+         3:       bf 20 00 00 00 00 00 00 r0 = r2
+         4:       0f 10 00 00 00 00 00 00 r0 += r1
+         5:       95 00 00 00 00 00 00 00 exit
+
+Three relocations are generated with ``llvm-readelf -r test.o``::
+
+  Relocation section '.rel.text' at offset 0x188 contains 3 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000010  000000040000000a R_BPF_64_32            0000000000000000 gfunc
+  0000000000000030  000000020000000a R_BPF_64_32            0000000000000000 sec1
+  0000000000000040  0000000600000001 R_BPF_64_64            0000000000000000 global
+
+The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,
+so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.
+The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section
+offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.
+
+The following is an example to show how R_BPF_64_ABS64 could be generated::
+
+  int global() { return 0; }
+  struct t { void *g; } gbl = { global };
+
+Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a
+relocation below in ``.data`` section with command
+``llvm-readelf -r test.o``::
+
+  Relocation section '.rel.data' at offset 0x458 contains 1 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000000  0000000700000002 R_BPF_64_ABS64         0000000000000000 global
+
+The relocation says the first 8-byte of ``.data`` section should be
+filled with address of ``global`` variable.
+
+With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of
+``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::
+
+  Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000006  0000000300000003 R_BPF_64_ABS32         0000000000000000 .debug_abbrev
+  000000000000000c  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  0000000000000012  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  0000000000000016  0000000600000003 R_BPF_64_ABS32         0000000000000000 .debug_line
+  000000000000001a  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  000000000000001e  0000000200000002 R_BPF_64_ABS64         0000000000000000 .text
+  000000000000002b  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  0000000000000037  0000000800000002 R_BPF_64_ABS64         0000000000000000 gbl
+  0000000000000040  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  ......
+
+The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
+
+  Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000084  0000000800000004 R_BPF_64_NODYLD32      0000000000000000 gbl
+
+  Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  000000000000002c  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
+  0000000000000040  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
-- 
2.30.2

Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[Yonghong Song]

On 5/22/21 9:39 AM, Yonghong Song wrote:
> LLVM upstream commit https://reviews.llvm.org/D102712
> made some changes to bpf relocations to make them
> llvm linker lld friendly. The scope of
> existing relocations R_BPF_64_{64,32} is narrowed
> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
> are introduced.

Daniel, John and Lorenz,

Could you help check how the new relocation scheme
may impact you? libbpf has a similar issue and is fixed by
   https://lore.kernel.org/bpf/20210522162341.3687617-1-yhs@fb.com/
In most cases, you should just change relocation enum number,
no relocation resolution is changed.

Please let me know. Thanks!

> 
> Let us add some documentation about llvm bpf
> relocations so people can understand how to resolve
> them properly in their respective tools.
> 
> Cc: John Fastabend <john.fastabend@gmail.com>
> Cc: Lorenz Bauer <lmb@cloudflare.com>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
>   Documentation/bpf/index.rst      |   1 +
>   Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
>   2 files changed, 169 insertions(+)
>   create mode 100644 Documentation/bpf/llvm_reloc.rst
> 
[..]

Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[Lorenz Bauer]

On Sat, 22 May 2021 at 17:44, Yonghong Song <yhs@fb.com> wrote:
>
> Daniel, John and Lorenz,
>
> Could you help check how the new relocation scheme
> may impact you? libbpf has a similar issue and is fixed by
>    https://lore.kernel.org/bpf/20210522162341.3687617-1-yhs@fb.com/
> In most cases, you should just change relocation enum number,
> no relocation resolution is changed.
>
> Please let me know. Thanks!

Thank you for the heads up :) cilium/ebpf currently doesn't look at
relocation types at all for better or worse. We simply collect
"well-known" sections like maps, programs, etc. and only process
relocations for these. So your change won't break cilium/ebpf, but it
makes me wonder whether we should check the relocation type.

Best
Lorenz

-- 
Lorenz Bauer  |  Systems Engineer
6th Floor, County Hall/The Riverside Building, SE1 7PB, UK

www.cloudflare.com

Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[Yonghong Song]

On 5/24/21 1:33 AM, Lorenz Bauer wrote:
> On Sat, 22 May 2021 at 17:44, Yonghong Song <yhs@fb.com> wrote:
>>
>> Daniel, John and Lorenz,
>>
>> Could you help check how the new relocation scheme
>> may impact you? libbpf has a similar issue and is fixed by
>>     https://lore.kernel.org/bpf/20210522162341.3687617-1-yhs@fb.com/
>> In most cases, you should just change relocation enum number,
>> no relocation resolution is changed.
>>
>> Please let me know. Thanks!
> 
> Thank you for the heads up :) cilium/ebpf currently doesn't look at
> relocation types at all for better or worse. We simply collect
> "well-known" sections like maps, programs, etc. and only process
> relocations for these. So your change won't break cilium/ebpf, but it

Thanks for confirmation. So yes, you should be fine though.

> makes me wonder whether we should check the relocation type.

If the library is used in a control environment, e.g., the object
file is generated by llvm, bpftool linker, etc. You should be fine.
But yes, checking relocation types will make the library more robust.

> 
> Best
> Lorenz
> 

Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[Andrii Nakryiko]

On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
>
> LLVM upstream commit https://reviews.llvm.org/D102712
> made some changes to bpf relocations to make them
> llvm linker lld friendly. The scope of
> existing relocations R_BPF_64_{64,32} is narrowed
> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
> are introduced.
>
> Let us add some documentation about llvm bpf
> relocations so people can understand how to resolve
> them properly in their respective tools.
>
> Cc: John Fastabend <john.fastabend@gmail.com>
> Cc: Lorenz Bauer <lmb@cloudflare.com>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
>  Documentation/bpf/index.rst      |   1 +
>  Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
>  2 files changed, 169 insertions(+)
>  create mode 100644 Documentation/bpf/llvm_reloc.rst
>
> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
> index a702f67dd45f..93e8cf12a6d4 100644
> --- a/Documentation/bpf/index.rst
> +++ b/Documentation/bpf/index.rst
> @@ -84,6 +84,7 @@ Other
>     :maxdepth: 1
>
>     ringbuf
> +   llvm_reloc
>
>  .. Links:
>  .. _networking-filter: ../networking/filter.rst
> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
> new file mode 100644
> index 000000000000..bc62bce591b1
> --- /dev/null
> +++ b/Documentation/bpf/llvm_reloc.rst
> @@ -0,0 +1,168 @@
> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> +
> +====================
> +BPF LLVM Relocations
> +====================
> +
> +This document describes LLVM BPF backend relocation types.
> +
> +Relocation Record
> +=================
> +
> +LLVM BPF backend records each relocation with the following 16-byte
> +ELF structure::
> +
> +  typedef struct
> +  {
> +    Elf64_Addr    r_offset;  // Offset from the beginning of section.
> +    Elf64_Xword   r_info;    // Relocation type and symbol index.
> +  } Elf64_Rel;
> +
> +For static function/variable references, the symbol often refers to
> +the section itself which has a value of 0. To identify actual static
> +function/variable, its section offset or some computation result
> +based on section offset is written to the original insn/data buffer,
> +which is called ``IA`` (implicit addend) below.  For global
> +function/variables, the symbol refers to actual global and the implicit
> +addend is 0.
> +
> +Different Relocation Types
> +==========================
> +
> +Six relocation types are supported. The following is an overview and
> +``S`` represents the value of the symbol in the symbol table::
> +
> +  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
> +  0     R_BPF_NONE         None
> +  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA

There are cases where we set all 64-bits of ld_imm64 (e.g., extern
ksym, global variables). Or those will be a different relocation now
(R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.

Looking at LLVM diff I haven't found a test for global variables (at
least I didn't realize it was there), so double-checking here (and it
might be a good idea to have an explicit test for global variables?)

> +  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
> +  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
> +  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
> +  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
> +
> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
> +data bitsize is 32 (4 bytes). The relocation can be resolved with
> +the symbol value plus implicit addend.
> +
> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
> +bitsize is 64 (8 bytes). The relocation can be resolved with
> +the symbol value plus implicit addend.
> +
> +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
> +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
> +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
> +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
> +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
> +become unusable by bcc and kernel.
> +
> +Type ``R_BPF_64_32`` is used for call instruction. The call target section
> +offset is stored at ``r_offset + 4`` (32bit) and calculated as
> +``(S + IA) / 8 - 1``.
> +
> +Examples
> +========
> +
> +Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``
> +and ``call`` instructions. For example::
> +
> +  __attribute__((noinline)) __attribute__((section("sec1")))
> +  int gfunc(int a, int b) {
> +    return a * b;
> +  }
> +  static __attribute__((noinline)) __attribute__((section("sec1")))
> +  int lfunc(int a, int b) {
> +    return a + b;
> +  }
> +  int global __attribute__((section("sec2")));
> +  int test(int a, int b) {
> +    return gfunc(a, b) +  lfunc(a, b) + global;
> +  }
> +
> +Compiled with ``clang -target bpf -O2 -c test.c``, we will have
> +following code with `llvm-objdump -d test.o``::

I recently learned about `llvm-objdump -dr test.o`, which shows
relocations inline, it would be nice to use that output here.

> +
> +  Disassembly of section .text:
> +
> +  0000000000000000 <test>:
> +         0:       bf 26 00 00 00 00 00 00 r6 = r2
> +         1:       bf 17 00 00 00 00 00 00 r7 = r1
> +         2:       85 10 00 00 ff ff ff ff call -1
> +         3:       bf 08 00 00 00 00 00 00 r8 = r0
> +         4:       bf 71 00 00 00 00 00 00 r1 = r7
> +         5:       bf 62 00 00 00 00 00 00 r2 = r6
> +         6:       85 10 00 00 02 00 00 00 call 2
> +         7:       0f 80 00 00 00 00 00 00 r0 += r8
> +         8:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
> +        10:       61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
> +        11:       0f 10 00 00 00 00 00 00 r0 += r1
> +        12:       95 00 00 00 00 00 00 00 exit
> +
> +  Disassembly of section sec1:
> +
> +  0000000000000000 <gfunc>:
> +         0:       bf 20 00 00 00 00 00 00 r0 = r2
> +         1:       2f 10 00 00 00 00 00 00 r0 *= r1
> +         2:       95 00 00 00 00 00 00 00 exit
> +
> +  0000000000000018 <lfunc>:
> +         3:       bf 20 00 00 00 00 00 00 r0 = r2
> +         4:       0f 10 00 00 00 00 00 00 r0 += r1
> +         5:       95 00 00 00 00 00 00 00 exit
> +
> +Three relocations are generated with ``llvm-readelf -r test.o``::
> +
> +  Relocation section '.rel.text' at offset 0x188 contains 3 entries:
> +      Offset             Info             Type               Symbol's Value  Symbol's Name
> +  0000000000000010  000000040000000a R_BPF_64_32            0000000000000000 gfunc
> +  0000000000000030  000000020000000a R_BPF_64_32            0000000000000000 sec1
> +  0000000000000040  0000000600000001 R_BPF_64_64            0000000000000000 global
> +
> +The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,
> +so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.
> +The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section
> +offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.
> +
> +The following is an example to show how R_BPF_64_ABS64 could be generated::
> +
> +  int global() { return 0; }
> +  struct t { void *g; } gbl = { global };
> +
> +Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a
> +relocation below in ``.data`` section with command
> +``llvm-readelf -r test.o``::
> +
> +  Relocation section '.rel.data' at offset 0x458 contains 1 entries:
> +      Offset             Info             Type               Symbol's Value  Symbol's Name
> +  0000000000000000  0000000700000002 R_BPF_64_ABS64         0000000000000000 global
> +
> +The relocation says the first 8-byte of ``.data`` section should be
> +filled with address of ``global`` variable.
> +
> +With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of
> +``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::
> +
> +  Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:
> +      Offset             Info             Type               Symbol's Value  Symbol's Name
> +  0000000000000006  0000000300000003 R_BPF_64_ABS32         0000000000000000 .debug_abbrev
> +  000000000000000c  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
> +  0000000000000012  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
> +  0000000000000016  0000000600000003 R_BPF_64_ABS32         0000000000000000 .debug_line
> +  000000000000001a  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
> +  000000000000001e  0000000200000002 R_BPF_64_ABS64         0000000000000000 .text
> +  000000000000002b  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
> +  0000000000000037  0000000800000002 R_BPF_64_ABS64         0000000000000000 gbl
> +  0000000000000040  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
> +  ......
> +
> +The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
> +
> +  Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:
> +      Offset             Info             Type               Symbol's Value  Symbol's Name
> +  0000000000000084  0000000800000004 R_BPF_64_NODYLD32      0000000000000000 gbl
> +
> +  Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:
> +      Offset             Info             Type               Symbol's Value  Symbol's Name
> +  000000000000002c  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
> +  0000000000000040  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
> --
> 2.30.2
>

Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[Yonghong Song]

On 5/24/21 10:23 AM, Andrii Nakryiko wrote:
> On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
>>
>> LLVM upstream commit https://reviews.llvm.org/D102712
>> made some changes to bpf relocations to make them
>> llvm linker lld friendly. The scope of
>> existing relocations R_BPF_64_{64,32} is narrowed
>> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
>> are introduced.
>>
>> Let us add some documentation about llvm bpf
>> relocations so people can understand how to resolve
>> them properly in their respective tools.
>>
>> Cc: John Fastabend <john.fastabend@gmail.com>
>> Cc: Lorenz Bauer <lmb@cloudflare.com>
>> Signed-off-by: Yonghong Song <yhs@fb.com>
>> ---
>>   Documentation/bpf/index.rst      |   1 +
>>   Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
>>   2 files changed, 169 insertions(+)
>>   create mode 100644 Documentation/bpf/llvm_reloc.rst
>>
>> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
>> index a702f67dd45f..93e8cf12a6d4 100644
>> --- a/Documentation/bpf/index.rst
>> +++ b/Documentation/bpf/index.rst
>> @@ -84,6 +84,7 @@ Other
>>      :maxdepth: 1
>>
>>      ringbuf
>> +   llvm_reloc
>>
>>   .. Links:
>>   .. _networking-filter: ../networking/filter.rst
>> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
>> new file mode 100644
>> index 000000000000..bc62bce591b1
>> --- /dev/null
>> +++ b/Documentation/bpf/llvm_reloc.rst
>> @@ -0,0 +1,168 @@
>> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
>> +
>> +====================
>> +BPF LLVM Relocations
>> +====================
>> +
>> +This document describes LLVM BPF backend relocation types.
>> +
>> +Relocation Record
>> +=================
>> +
>> +LLVM BPF backend records each relocation with the following 16-byte
>> +ELF structure::
>> +
>> +  typedef struct
>> +  {
>> +    Elf64_Addr    r_offset;  // Offset from the beginning of section.
>> +    Elf64_Xword   r_info;    // Relocation type and symbol index.
>> +  } Elf64_Rel;
>> +
>> +For static function/variable references, the symbol often refers to
>> +the section itself which has a value of 0. To identify actual static
>> +function/variable, its section offset or some computation result
>> +based on section offset is written to the original insn/data buffer,
>> +which is called ``IA`` (implicit addend) below.  For global
>> +function/variables, the symbol refers to actual global and the implicit
>> +addend is 0.
>> +
>> +Different Relocation Types
>> +==========================
>> +
>> +Six relocation types are supported. The following is an overview and
>> +``S`` represents the value of the symbol in the symbol table::
>> +
>> +  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
>> +  0     R_BPF_NONE         None
>> +  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA
> 
> There are cases where we set all 64-bits of ld_imm64 (e.g., extern
> ksym, global variables). Or those will be a different relocation now
> (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.

It is still R_BPF_64_64. In llvm, we have restriction that section
offset must be <= UINT32_MAX, and that is why only 32bit is used
to find the actual symbol in symbol table. 32bit permits 4GB section
which should enough in practice for a bpf program.

libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.

The name is a little bit misleading, but it has become part of ABI
and lives in /usr/include/elf.h and we are not able to change it
any more.

> 
> Looking at LLVM diff I haven't found a test for global variables (at
> least I didn't realize it was there), so double-checking here (and it
> might be a good idea to have an explicit test for global variables?)

We have llvm/test/CodeGen/BPF/reloc.ll and 
llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think 
I can enhance
llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.

> 
>> +  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
>> +  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
>> +  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
>> +  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
>> +
>> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
>> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
>> +data bitsize is 32 (4 bytes). The relocation can be resolved with
>> +the symbol value plus implicit addend.
>> +
>> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
>> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
>> +bitsize is 64 (8 bytes). The relocation can be resolved with
>> +the symbol value plus implicit addend.
>> +
>> +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
>> +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
>> +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
>> +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
>> +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
>> +become unusable by bcc and kernel.
>> +
>> +Type ``R_BPF_64_32`` is used for call instruction. The call target section
>> +offset is stored at ``r_offset + 4`` (32bit) and calculated as
>> +``(S + IA) / 8 - 1``.
>> +
>> +Examples
>> +========
>> +
>> +Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``
>> +and ``call`` instructions. For example::
>> +
>> +  __attribute__((noinline)) __attribute__((section("sec1")))
>> +  int gfunc(int a, int b) {
>> +    return a * b;
>> +  }
>> +  static __attribute__((noinline)) __attribute__((section("sec1")))
>> +  int lfunc(int a, int b) {
>> +    return a + b;
>> +  }
>> +  int global __attribute__((section("sec2")));
>> +  int test(int a, int b) {
>> +    return gfunc(a, b) +  lfunc(a, b) + global;
>> +  }
>> +
>> +Compiled with ``clang -target bpf -O2 -c test.c``, we will have
>> +following code with `llvm-objdump -d test.o``::
> 
> I recently learned about `llvm-objdump -dr test.o`, which shows
> relocations inline, it would be nice to use that output here.

Yes, will do.

> 
>> +
>> +  Disassembly of section .text:
>> +
>> +  0000000000000000 <test>:
>> +         0:       bf 26 00 00 00 00 00 00 r6 = r2
>> +         1:       bf 17 00 00 00 00 00 00 r7 = r1
>> +         2:       85 10 00 00 ff ff ff ff call -1
>> +         3:       bf 08 00 00 00 00 00 00 r8 = r0
>> +         4:       bf 71 00 00 00 00 00 00 r1 = r7
>> +         5:       bf 62 00 00 00 00 00 00 r2 = r6
>> +         6:       85 10 00 00 02 00 00 00 call 2
>> +         7:       0f 80 00 00 00 00 00 00 r0 += r8
>> +         8:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
>> +        10:       61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
>> +        11:       0f 10 00 00 00 00 00 00 r0 += r1
>> +        12:       95 00 00 00 00 00 00 00 exit
>> +
>> +  Disassembly of section sec1:
>> +
>> +  0000000000000000 <gfunc>:
>> +         0:       bf 20 00 00 00 00 00 00 r0 = r2
>> +         1:       2f 10 00 00 00 00 00 00 r0 *= r1
>> +         2:       95 00 00 00 00 00 00 00 exit
>> +
>> +  0000000000000018 <lfunc>:
>> +         3:       bf 20 00 00 00 00 00 00 r0 = r2
>> +         4:       0f 10 00 00 00 00 00 00 r0 += r1
>> +         5:       95 00 00 00 00 00 00 00 exit
>> +
>> +Three relocations are generated with ``llvm-readelf -r test.o``::
>> +
>> +  Relocation section '.rel.text' at offset 0x188 contains 3 entries:
>> +      Offset             Info             Type               Symbol's Value  Symbol's Name
>> +  0000000000000010  000000040000000a R_BPF_64_32            0000000000000000 gfunc
>> +  0000000000000030  000000020000000a R_BPF_64_32            0000000000000000 sec1
>> +  0000000000000040  0000000600000001 R_BPF_64_64            0000000000000000 global
>> +
>> +The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,
>> +so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.
>> +The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section
>> +offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.
>> +
>> +The following is an example to show how R_BPF_64_ABS64 could be generated::
>> +
>> +  int global() { return 0; }
>> +  struct t { void *g; } gbl = { global };
>> +
>> +Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a
>> +relocation below in ``.data`` section with command
>> +``llvm-readelf -r test.o``::
>> +
>> +  Relocation section '.rel.data' at offset 0x458 contains 1 entries:
>> +      Offset             Info             Type               Symbol's Value  Symbol's Name
>> +  0000000000000000  0000000700000002 R_BPF_64_ABS64         0000000000000000 global
>> +
>> +The relocation says the first 8-byte of ``.data`` section should be
>> +filled with address of ``global`` variable.
>> +
>> +With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of
>> +``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::
>> +
>> +  Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:
>> +      Offset             Info             Type               Symbol's Value  Symbol's Name
>> +  0000000000000006  0000000300000003 R_BPF_64_ABS32         0000000000000000 .debug_abbrev
>> +  000000000000000c  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
>> +  0000000000000012  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
>> +  0000000000000016  0000000600000003 R_BPF_64_ABS32         0000000000000000 .debug_line
>> +  000000000000001a  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
>> +  000000000000001e  0000000200000002 R_BPF_64_ABS64         0000000000000000 .text
>> +  000000000000002b  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
>> +  0000000000000037  0000000800000002 R_BPF_64_ABS64         0000000000000000 gbl
>> +  0000000000000040  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
>> +  ......
>> +
>> +The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
>> +
>> +  Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:
>> +      Offset             Info             Type               Symbol's Value  Symbol's Name
>> +  0000000000000084  0000000800000004 R_BPF_64_NODYLD32      0000000000000000 gbl
>> +
>> +  Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:
>> +      Offset             Info             Type               Symbol's Value  Symbol's Name
>> +  000000000000002c  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
>> +  0000000000000040  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
>> --
>> 2.30.2
>>

Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[Andrii Nakryiko]

On Mon, May 24, 2021 at 11:01 AM Yonghong Song <yhs@fb.com> wrote:
>
>
>
> On 5/24/21 10:23 AM, Andrii Nakryiko wrote:
> > On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
> >>
> >> LLVM upstream commit https://reviews.llvm.org/D102712
> >> made some changes to bpf relocations to make them
> >> llvm linker lld friendly. The scope of
> >> existing relocations R_BPF_64_{64,32} is narrowed
> >> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
> >> are introduced.
> >>
> >> Let us add some documentation about llvm bpf
> >> relocations so people can understand how to resolve
> >> them properly in their respective tools.
> >>
> >> Cc: John Fastabend <john.fastabend@gmail.com>
> >> Cc: Lorenz Bauer <lmb@cloudflare.com>
> >> Signed-off-by: Yonghong Song <yhs@fb.com>
> >> ---
> >>   Documentation/bpf/index.rst      |   1 +
> >>   Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
> >>   2 files changed, 169 insertions(+)
> >>   create mode 100644 Documentation/bpf/llvm_reloc.rst
> >>
> >> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
> >> index a702f67dd45f..93e8cf12a6d4 100644
> >> --- a/Documentation/bpf/index.rst
> >> +++ b/Documentation/bpf/index.rst
> >> @@ -84,6 +84,7 @@ Other
> >>      :maxdepth: 1
> >>
> >>      ringbuf
> >> +   llvm_reloc
> >>
> >>   .. Links:
> >>   .. _networking-filter: ../networking/filter.rst
> >> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
> >> new file mode 100644
> >> index 000000000000..bc62bce591b1
> >> --- /dev/null
> >> +++ b/Documentation/bpf/llvm_reloc.rst
> >> @@ -0,0 +1,168 @@
> >> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> >> +
> >> +====================
> >> +BPF LLVM Relocations
> >> +====================
> >> +
> >> +This document describes LLVM BPF backend relocation types.
> >> +
> >> +Relocation Record
> >> +=================
> >> +
> >> +LLVM BPF backend records each relocation with the following 16-byte
> >> +ELF structure::
> >> +
> >> +  typedef struct
> >> +  {
> >> +    Elf64_Addr    r_offset;  // Offset from the beginning of section.
> >> +    Elf64_Xword   r_info;    // Relocation type and symbol index.
> >> +  } Elf64_Rel;
> >> +
> >> +For static function/variable references, the symbol often refers to
> >> +the section itself which has a value of 0. To identify actual static
> >> +function/variable, its section offset or some computation result
> >> +based on section offset is written to the original insn/data buffer,
> >> +which is called ``IA`` (implicit addend) below.  For global
> >> +function/variables, the symbol refers to actual global and the implicit
> >> +addend is 0.
> >> +
> >> +Different Relocation Types
> >> +==========================
> >> +
> >> +Six relocation types are supported. The following is an overview and
> >> +``S`` represents the value of the symbol in the symbol table::
> >> +
> >> +  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
> >> +  0     R_BPF_NONE         None
> >> +  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA
> >
> > There are cases where we set all 64-bits of ld_imm64 (e.g., extern
> > ksym, global variables). Or those will be a different relocation now
> > (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
>
> It is still R_BPF_64_64. In llvm, we have restriction that section
> offset must be <= UINT32_MAX, and that is why only 32bit is used
> to find the actual symbol in symbol table. 32bit permits 4GB section
> which should enough in practice for a bpf program.
>
> libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.
>

Ok, sounds good.

> The name is a little bit misleading, but it has become part of ABI
> and lives in /usr/include/elf.h and we are not able to change it
> any more.
>
> >
> > Looking at LLVM diff I haven't found a test for global variables (at
> > least I didn't realize it was there), so double-checking here (and it
> > might be a good idea to have an explicit test for global variables?)
>
> We have llvm/test/CodeGen/BPF/reloc.ll and
> llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think
> I can enhance
> llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.
>

Great, thanks.

> >
> >> +  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
> >> +  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
> >> +  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
> >> +  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
> >> +
> >> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
> >> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
> >> +data bitsize is 32 (4 bytes). The relocation can be resolved with
> >> +the symbol value plus implicit addend.
> >> +
> >> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
> >> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
> >> +bitsize is 64 (8 bytes). The relocation can be resolved with
> >> +the symbol value plus implicit addend.
> >> +

[...]

Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[John Fastabend]

Yonghong Song wrote:
> 
> 
> On 5/24/21 10:23 AM, Andrii Nakryiko wrote:
> > On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
> >>
> >> LLVM upstream commit https://reviews.llvm.org/D102712
> >> made some changes to bpf relocations to make them
> >> llvm linker lld friendly. The scope of
> >> existing relocations R_BPF_64_{64,32} is narrowed
> >> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
> >> are introduced.
> >>
> >> Let us add some documentation about llvm bpf
> >> relocations so people can understand how to resolve
> >> them properly in their respective tools.
> >>
> >> Cc: John Fastabend <john.fastabend@gmail.com>
> >> Cc: Lorenz Bauer <lmb@cloudflare.com>
> >> Signed-off-by: Yonghong Song <yhs@fb.com>
> >> ---
> >>   Documentation/bpf/index.rst      |   1 +
> >>   Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
> >>   2 files changed, 169 insertions(+)
> >>   create mode 100644 Documentation/bpf/llvm_reloc.rst
> >>
> >> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
> >> index a702f67dd45f..93e8cf12a6d4 100644
> >> --- a/Documentation/bpf/index.rst
> >> +++ b/Documentation/bpf/index.rst
> >> @@ -84,6 +84,7 @@ Other
> >>      :maxdepth: 1
> >>
> >>      ringbuf
> >> +   llvm_reloc
> >>

Thanks Yonghong, I found this helpful. I still had to crack
open llvm code though to follow along. A couple small suggestions
below, may or may not be useful. Overall looks good. 

> >>   .. Links:
> >>   .. _networking-filter: ../networking/filter.rst
> >> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
> >> new file mode 100644
> >> index 000000000000..bc62bce591b1
> >> --- /dev/null
> >> +++ b/Documentation/bpf/llvm_reloc.rst
> >> @@ -0,0 +1,168 @@
> >> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> >> +
> >> +====================
> >> +BPF LLVM Relocations
> >> +====================
> >> +
> >> +This document describes LLVM BPF backend relocation types.
> >> +
> >> +Relocation Record
> >> +=================
> >> +
> >> +LLVM BPF backend records each relocation with the following 16-byte
> >> +ELF structure::
> >> +
> >> +  typedef struct
> >> +  {
> >> +    Elf64_Addr    r_offset;  // Offset from the beginning of section.
> >> +    Elf64_Xword   r_info;    // Relocation type and symbol index.
> >> +  } Elf64_Rel;
> >> +
> >> +For static function/variable references, the symbol often refers to
> >> +the section itself which has a value of 0. To identify actual static
> >> +function/variable, its section offset or some computation result
> >> +based on section offset is written to the original insn/data buffer,
> >> +which is called ``IA`` (implicit addend) below.  For global
> >> +function/variables, the symbol refers to actual global and the implicit
> >> +addend is 0.

Above was too terse for me to follow without looking into some clang
examples. Maybe an example right here would help not sure? Maybe expand
the text a bit? I don't have a really good suggestion.

> >> +
> >> +Different Relocation Types
> >> +==========================
> >> +
> >> +Six relocation types are supported. The following is an overview and
> >> +``S`` represents the value of the symbol in the symbol table::
> >> +
> >> +  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
> >> +  0     R_BPF_NONE         None
> >> +  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA
> > 
> > There are cases where we set all 64-bits of ld_imm64 (e.g., extern
> > ksym, global variables). Or those will be a different relocation now
> > (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
> 
> It is still R_BPF_64_64. In llvm, we have restriction that section
> offset must be <= UINT32_MAX, and that is why only 32bit is used
> to find the actual symbol in symbol table. 32bit permits 4GB section
> which should enough in practice for a bpf program.

^^^ maybe add this note in the doc somewhere? I had similar questions.

> 
> libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.
> 
> The name is a little bit misleading, but it has become part of ABI
> and lives in /usr/include/elf.h and we are not able to change it
> any more.
> 
> > 
> > Looking at LLVM diff I haven't found a test for global variables (at
> > least I didn't realize it was there), so double-checking here (and it
> > might be a good idea to have an explicit test for global variables?)
> 
> We have llvm/test/CodeGen/BPF/reloc.ll and 
> llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think 
> I can enhance
> llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.

^^^ maybe cross-reference llvm tests from kernel docs side? I often look at
these when I get something unexpected/unknown maybe others would find
it helpful, but not know where to look?

> 
> > 
> >> +  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
> >> +  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
> >> +  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
> >> +  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
> >> +
> >> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
> >> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
> >> +data bitsize is 32 (4 bytes). The relocation can be resolved with
> >> +the symbol value plus implicit addend.
> >> +
> >> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
> >> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
> >> +bitsize is 64 (8 bytes). The relocation can be resolved with
> >> +the symbol value plus implicit addend.
> >> +
> >> +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
> >> +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
> >> +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
> >> +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
> >> +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
> >> +become unusable by bcc and kernel.
> >> +
> >> +Type ``R_BPF_64_32`` is used for call instruction. The call target section
> >> +offset is stored at ``r_offset + 4`` (32bit) and calculated as
> >> +``(S + IA) / 8 - 1``.
> >> +
> >> +Examples
> >> +========
> >> +

I liked the examples.

Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

[Yonghong Song]

On 5/24/21 12:24 PM, John Fastabend wrote:
> Yonghong Song wrote:
>>
>>
>> On 5/24/21 10:23 AM, Andrii Nakryiko wrote:
>>> On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
>>>>
>>>> LLVM upstream commit https://reviews.llvm.org/D102712
>>>> made some changes to bpf relocations to make them
>>>> llvm linker lld friendly. The scope of
>>>> existing relocations R_BPF_64_{64,32} is narrowed
>>>> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
>>>> are introduced.
>>>>
>>>> Let us add some documentation about llvm bpf
>>>> relocations so people can understand how to resolve
>>>> them properly in their respective tools.
>>>>
>>>> Cc: John Fastabend <john.fastabend@gmail.com>
>>>> Cc: Lorenz Bauer <lmb@cloudflare.com>
>>>> Signed-off-by: Yonghong Song <yhs@fb.com>
>>>> ---
>>>>    Documentation/bpf/index.rst      |   1 +
>>>>    Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
>>>>    2 files changed, 169 insertions(+)
>>>>    create mode 100644 Documentation/bpf/llvm_reloc.rst
>>>>
>>>> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
>>>> index a702f67dd45f..93e8cf12a6d4 100644
>>>> --- a/Documentation/bpf/index.rst
>>>> +++ b/Documentation/bpf/index.rst
>>>> @@ -84,6 +84,7 @@ Other
>>>>       :maxdepth: 1
>>>>
>>>>       ringbuf
>>>> +   llvm_reloc
>>>>
> 
> Thanks Yonghong, I found this helpful. I still had to crack
> open llvm code though to follow along. A couple small suggestions
> below, may or may not be useful. Overall looks good.
> 
>>>>    .. Links:
>>>>    .. _networking-filter: ../networking/filter.rst
>>>> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
>>>> new file mode 100644
>>>> index 000000000000..bc62bce591b1
>>>> --- /dev/null
>>>> +++ b/Documentation/bpf/llvm_reloc.rst
>>>> @@ -0,0 +1,168 @@
>>>> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
>>>> +
>>>> +====================
>>>> +BPF LLVM Relocations
>>>> +====================
>>>> +
>>>> +This document describes LLVM BPF backend relocation types.
>>>> +
>>>> +Relocation Record
>>>> +=================
>>>> +
>>>> +LLVM BPF backend records each relocation with the following 16-byte
>>>> +ELF structure::
>>>> +
>>>> +  typedef struct
>>>> +  {
>>>> +    Elf64_Addr    r_offset;  // Offset from the beginning of section.
>>>> +    Elf64_Xword   r_info;    // Relocation type and symbol index.
>>>> +  } Elf64_Rel;
>>>> +
>>>> +For static function/variable references, the symbol often refers to
>>>> +the section itself which has a value of 0. To identify actual static
>>>> +function/variable, its section offset or some computation result
>>>> +based on section offset is written to the original insn/data buffer,
>>>> +which is called ``IA`` (implicit addend) below.  For global
>>>> +function/variables, the symbol refers to actual global and the implicit
>>>> +addend is 0.
> 
> Above was too terse for me to follow without looking into some clang
> examples. Maybe an example right here would help not sure? Maybe expand
> the text a bit? I don't have a really good suggestion.

Just send a new revision with an example. Hope it will make it easy to
understand the above ``IA`` concept.

> 
>>>> +
>>>> +Different Relocation Types
>>>> +==========================
>>>> +
>>>> +Six relocation types are supported. The following is an overview and
>>>> +``S`` represents the value of the symbol in the symbol table::
>>>> +
>>>> +  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
>>>> +  0     R_BPF_NONE         None
>>>> +  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA
>>>
>>> There are cases where we set all 64-bits of ld_imm64 (e.g., extern
>>> ksym, global variables). Or those will be a different relocation now
>>> (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
>>
>> It is still R_BPF_64_64. In llvm, we have restriction that section
>> offset must be <= UINT32_MAX, and that is why only 32bit is used
>> to find the actual symbol in symbol table. 32bit permits 4GB section
>> which should enough in practice for a bpf program.
> 
> ^^^ maybe add this note in the doc somewhere? I had similar questions.

Added in the new revision.

> 
>>
>> libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.
>>
>> The name is a little bit misleading, but it has become part of ABI
>> and lives in /usr/include/elf.h and we are not able to change it
>> any more.
>>
>>>
>>> Looking at LLVM diff I haven't found a test for global variables (at
>>> least I didn't realize it was there), so double-checking here (and it
>>> might be a good idea to have an explicit test for global variables?)
>>
>> We have llvm/test/CodeGen/BPF/reloc.ll and
>> llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think
>> I can enhance
>> llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.
> 
> ^^^ maybe cross-reference llvm tests from kernel docs side? I often look at
> these when I get something unexpected/unknown maybe others would find
> it helpful, but not know where to look?

The llvm patch has not merged. We need to merge libbpf patch first. 
Otherwise, nightly libbpf CI will fail. But this doc includes a link
to the LLVM patch and you can just go to that llvm patch to find
examples!

> 
>>
>>>
>>>> +  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
>>>> +  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
>>>> +  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
>>>> +  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
>>>> +
>>>> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
>>>> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
>>>> +data bitsize is 32 (4 bytes). The relocation can be resolved with
>>>> +the symbol value plus implicit addend.
>>>> +
>>>> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
>>>> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
>>>> +bitsize is 64 (8 bytes). The relocation can be resolved with
>>>> +the symbol value plus implicit addend.
>>>> +
>>>> +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
>>>> +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
>>>> +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
>>>> +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
>>>> +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
>>>> +become unusable by bcc and kernel.
>>>> +
>>>> +Type ``R_BPF_64_32`` is used for call instruction. The call target section
>>>> +offset is stored at ``r_offset + 4`` (32bit) and calculated as
>>>> +``(S + IA) / 8 - 1``.
>>>> +
>>>> +Examples
>>>> +========
>>>> +
> 
> I liked the examples.

Great. Just added one more in the new revision!