* [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
@ 2021-05-22 16:39 Yonghong Song
2021-05-22 16:44 ` Yonghong Song
2021-05-24 17:23 ` Andrii Nakryiko
0 siblings, 2 replies; 9+ messages in thread
From: Yonghong Song @ 2021-05-22 16:39 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
kernel-team, John Fastabend, Lorenz Bauer
LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them
llvm linker lld friendly. The scope of
existing relocations R_BPF_64_{64,32} is narrowed
and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
are introduced.
Let us add some documentation about llvm bpf
relocations so people can understand how to resolve
them properly in their respective tools.
Cc: John Fastabend <john.fastabend@gmail.com>
Cc: Lorenz Bauer <lmb@cloudflare.com>
Signed-off-by: Yonghong Song <yhs@fb.com>
---
Documentation/bpf/index.rst | 1 +
Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
2 files changed, 169 insertions(+)
create mode 100644 Documentation/bpf/llvm_reloc.rst
diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
index a702f67dd45f..93e8cf12a6d4 100644
--- a/Documentation/bpf/index.rst
+++ b/Documentation/bpf/index.rst
@@ -84,6 +84,7 @@ Other
:maxdepth: 1
ringbuf
+ llvm_reloc
.. Links:
.. _networking-filter: ../networking/filter.rst
diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
new file mode 100644
index 000000000000..bc62bce591b1
--- /dev/null
+++ b/Documentation/bpf/llvm_reloc.rst
@@ -0,0 +1,168 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+====================
+BPF LLVM Relocations
+====================
+
+This document describes LLVM BPF backend relocation types.
+
+Relocation Record
+=================
+
+LLVM BPF backend records each relocation with the following 16-byte
+ELF structure::
+
+ typedef struct
+ {
+ Elf64_Addr r_offset; // Offset from the beginning of section.
+ Elf64_Xword r_info; // Relocation type and symbol index.
+ } Elf64_Rel;
+
+For static function/variable references, the symbol often refers to
+the section itself which has a value of 0. To identify actual static
+function/variable, its section offset or some computation result
+based on section offset is written to the original insn/data buffer,
+which is called ``IA`` (implicit addend) below. For global
+function/variables, the symbol refers to actual global and the implicit
+addend is 0.
+
+Different Relocation Types
+==========================
+
+Six relocation types are supported. The following is an overview and
+``S`` represents the value of the symbol in the symbol table::
+
+ Enum ELF Reloc Type Description BitSize Offset Calculation
+ 0 R_BPF_NONE None
+ 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IA
+ 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA
+ 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA
+ 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA
+ 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1
+
+For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
+The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
+data bitsize is 32 (4 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
+The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
+bitsize is 64 (8 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
+But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
+``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
+is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
+to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
+become unusable by bcc and kernel.
+
+Type ``R_BPF_64_32`` is used for call instruction. The call target section
+offset is stored at ``r_offset + 4`` (32bit) and calculated as
+``(S + IA) / 8 - 1``.
+
+Examples
+========
+
+Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``
+and ``call`` instructions. For example::
+
+ __attribute__((noinline)) __attribute__((section("sec1")))
+ int gfunc(int a, int b) {
+ return a * b;
+ }
+ static __attribute__((noinline)) __attribute__((section("sec1")))
+ int lfunc(int a, int b) {
+ return a + b;
+ }
+ int global __attribute__((section("sec2")));
+ int test(int a, int b) {
+ return gfunc(a, b) + lfunc(a, b) + global;
+ }
+
+Compiled with ``clang -target bpf -O2 -c test.c``, we will have
+following code with `llvm-objdump -d test.o``::
+
+ Disassembly of section .text:
+
+ 0000000000000000 <test>:
+ 0: bf 26 00 00 00 00 00 00 r6 = r2
+ 1: bf 17 00 00 00 00 00 00 r7 = r1
+ 2: 85 10 00 00 ff ff ff ff call -1
+ 3: bf 08 00 00 00 00 00 00 r8 = r0
+ 4: bf 71 00 00 00 00 00 00 r1 = r7
+ 5: bf 62 00 00 00 00 00 00 r2 = r6
+ 6: 85 10 00 00 02 00 00 00 call 2
+ 7: 0f 80 00 00 00 00 00 00 r0 += r8
+ 8: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
+ 10: 61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
+ 11: 0f 10 00 00 00 00 00 00 r0 += r1
+ 12: 95 00 00 00 00 00 00 00 exit
+
+ Disassembly of section sec1:
+
+ 0000000000000000 <gfunc>:
+ 0: bf 20 00 00 00 00 00 00 r0 = r2
+ 1: 2f 10 00 00 00 00 00 00 r0 *= r1
+ 2: 95 00 00 00 00 00 00 00 exit
+
+ 0000000000000018 <lfunc>:
+ 3: bf 20 00 00 00 00 00 00 r0 = r2
+ 4: 0f 10 00 00 00 00 00 00 r0 += r1
+ 5: 95 00 00 00 00 00 00 00 exit
+
+Three relocations are generated with ``llvm-readelf -r test.o``::
+
+ Relocation section '.rel.text' at offset 0x188 contains 3 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 0000000000000010 000000040000000a R_BPF_64_32 0000000000000000 gfunc
+ 0000000000000030 000000020000000a R_BPF_64_32 0000000000000000 sec1
+ 0000000000000040 0000000600000001 R_BPF_64_64 0000000000000000 global
+
+The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,
+so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.
+The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section
+offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.
+
+The following is an example to show how R_BPF_64_ABS64 could be generated::
+
+ int global() { return 0; }
+ struct t { void *g; } gbl = { global };
+
+Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a
+relocation below in ``.data`` section with command
+``llvm-readelf -r test.o``::
+
+ Relocation section '.rel.data' at offset 0x458 contains 1 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 0000000000000000 0000000700000002 R_BPF_64_ABS64 0000000000000000 global
+
+The relocation says the first 8-byte of ``.data`` section should be
+filled with address of ``global`` variable.
+
+With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of
+``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::
+
+ Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 0000000000000006 0000000300000003 R_BPF_64_ABS32 0000000000000000 .debug_abbrev
+ 000000000000000c 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ 0000000000000012 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ 0000000000000016 0000000600000003 R_BPF_64_ABS32 0000000000000000 .debug_line
+ 000000000000001a 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ 000000000000001e 0000000200000002 R_BPF_64_ABS64 0000000000000000 .text
+ 000000000000002b 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ 0000000000000037 0000000800000002 R_BPF_64_ABS64 0000000000000000 gbl
+ 0000000000000040 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
+ ......
+
+The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
+
+ Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 0000000000000084 0000000800000004 R_BPF_64_NODYLD32 0000000000000000 gbl
+
+ Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:
+ Offset Info Type Symbol's Value Symbol's Name
+ 000000000000002c 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
+ 0000000000000040 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
--
2.30.2
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
2021-05-22 16:39 [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations Yonghong Song
@ 2021-05-22 16:44 ` Yonghong Song
2021-05-24 8:33 ` Lorenz Bauer
2021-05-24 17:23 ` Andrii Nakryiko
1 sibling, 1 reply; 9+ messages in thread
From: Yonghong Song @ 2021-05-22 16:44 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
kernel-team, John Fastabend, Lorenz Bauer
On 5/22/21 9:39 AM, Yonghong Song wrote:
> LLVM upstream commit https://reviews.llvm.org/D102712
> made some changes to bpf relocations to make them
> llvm linker lld friendly. The scope of
> existing relocations R_BPF_64_{64,32} is narrowed
> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
> are introduced.
Daniel, John and Lorenz,
Could you help check how the new relocation scheme
may impact you? libbpf has a similar issue and is fixed by
https://lore.kernel.org/bpf/20210522162341.3687617-1-yhs@fb.com/
In most cases, you should just change relocation enum number,
no relocation resolution is changed.
Please let me know. Thanks!
>
> Let us add some documentation about llvm bpf
> relocations so people can understand how to resolve
> them properly in their respective tools.
>
> Cc: John Fastabend <john.fastabend@gmail.com>
> Cc: Lorenz Bauer <lmb@cloudflare.com>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
> Documentation/bpf/index.rst | 1 +
> Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
> 2 files changed, 169 insertions(+)
> create mode 100644 Documentation/bpf/llvm_reloc.rst
>
[..]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
2021-05-22 16:44 ` Yonghong Song
@ 2021-05-24 8:33 ` Lorenz Bauer
2021-05-24 15:06 ` Yonghong Song
0 siblings, 1 reply; 9+ messages in thread
From: Lorenz Bauer @ 2021-05-24 8:33 UTC (permalink / raw)
To: Yonghong Song
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Kernel Team, John Fastabend
On Sat, 22 May 2021 at 17:44, Yonghong Song <yhs@fb.com> wrote:
>
> Daniel, John and Lorenz,
>
> Could you help check how the new relocation scheme
> may impact you? libbpf has a similar issue and is fixed by
> https://lore.kernel.org/bpf/20210522162341.3687617-1-yhs@fb.com/
> In most cases, you should just change relocation enum number,
> no relocation resolution is changed.
>
> Please let me know. Thanks!
Thank you for the heads up :) cilium/ebpf currently doesn't look at
relocation types at all for better or worse. We simply collect
"well-known" sections like maps, programs, etc. and only process
relocations for these. So your change won't break cilium/ebpf, but it
makes me wonder whether we should check the relocation type.
Best
Lorenz
--
Lorenz Bauer | Systems Engineer
6th Floor, County Hall/The Riverside Building, SE1 7PB, UK
www.cloudflare.com
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
2021-05-24 8:33 ` Lorenz Bauer
@ 2021-05-24 15:06 ` Yonghong Song
0 siblings, 0 replies; 9+ messages in thread
From: Yonghong Song @ 2021-05-24 15:06 UTC (permalink / raw)
To: Lorenz Bauer
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Kernel Team, John Fastabend
On 5/24/21 1:33 AM, Lorenz Bauer wrote:
> On Sat, 22 May 2021 at 17:44, Yonghong Song <yhs@fb.com> wrote:
>>
>> Daniel, John and Lorenz,
>>
>> Could you help check how the new relocation scheme
>> may impact you? libbpf has a similar issue and is fixed by
>> https://lore.kernel.org/bpf/20210522162341.3687617-1-yhs@fb.com/
>> In most cases, you should just change relocation enum number,
>> no relocation resolution is changed.
>>
>> Please let me know. Thanks!
>
> Thank you for the heads up :) cilium/ebpf currently doesn't look at
> relocation types at all for better or worse. We simply collect
> "well-known" sections like maps, programs, etc. and only process
> relocations for these. So your change won't break cilium/ebpf, but it
Thanks for confirmation. So yes, you should be fine though.
> makes me wonder whether we should check the relocation type.
If the library is used in a control environment, e.g., the object
file is generated by llvm, bpftool linker, etc. You should be fine.
But yes, checking relocation types will make the library more robust.
>
> Best
> Lorenz
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
2021-05-22 16:39 [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations Yonghong Song
2021-05-22 16:44 ` Yonghong Song
@ 2021-05-24 17:23 ` Andrii Nakryiko
2021-05-24 18:01 ` Yonghong Song
1 sibling, 1 reply; 9+ messages in thread
From: Andrii Nakryiko @ 2021-05-24 17:23 UTC (permalink / raw)
To: Yonghong Song
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Kernel Team, John Fastabend, Lorenz Bauer
On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
>
> LLVM upstream commit https://reviews.llvm.org/D102712
> made some changes to bpf relocations to make them
> llvm linker lld friendly. The scope of
> existing relocations R_BPF_64_{64,32} is narrowed
> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
> are introduced.
>
> Let us add some documentation about llvm bpf
> relocations so people can understand how to resolve
> them properly in their respective tools.
>
> Cc: John Fastabend <john.fastabend@gmail.com>
> Cc: Lorenz Bauer <lmb@cloudflare.com>
> Signed-off-by: Yonghong Song <yhs@fb.com>
> ---
> Documentation/bpf/index.rst | 1 +
> Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
> 2 files changed, 169 insertions(+)
> create mode 100644 Documentation/bpf/llvm_reloc.rst
>
> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
> index a702f67dd45f..93e8cf12a6d4 100644
> --- a/Documentation/bpf/index.rst
> +++ b/Documentation/bpf/index.rst
> @@ -84,6 +84,7 @@ Other
> :maxdepth: 1
>
> ringbuf
> + llvm_reloc
>
> .. Links:
> .. _networking-filter: ../networking/filter.rst
> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
> new file mode 100644
> index 000000000000..bc62bce591b1
> --- /dev/null
> +++ b/Documentation/bpf/llvm_reloc.rst
> @@ -0,0 +1,168 @@
> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> +
> +====================
> +BPF LLVM Relocations
> +====================
> +
> +This document describes LLVM BPF backend relocation types.
> +
> +Relocation Record
> +=================
> +
> +LLVM BPF backend records each relocation with the following 16-byte
> +ELF structure::
> +
> + typedef struct
> + {
> + Elf64_Addr r_offset; // Offset from the beginning of section.
> + Elf64_Xword r_info; // Relocation type and symbol index.
> + } Elf64_Rel;
> +
> +For static function/variable references, the symbol often refers to
> +the section itself which has a value of 0. To identify actual static
> +function/variable, its section offset or some computation result
> +based on section offset is written to the original insn/data buffer,
> +which is called ``IA`` (implicit addend) below. For global
> +function/variables, the symbol refers to actual global and the implicit
> +addend is 0.
> +
> +Different Relocation Types
> +==========================
> +
> +Six relocation types are supported. The following is an overview and
> +``S`` represents the value of the symbol in the symbol table::
> +
> + Enum ELF Reloc Type Description BitSize Offset Calculation
> + 0 R_BPF_NONE None
> + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IA
There are cases where we set all 64-bits of ld_imm64 (e.g., extern
ksym, global variables). Or those will be a different relocation now
(R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
Looking at LLVM diff I haven't found a test for global variables (at
least I didn't realize it was there), so double-checking here (and it
might be a good idea to have an explicit test for global variables?)
> + 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA
> + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA
> + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA
> + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1
> +
> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
> +data bitsize is 32 (4 bytes). The relocation can be resolved with
> +the symbol value plus implicit addend.
> +
> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
> +bitsize is 64 (8 bytes). The relocation can be resolved with
> +the symbol value plus implicit addend.
> +
> +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
> +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
> +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
> +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
> +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
> +become unusable by bcc and kernel.
> +
> +Type ``R_BPF_64_32`` is used for call instruction. The call target section
> +offset is stored at ``r_offset + 4`` (32bit) and calculated as
> +``(S + IA) / 8 - 1``.
> +
> +Examples
> +========
> +
> +Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``
> +and ``call`` instructions. For example::
> +
> + __attribute__((noinline)) __attribute__((section("sec1")))
> + int gfunc(int a, int b) {
> + return a * b;
> + }
> + static __attribute__((noinline)) __attribute__((section("sec1")))
> + int lfunc(int a, int b) {
> + return a + b;
> + }
> + int global __attribute__((section("sec2")));
> + int test(int a, int b) {
> + return gfunc(a, b) + lfunc(a, b) + global;
> + }
> +
> +Compiled with ``clang -target bpf -O2 -c test.c``, we will have
> +following code with `llvm-objdump -d test.o``::
I recently learned about `llvm-objdump -dr test.o`, which shows
relocations inline, it would be nice to use that output here.
> +
> + Disassembly of section .text:
> +
> + 0000000000000000 <test>:
> + 0: bf 26 00 00 00 00 00 00 r6 = r2
> + 1: bf 17 00 00 00 00 00 00 r7 = r1
> + 2: 85 10 00 00 ff ff ff ff call -1
> + 3: bf 08 00 00 00 00 00 00 r8 = r0
> + 4: bf 71 00 00 00 00 00 00 r1 = r7
> + 5: bf 62 00 00 00 00 00 00 r2 = r6
> + 6: 85 10 00 00 02 00 00 00 call 2
> + 7: 0f 80 00 00 00 00 00 00 r0 += r8
> + 8: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
> + 10: 61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
> + 11: 0f 10 00 00 00 00 00 00 r0 += r1
> + 12: 95 00 00 00 00 00 00 00 exit
> +
> + Disassembly of section sec1:
> +
> + 0000000000000000 <gfunc>:
> + 0: bf 20 00 00 00 00 00 00 r0 = r2
> + 1: 2f 10 00 00 00 00 00 00 r0 *= r1
> + 2: 95 00 00 00 00 00 00 00 exit
> +
> + 0000000000000018 <lfunc>:
> + 3: bf 20 00 00 00 00 00 00 r0 = r2
> + 4: 0f 10 00 00 00 00 00 00 r0 += r1
> + 5: 95 00 00 00 00 00 00 00 exit
> +
> +Three relocations are generated with ``llvm-readelf -r test.o``::
> +
> + Relocation section '.rel.text' at offset 0x188 contains 3 entries:
> + Offset Info Type Symbol's Value Symbol's Name
> + 0000000000000010 000000040000000a R_BPF_64_32 0000000000000000 gfunc
> + 0000000000000030 000000020000000a R_BPF_64_32 0000000000000000 sec1
> + 0000000000000040 0000000600000001 R_BPF_64_64 0000000000000000 global
> +
> +The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,
> +so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.
> +The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section
> +offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.
> +
> +The following is an example to show how R_BPF_64_ABS64 could be generated::
> +
> + int global() { return 0; }
> + struct t { void *g; } gbl = { global };
> +
> +Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a
> +relocation below in ``.data`` section with command
> +``llvm-readelf -r test.o``::
> +
> + Relocation section '.rel.data' at offset 0x458 contains 1 entries:
> + Offset Info Type Symbol's Value Symbol's Name
> + 0000000000000000 0000000700000002 R_BPF_64_ABS64 0000000000000000 global
> +
> +The relocation says the first 8-byte of ``.data`` section should be
> +filled with address of ``global`` variable.
> +
> +With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of
> +``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::
> +
> + Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:
> + Offset Info Type Symbol's Value Symbol's Name
> + 0000000000000006 0000000300000003 R_BPF_64_ABS32 0000000000000000 .debug_abbrev
> + 000000000000000c 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
> + 0000000000000012 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
> + 0000000000000016 0000000600000003 R_BPF_64_ABS32 0000000000000000 .debug_line
> + 000000000000001a 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
> + 000000000000001e 0000000200000002 R_BPF_64_ABS64 0000000000000000 .text
> + 000000000000002b 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
> + 0000000000000037 0000000800000002 R_BPF_64_ABS64 0000000000000000 gbl
> + 0000000000000040 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
> + ......
> +
> +The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
> +
> + Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:
> + Offset Info Type Symbol's Value Symbol's Name
> + 0000000000000084 0000000800000004 R_BPF_64_NODYLD32 0000000000000000 gbl
> +
> + Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:
> + Offset Info Type Symbol's Value Symbol's Name
> + 000000000000002c 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
> + 0000000000000040 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
> --
> 2.30.2
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
2021-05-24 17:23 ` Andrii Nakryiko
@ 2021-05-24 18:01 ` Yonghong Song
2021-05-24 19:20 ` Andrii Nakryiko
2021-05-24 19:24 ` John Fastabend
0 siblings, 2 replies; 9+ messages in thread
From: Yonghong Song @ 2021-05-24 18:01 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Kernel Team, John Fastabend, Lorenz Bauer
On 5/24/21 10:23 AM, Andrii Nakryiko wrote:
> On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
>>
>> LLVM upstream commit https://reviews.llvm.org/D102712
>> made some changes to bpf relocations to make them
>> llvm linker lld friendly. The scope of
>> existing relocations R_BPF_64_{64,32} is narrowed
>> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
>> are introduced.
>>
>> Let us add some documentation about llvm bpf
>> relocations so people can understand how to resolve
>> them properly in their respective tools.
>>
>> Cc: John Fastabend <john.fastabend@gmail.com>
>> Cc: Lorenz Bauer <lmb@cloudflare.com>
>> Signed-off-by: Yonghong Song <yhs@fb.com>
>> ---
>> Documentation/bpf/index.rst | 1 +
>> Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
>> 2 files changed, 169 insertions(+)
>> create mode 100644 Documentation/bpf/llvm_reloc.rst
>>
>> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
>> index a702f67dd45f..93e8cf12a6d4 100644
>> --- a/Documentation/bpf/index.rst
>> +++ b/Documentation/bpf/index.rst
>> @@ -84,6 +84,7 @@ Other
>> :maxdepth: 1
>>
>> ringbuf
>> + llvm_reloc
>>
>> .. Links:
>> .. _networking-filter: ../networking/filter.rst
>> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
>> new file mode 100644
>> index 000000000000..bc62bce591b1
>> --- /dev/null
>> +++ b/Documentation/bpf/llvm_reloc.rst
>> @@ -0,0 +1,168 @@
>> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
>> +
>> +====================
>> +BPF LLVM Relocations
>> +====================
>> +
>> +This document describes LLVM BPF backend relocation types.
>> +
>> +Relocation Record
>> +=================
>> +
>> +LLVM BPF backend records each relocation with the following 16-byte
>> +ELF structure::
>> +
>> + typedef struct
>> + {
>> + Elf64_Addr r_offset; // Offset from the beginning of section.
>> + Elf64_Xword r_info; // Relocation type and symbol index.
>> + } Elf64_Rel;
>> +
>> +For static function/variable references, the symbol often refers to
>> +the section itself which has a value of 0. To identify actual static
>> +function/variable, its section offset or some computation result
>> +based on section offset is written to the original insn/data buffer,
>> +which is called ``IA`` (implicit addend) below. For global
>> +function/variables, the symbol refers to actual global and the implicit
>> +addend is 0.
>> +
>> +Different Relocation Types
>> +==========================
>> +
>> +Six relocation types are supported. The following is an overview and
>> +``S`` represents the value of the symbol in the symbol table::
>> +
>> + Enum ELF Reloc Type Description BitSize Offset Calculation
>> + 0 R_BPF_NONE None
>> + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IA
>
> There are cases where we set all 64-bits of ld_imm64 (e.g., extern
> ksym, global variables). Or those will be a different relocation now
> (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
It is still R_BPF_64_64. In llvm, we have restriction that section
offset must be <= UINT32_MAX, and that is why only 32bit is used
to find the actual symbol in symbol table. 32bit permits 4GB section
which should enough in practice for a bpf program.
libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.
The name is a little bit misleading, but it has become part of ABI
and lives in /usr/include/elf.h and we are not able to change it
any more.
>
> Looking at LLVM diff I haven't found a test for global variables (at
> least I didn't realize it was there), so double-checking here (and it
> might be a good idea to have an explicit test for global variables?)
We have llvm/test/CodeGen/BPF/reloc.ll and
llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think
I can enhance
llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.
>
>> + 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA
>> + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA
>> + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA
>> + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1
>> +
>> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
>> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
>> +data bitsize is 32 (4 bytes). The relocation can be resolved with
>> +the symbol value plus implicit addend.
>> +
>> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
>> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
>> +bitsize is 64 (8 bytes). The relocation can be resolved with
>> +the symbol value plus implicit addend.
>> +
>> +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
>> +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
>> +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
>> +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
>> +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
>> +become unusable by bcc and kernel.
>> +
>> +Type ``R_BPF_64_32`` is used for call instruction. The call target section
>> +offset is stored at ``r_offset + 4`` (32bit) and calculated as
>> +``(S + IA) / 8 - 1``.
>> +
>> +Examples
>> +========
>> +
>> +Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``
>> +and ``call`` instructions. For example::
>> +
>> + __attribute__((noinline)) __attribute__((section("sec1")))
>> + int gfunc(int a, int b) {
>> + return a * b;
>> + }
>> + static __attribute__((noinline)) __attribute__((section("sec1")))
>> + int lfunc(int a, int b) {
>> + return a + b;
>> + }
>> + int global __attribute__((section("sec2")));
>> + int test(int a, int b) {
>> + return gfunc(a, b) + lfunc(a, b) + global;
>> + }
>> +
>> +Compiled with ``clang -target bpf -O2 -c test.c``, we will have
>> +following code with `llvm-objdump -d test.o``::
>
> I recently learned about `llvm-objdump -dr test.o`, which shows
> relocations inline, it would be nice to use that output here.
Yes, will do.
>
>> +
>> + Disassembly of section .text:
>> +
>> + 0000000000000000 <test>:
>> + 0: bf 26 00 00 00 00 00 00 r6 = r2
>> + 1: bf 17 00 00 00 00 00 00 r7 = r1
>> + 2: 85 10 00 00 ff ff ff ff call -1
>> + 3: bf 08 00 00 00 00 00 00 r8 = r0
>> + 4: bf 71 00 00 00 00 00 00 r1 = r7
>> + 5: bf 62 00 00 00 00 00 00 r2 = r6
>> + 6: 85 10 00 00 02 00 00 00 call 2
>> + 7: 0f 80 00 00 00 00 00 00 r0 += r8
>> + 8: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
>> + 10: 61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
>> + 11: 0f 10 00 00 00 00 00 00 r0 += r1
>> + 12: 95 00 00 00 00 00 00 00 exit
>> +
>> + Disassembly of section sec1:
>> +
>> + 0000000000000000 <gfunc>:
>> + 0: bf 20 00 00 00 00 00 00 r0 = r2
>> + 1: 2f 10 00 00 00 00 00 00 r0 *= r1
>> + 2: 95 00 00 00 00 00 00 00 exit
>> +
>> + 0000000000000018 <lfunc>:
>> + 3: bf 20 00 00 00 00 00 00 r0 = r2
>> + 4: 0f 10 00 00 00 00 00 00 r0 += r1
>> + 5: 95 00 00 00 00 00 00 00 exit
>> +
>> +Three relocations are generated with ``llvm-readelf -r test.o``::
>> +
>> + Relocation section '.rel.text' at offset 0x188 contains 3 entries:
>> + Offset Info Type Symbol's Value Symbol's Name
>> + 0000000000000010 000000040000000a R_BPF_64_32 0000000000000000 gfunc
>> + 0000000000000030 000000020000000a R_BPF_64_32 0000000000000000 sec1
>> + 0000000000000040 0000000600000001 R_BPF_64_64 0000000000000000 global
>> +
>> +The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,
>> +so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.
>> +The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section
>> +offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.
>> +
>> +The following is an example to show how R_BPF_64_ABS64 could be generated::
>> +
>> + int global() { return 0; }
>> + struct t { void *g; } gbl = { global };
>> +
>> +Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a
>> +relocation below in ``.data`` section with command
>> +``llvm-readelf -r test.o``::
>> +
>> + Relocation section '.rel.data' at offset 0x458 contains 1 entries:
>> + Offset Info Type Symbol's Value Symbol's Name
>> + 0000000000000000 0000000700000002 R_BPF_64_ABS64 0000000000000000 global
>> +
>> +The relocation says the first 8-byte of ``.data`` section should be
>> +filled with address of ``global`` variable.
>> +
>> +With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of
>> +``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::
>> +
>> + Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:
>> + Offset Info Type Symbol's Value Symbol's Name
>> + 0000000000000006 0000000300000003 R_BPF_64_ABS32 0000000000000000 .debug_abbrev
>> + 000000000000000c 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
>> + 0000000000000012 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
>> + 0000000000000016 0000000600000003 R_BPF_64_ABS32 0000000000000000 .debug_line
>> + 000000000000001a 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
>> + 000000000000001e 0000000200000002 R_BPF_64_ABS64 0000000000000000 .text
>> + 000000000000002b 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
>> + 0000000000000037 0000000800000002 R_BPF_64_ABS64 0000000000000000 gbl
>> + 0000000000000040 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str
>> + ......
>> +
>> +The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
>> +
>> + Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:
>> + Offset Info Type Symbol's Value Symbol's Name
>> + 0000000000000084 0000000800000004 R_BPF_64_NODYLD32 0000000000000000 gbl
>> +
>> + Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:
>> + Offset Info Type Symbol's Value Symbol's Name
>> + 000000000000002c 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
>> + 0000000000000040 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text
>> --
>> 2.30.2
>>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
2021-05-24 18:01 ` Yonghong Song
@ 2021-05-24 19:20 ` Andrii Nakryiko
2021-05-24 19:24 ` John Fastabend
1 sibling, 0 replies; 9+ messages in thread
From: Andrii Nakryiko @ 2021-05-24 19:20 UTC (permalink / raw)
To: Yonghong Song
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Kernel Team, John Fastabend, Lorenz Bauer
On Mon, May 24, 2021 at 11:01 AM Yonghong Song <yhs@fb.com> wrote:
>
>
>
> On 5/24/21 10:23 AM, Andrii Nakryiko wrote:
> > On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
> >>
> >> LLVM upstream commit https://reviews.llvm.org/D102712
> >> made some changes to bpf relocations to make them
> >> llvm linker lld friendly. The scope of
> >> existing relocations R_BPF_64_{64,32} is narrowed
> >> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
> >> are introduced.
> >>
> >> Let us add some documentation about llvm bpf
> >> relocations so people can understand how to resolve
> >> them properly in their respective tools.
> >>
> >> Cc: John Fastabend <john.fastabend@gmail.com>
> >> Cc: Lorenz Bauer <lmb@cloudflare.com>
> >> Signed-off-by: Yonghong Song <yhs@fb.com>
> >> ---
> >> Documentation/bpf/index.rst | 1 +
> >> Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
> >> 2 files changed, 169 insertions(+)
> >> create mode 100644 Documentation/bpf/llvm_reloc.rst
> >>
> >> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
> >> index a702f67dd45f..93e8cf12a6d4 100644
> >> --- a/Documentation/bpf/index.rst
> >> +++ b/Documentation/bpf/index.rst
> >> @@ -84,6 +84,7 @@ Other
> >> :maxdepth: 1
> >>
> >> ringbuf
> >> + llvm_reloc
> >>
> >> .. Links:
> >> .. _networking-filter: ../networking/filter.rst
> >> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
> >> new file mode 100644
> >> index 000000000000..bc62bce591b1
> >> --- /dev/null
> >> +++ b/Documentation/bpf/llvm_reloc.rst
> >> @@ -0,0 +1,168 @@
> >> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> >> +
> >> +====================
> >> +BPF LLVM Relocations
> >> +====================
> >> +
> >> +This document describes LLVM BPF backend relocation types.
> >> +
> >> +Relocation Record
> >> +=================
> >> +
> >> +LLVM BPF backend records each relocation with the following 16-byte
> >> +ELF structure::
> >> +
> >> + typedef struct
> >> + {
> >> + Elf64_Addr r_offset; // Offset from the beginning of section.
> >> + Elf64_Xword r_info; // Relocation type and symbol index.
> >> + } Elf64_Rel;
> >> +
> >> +For static function/variable references, the symbol often refers to
> >> +the section itself which has a value of 0. To identify actual static
> >> +function/variable, its section offset or some computation result
> >> +based on section offset is written to the original insn/data buffer,
> >> +which is called ``IA`` (implicit addend) below. For global
> >> +function/variables, the symbol refers to actual global and the implicit
> >> +addend is 0.
> >> +
> >> +Different Relocation Types
> >> +==========================
> >> +
> >> +Six relocation types are supported. The following is an overview and
> >> +``S`` represents the value of the symbol in the symbol table::
> >> +
> >> + Enum ELF Reloc Type Description BitSize Offset Calculation
> >> + 0 R_BPF_NONE None
> >> + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IA
> >
> > There are cases where we set all 64-bits of ld_imm64 (e.g., extern
> > ksym, global variables). Or those will be a different relocation now
> > (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
>
> It is still R_BPF_64_64. In llvm, we have restriction that section
> offset must be <= UINT32_MAX, and that is why only 32bit is used
> to find the actual symbol in symbol table. 32bit permits 4GB section
> which should enough in practice for a bpf program.
>
> libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.
>
Ok, sounds good.
> The name is a little bit misleading, but it has become part of ABI
> and lives in /usr/include/elf.h and we are not able to change it
> any more.
>
> >
> > Looking at LLVM diff I haven't found a test for global variables (at
> > least I didn't realize it was there), so double-checking here (and it
> > might be a good idea to have an explicit test for global variables?)
>
> We have llvm/test/CodeGen/BPF/reloc.ll and
> llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think
> I can enhance
> llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.
>
Great, thanks.
> >
> >> + 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA
> >> + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA
> >> + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA
> >> + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1
> >> +
> >> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
> >> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
> >> +data bitsize is 32 (4 bytes). The relocation can be resolved with
> >> +the symbol value plus implicit addend.
> >> +
> >> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
> >> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
> >> +bitsize is 64 (8 bytes). The relocation can be resolved with
> >> +the symbol value plus implicit addend.
> >> +
[...]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
2021-05-24 18:01 ` Yonghong Song
2021-05-24 19:20 ` Andrii Nakryiko
@ 2021-05-24 19:24 ` John Fastabend
2021-05-25 3:39 ` Yonghong Song
1 sibling, 1 reply; 9+ messages in thread
From: John Fastabend @ 2021-05-24 19:24 UTC (permalink / raw)
To: Yonghong Song, Andrii Nakryiko
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Kernel Team, John Fastabend, Lorenz Bauer
Yonghong Song wrote:
>
>
> On 5/24/21 10:23 AM, Andrii Nakryiko wrote:
> > On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
> >>
> >> LLVM upstream commit https://reviews.llvm.org/D102712
> >> made some changes to bpf relocations to make them
> >> llvm linker lld friendly. The scope of
> >> existing relocations R_BPF_64_{64,32} is narrowed
> >> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
> >> are introduced.
> >>
> >> Let us add some documentation about llvm bpf
> >> relocations so people can understand how to resolve
> >> them properly in their respective tools.
> >>
> >> Cc: John Fastabend <john.fastabend@gmail.com>
> >> Cc: Lorenz Bauer <lmb@cloudflare.com>
> >> Signed-off-by: Yonghong Song <yhs@fb.com>
> >> ---
> >> Documentation/bpf/index.rst | 1 +
> >> Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
> >> 2 files changed, 169 insertions(+)
> >> create mode 100644 Documentation/bpf/llvm_reloc.rst
> >>
> >> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
> >> index a702f67dd45f..93e8cf12a6d4 100644
> >> --- a/Documentation/bpf/index.rst
> >> +++ b/Documentation/bpf/index.rst
> >> @@ -84,6 +84,7 @@ Other
> >> :maxdepth: 1
> >>
> >> ringbuf
> >> + llvm_reloc
> >>
Thanks Yonghong, I found this helpful. I still had to crack
open llvm code though to follow along. A couple small suggestions
below, may or may not be useful. Overall looks good.
> >> .. Links:
> >> .. _networking-filter: ../networking/filter.rst
> >> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
> >> new file mode 100644
> >> index 000000000000..bc62bce591b1
> >> --- /dev/null
> >> +++ b/Documentation/bpf/llvm_reloc.rst
> >> @@ -0,0 +1,168 @@
> >> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> >> +
> >> +====================
> >> +BPF LLVM Relocations
> >> +====================
> >> +
> >> +This document describes LLVM BPF backend relocation types.
> >> +
> >> +Relocation Record
> >> +=================
> >> +
> >> +LLVM BPF backend records each relocation with the following 16-byte
> >> +ELF structure::
> >> +
> >> + typedef struct
> >> + {
> >> + Elf64_Addr r_offset; // Offset from the beginning of section.
> >> + Elf64_Xword r_info; // Relocation type and symbol index.
> >> + } Elf64_Rel;
> >> +
> >> +For static function/variable references, the symbol often refers to
> >> +the section itself which has a value of 0. To identify actual static
> >> +function/variable, its section offset or some computation result
> >> +based on section offset is written to the original insn/data buffer,
> >> +which is called ``IA`` (implicit addend) below. For global
> >> +function/variables, the symbol refers to actual global and the implicit
> >> +addend is 0.
Above was too terse for me to follow without looking into some clang
examples. Maybe an example right here would help not sure? Maybe expand
the text a bit? I don't have a really good suggestion.
> >> +
> >> +Different Relocation Types
> >> +==========================
> >> +
> >> +Six relocation types are supported. The following is an overview and
> >> +``S`` represents the value of the symbol in the symbol table::
> >> +
> >> + Enum ELF Reloc Type Description BitSize Offset Calculation
> >> + 0 R_BPF_NONE None
> >> + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IA
> >
> > There are cases where we set all 64-bits of ld_imm64 (e.g., extern
> > ksym, global variables). Or those will be a different relocation now
> > (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
>
> It is still R_BPF_64_64. In llvm, we have restriction that section
> offset must be <= UINT32_MAX, and that is why only 32bit is used
> to find the actual symbol in symbol table. 32bit permits 4GB section
> which should enough in practice for a bpf program.
^^^ maybe add this note in the doc somewhere? I had similar questions.
>
> libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.
>
> The name is a little bit misleading, but it has become part of ABI
> and lives in /usr/include/elf.h and we are not able to change it
> any more.
>
> >
> > Looking at LLVM diff I haven't found a test for global variables (at
> > least I didn't realize it was there), so double-checking here (and it
> > might be a good idea to have an explicit test for global variables?)
>
> We have llvm/test/CodeGen/BPF/reloc.ll and
> llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think
> I can enhance
> llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.
^^^ maybe cross-reference llvm tests from kernel docs side? I often look at
these when I get something unexpected/unknown maybe others would find
it helpful, but not know where to look?
>
> >
> >> + 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA
> >> + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA
> >> + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA
> >> + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1
> >> +
> >> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
> >> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
> >> +data bitsize is 32 (4 bytes). The relocation can be resolved with
> >> +the symbol value plus implicit addend.
> >> +
> >> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
> >> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
> >> +bitsize is 64 (8 bytes). The relocation can be resolved with
> >> +the symbol value plus implicit addend.
> >> +
> >> +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
> >> +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
> >> +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
> >> +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
> >> +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
> >> +become unusable by bcc and kernel.
> >> +
> >> +Type ``R_BPF_64_32`` is used for call instruction. The call target section
> >> +offset is stored at ``r_offset + 4`` (32bit) and calculated as
> >> +``(S + IA) / 8 - 1``.
> >> +
> >> +Examples
> >> +========
> >> +
I liked the examples.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
2021-05-24 19:24 ` John Fastabend
@ 2021-05-25 3:39 ` Yonghong Song
0 siblings, 0 replies; 9+ messages in thread
From: Yonghong Song @ 2021-05-25 3:39 UTC (permalink / raw)
To: John Fastabend, Andrii Nakryiko
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Kernel Team, Lorenz Bauer
On 5/24/21 12:24 PM, John Fastabend wrote:
> Yonghong Song wrote:
>>
>>
>> On 5/24/21 10:23 AM, Andrii Nakryiko wrote:
>>> On Sat, May 22, 2021 at 9:39 AM Yonghong Song <yhs@fb.com> wrote:
>>>>
>>>> LLVM upstream commit https://reviews.llvm.org/D102712
>>>> made some changes to bpf relocations to make them
>>>> llvm linker lld friendly. The scope of
>>>> existing relocations R_BPF_64_{64,32} is narrowed
>>>> and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32}
>>>> are introduced.
>>>>
>>>> Let us add some documentation about llvm bpf
>>>> relocations so people can understand how to resolve
>>>> them properly in their respective tools.
>>>>
>>>> Cc: John Fastabend <john.fastabend@gmail.com>
>>>> Cc: Lorenz Bauer <lmb@cloudflare.com>
>>>> Signed-off-by: Yonghong Song <yhs@fb.com>
>>>> ---
>>>> Documentation/bpf/index.rst | 1 +
>>>> Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
>>>> 2 files changed, 169 insertions(+)
>>>> create mode 100644 Documentation/bpf/llvm_reloc.rst
>>>>
>>>> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
>>>> index a702f67dd45f..93e8cf12a6d4 100644
>>>> --- a/Documentation/bpf/index.rst
>>>> +++ b/Documentation/bpf/index.rst
>>>> @@ -84,6 +84,7 @@ Other
>>>> :maxdepth: 1
>>>>
>>>> ringbuf
>>>> + llvm_reloc
>>>>
>
> Thanks Yonghong, I found this helpful. I still had to crack
> open llvm code though to follow along. A couple small suggestions
> below, may or may not be useful. Overall looks good.
>
>>>> .. Links:
>>>> .. _networking-filter: ../networking/filter.rst
>>>> diff --git a/Documentation/bpf/llvm_reloc.rst b/Documentation/bpf/llvm_reloc.rst
>>>> new file mode 100644
>>>> index 000000000000..bc62bce591b1
>>>> --- /dev/null
>>>> +++ b/Documentation/bpf/llvm_reloc.rst
>>>> @@ -0,0 +1,168 @@
>>>> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
>>>> +
>>>> +====================
>>>> +BPF LLVM Relocations
>>>> +====================
>>>> +
>>>> +This document describes LLVM BPF backend relocation types.
>>>> +
>>>> +Relocation Record
>>>> +=================
>>>> +
>>>> +LLVM BPF backend records each relocation with the following 16-byte
>>>> +ELF structure::
>>>> +
>>>> + typedef struct
>>>> + {
>>>> + Elf64_Addr r_offset; // Offset from the beginning of section.
>>>> + Elf64_Xword r_info; // Relocation type and symbol index.
>>>> + } Elf64_Rel;
>>>> +
>>>> +For static function/variable references, the symbol often refers to
>>>> +the section itself which has a value of 0. To identify actual static
>>>> +function/variable, its section offset or some computation result
>>>> +based on section offset is written to the original insn/data buffer,
>>>> +which is called ``IA`` (implicit addend) below. For global
>>>> +function/variables, the symbol refers to actual global and the implicit
>>>> +addend is 0.
>
> Above was too terse for me to follow without looking into some clang
> examples. Maybe an example right here would help not sure? Maybe expand
> the text a bit? I don't have a really good suggestion.
Just send a new revision with an example. Hope it will make it easy to
understand the above ``IA`` concept.
>
>>>> +
>>>> +Different Relocation Types
>>>> +==========================
>>>> +
>>>> +Six relocation types are supported. The following is an overview and
>>>> +``S`` represents the value of the symbol in the symbol table::
>>>> +
>>>> + Enum ELF Reloc Type Description BitSize Offset Calculation
>>>> + 0 R_BPF_NONE None
>>>> + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IA
>>>
>>> There are cases where we set all 64-bits of ld_imm64 (e.g., extern
>>> ksym, global variables). Or those will be a different relocation now
>>> (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
>>
>> It is still R_BPF_64_64. In llvm, we have restriction that section
>> offset must be <= UINT32_MAX, and that is why only 32bit is used
>> to find the actual symbol in symbol table. 32bit permits 4GB section
>> which should enough in practice for a bpf program.
>
> ^^^ maybe add this note in the doc somewhere? I had similar questions.
Added in the new revision.
>
>>
>> libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.
>>
>> The name is a little bit misleading, but it has become part of ABI
>> and lives in /usr/include/elf.h and we are not able to change it
>> any more.
>>
>>>
>>> Looking at LLVM diff I haven't found a test for global variables (at
>>> least I didn't realize it was there), so double-checking here (and it
>>> might be a good idea to have an explicit test for global variables?)
>>
>> We have llvm/test/CodeGen/BPF/reloc.ll and
>> llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think
>> I can enhance
>> llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.
>
> ^^^ maybe cross-reference llvm tests from kernel docs side? I often look at
> these when I get something unexpected/unknown maybe others would find
> it helpful, but not know where to look?
The llvm patch has not merged. We need to merge libbpf patch first.
Otherwise, nightly libbpf CI will fail. But this doc includes a link
to the LLVM patch and you can just go to that llvm patch to find
examples!
>
>>
>>>
>>>> + 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA
>>>> + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA
>>>> + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA
>>>> + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1
>>>> +
>>>> +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
>>>> +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
>>>> +data bitsize is 32 (4 bytes). The relocation can be resolved with
>>>> +the symbol value plus implicit addend.
>>>> +
>>>> +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
>>>> +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
>>>> +bitsize is 64 (8 bytes). The relocation can be resolved with
>>>> +the symbol value plus implicit addend.
>>>> +
>>>> +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
>>>> +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
>>>> +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
>>>> +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
>>>> +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
>>>> +become unusable by bcc and kernel.
>>>> +
>>>> +Type ``R_BPF_64_32`` is used for call instruction. The call target section
>>>> +offset is stored at ``r_offset + 4`` (32bit) and calculated as
>>>> +``(S + IA) / 8 - 1``.
>>>> +
>>>> +Examples
>>>> +========
>>>> +
>
> I liked the examples.
Great. Just added one more in the new revision!
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-05-25 3:40 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-22 16:39 [PATCH bpf-next] docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations Yonghong Song
2021-05-22 16:44 ` Yonghong Song
2021-05-24 8:33 ` Lorenz Bauer
2021-05-24 15:06 ` Yonghong Song
2021-05-24 17:23 ` Andrii Nakryiko
2021-05-24 18:01 ` Yonghong Song
2021-05-24 19:20 ` Andrii Nakryiko
2021-05-24 19:24 ` John Fastabend
2021-05-25 3:39 ` Yonghong Song
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.