bpf.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Song Liu <songliubraving@fb.com>
To: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Song Liu <liu.song.a23@gmail.com>,
	Andrii Nakryiko <andriin@fb.com>, bpf <bpf@vger.kernel.org>,
	Networking <netdev@vger.kernel.org>,
	"Alexei Starovoitov" <ast@fb.com>,
	Daniel Borkmann <daniel@iogearbox.net>,
	"Yonghong Song" <yhs@fb.com>, Kernel Team <Kernel-team@fb.com>
Subject: Re: [PATCH bpf-next 00/10] CO-RE offset relocations
Date: Tue, 30 Jul 2019 05:27:55 +0000	[thread overview]
Message-ID: <FCAF9AD8-C138-4AA7-AD68-195B01C6A25C@fb.com> (raw)
In-Reply-To: <CAEf4BzbKYi53TdF9nAB3i3gAuca8FjM_P3F5aHp1uQ6coMgZ9A@mail.gmail.com>



> On Jul 29, 2019, at 4:09 PM, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> 
> On Mon, Jul 29, 2019 at 1:37 PM Song Liu <liu.song.a23@gmail.com> wrote:
>> 
>> On Mon, Jul 29, 2019 at 1:20 PM Song Liu <liu.song.a23@gmail.com> wrote:
>>> 
>>> On Wed, Jul 24, 2019 at 1:34 PM Andrii Nakryiko <andriin@fb.com> wrote:
>>>> 
>>>> This patch set implements central part of CO-RE (Compile Once - Run
>>>> Everywhere, see [0] and [1] for slides and video): relocating field offsets.
>>>> Most of the details are written down as comments to corresponding parts of the
>>>> code.
>>>> 
>>>> Patch #1 adds loading of .BTF.ext offset relocations section and macros to
>>>> work with its contents.
>>>> Patch #2 implements CO-RE relocations algorithm in libbpf.
>>>> Patches #3-#10 adds selftests validating various parts of relocation handling,
>>>> type compatibility, etc.
>>>> 
>>>> For all tests to work, you'll need latest Clang/LLVM supporting
>>>> __builtin_preserve_access_index intrinsic, used for recording offset
>>>> relocations. Kernel on which selftests run should have BTF information built
>>>> in (CONFIG_DEBUG_INFO_BTF=y).
>>>> 
>>>>  [0] http://vger.kernel.org/bpfconf2019.html#session-2
>>>>  [1] http://vger.kernel.org/lpc-bpf2018.html#session-2CO-RE relocations
>>>> 
>>>> This patch set implements central part of CO-RE (Compile Once - Run
>>>> Everywhere, see [0] and [1] for slides and video): relocating field offsets.
>>>> Most of the details are written down as comments to corresponding parts of the
>>>> code.
>>>> 
>>>> Patch #1 adds loading of .BTF.ext offset relocations section and macros to
>>>> work with its contents.
>>>> Patch #2 implements CO-RE relocations algorithm in libbpf.
>>>> Patches #3-#10 adds selftests validating various parts of relocation handling,
>>>> type compatibility, etc.
>>>> 
>>>> For all tests to work, you'll need latest Clang/LLVM supporting
>>>> __builtin_preserve_access_index intrinsic, used for recording offset
>>>> relocations. Kernel on which selftests run should have BTF information built
>>>> in (CONFIG_DEBUG_INFO_BTF=y).
>>>> 
>>>>  [0] http://vger.kernel.org/bpfconf2019.html#session-2
>>>>  [1] http://vger.kernel.org/lpc-bpf2018.html#session-2
>>>> 
>>>> Andrii Nakryiko (10):
>>>>  libbpf: add .BTF.ext offset relocation section loading
>>>>  libbpf: implement BPF CO-RE offset relocation algorithm
>>>>  selftests/bpf: add CO-RE relocs testing setup
>>>>  selftests/bpf: add CO-RE relocs struct flavors tests
>>>>  selftests/bpf: add CO-RE relocs nesting tests
>>>>  selftests/bpf: add CO-RE relocs array tests
>>>>  selftests/bpf: add CO-RE relocs enum/ptr/func_proto tests
>>>>  selftests/bpf: add CO-RE relocs modifiers/typedef tests
>>>>  selftest/bpf: add CO-RE relocs ptr-as-array tests
>>>>  selftests/bpf: add CO-RE relocs ints tests
>>>> 
>>>> tools/lib/bpf/btf.c                           |  64 +-
>>>> tools/lib/bpf/btf.h                           |   4 +
>>>> tools/lib/bpf/libbpf.c                        | 866 +++++++++++++++++-
>>>> tools/lib/bpf/libbpf.h                        |   1 +
>>>> tools/lib/bpf/libbpf_internal.h               |  91 ++
>>>> .../selftests/bpf/prog_tests/core_reloc.c     | 363 ++++++++
>>>> .../bpf/progs/btf__core_reloc_arrays.c        |   3 +
>>>> .../btf__core_reloc_arrays___diff_arr_dim.c   |   3 +
>>>> ...btf__core_reloc_arrays___diff_arr_val_sz.c |   3 +
>>>> .../btf__core_reloc_arrays___err_non_array.c  |   3 +
>>>> ...btf__core_reloc_arrays___err_too_shallow.c |   3 +
>>>> .../btf__core_reloc_arrays___err_too_small.c  |   3 +
>>>> ..._core_reloc_arrays___err_wrong_val_type1.c |   3 +
>>>> ..._core_reloc_arrays___err_wrong_val_type2.c |   3 +
>>>> .../bpf/progs/btf__core_reloc_flavors.c       |   3 +
>>>> .../btf__core_reloc_flavors__err_wrong_name.c |   3 +
>>>> .../bpf/progs/btf__core_reloc_ints.c          |   3 +
>>>> .../bpf/progs/btf__core_reloc_ints___bool.c   |   3 +
>>>> .../btf__core_reloc_ints___err_bitfield.c     |   3 +
>>>> .../btf__core_reloc_ints___err_wrong_sz_16.c  |   3 +
>>>> .../btf__core_reloc_ints___err_wrong_sz_32.c  |   3 +
>>>> .../btf__core_reloc_ints___err_wrong_sz_64.c  |   3 +
>>>> .../btf__core_reloc_ints___err_wrong_sz_8.c   |   3 +
>>>> .../btf__core_reloc_ints___reverse_sign.c     |   3 +
>>>> .../bpf/progs/btf__core_reloc_mods.c          |   3 +
>>>> .../progs/btf__core_reloc_mods___mod_swap.c   |   3 +
>>>> .../progs/btf__core_reloc_mods___typedefs.c   |   3 +
>>>> .../bpf/progs/btf__core_reloc_nesting.c       |   3 +
>>>> .../btf__core_reloc_nesting___anon_embed.c    |   3 +
>>>> ...f__core_reloc_nesting___dup_compat_types.c |   5 +
>>>> ...core_reloc_nesting___err_array_container.c |   3 +
>>>> ...tf__core_reloc_nesting___err_array_field.c |   3 +
>>>> ...e_reloc_nesting___err_dup_incompat_types.c |   4 +
>>>> ...re_reloc_nesting___err_missing_container.c |   3 +
>>>> ...__core_reloc_nesting___err_missing_field.c |   3 +
>>>> ..._reloc_nesting___err_nonstruct_container.c |   3 +
>>>> ...e_reloc_nesting___err_partial_match_dups.c |   4 +
>>>> .../btf__core_reloc_nesting___err_too_deep.c  |   3 +
>>>> .../btf__core_reloc_nesting___extra_nesting.c |   3 +
>>>> ..._core_reloc_nesting___struct_union_mixup.c |   3 +
>>>> .../bpf/progs/btf__core_reloc_primitives.c    |   3 +
>>>> ...f__core_reloc_primitives___diff_enum_def.c |   3 +
>>>> ..._core_reloc_primitives___diff_func_proto.c |   3 +
>>>> ...f__core_reloc_primitives___diff_ptr_type.c |   3 +
>>>> ...tf__core_reloc_primitives___err_non_enum.c |   3 +
>>>> ...btf__core_reloc_primitives___err_non_int.c |   3 +
>>>> ...btf__core_reloc_primitives___err_non_ptr.c |   3 +
>>>> .../bpf/progs/btf__core_reloc_ptr_as_arr.c    |   3 +
>>>> .../btf__core_reloc_ptr_as_arr___diff_sz.c    |   3 +
>>>> .../selftests/bpf/progs/core_reloc_types.h    | 642 +++++++++++++
>>>> .../bpf/progs/test_core_reloc_arrays.c        |  58 ++
>>>> .../bpf/progs/test_core_reloc_flavors.c       |  65 ++
>>>> .../bpf/progs/test_core_reloc_ints.c          |  48 +
>>>> .../bpf/progs/test_core_reloc_kernel.c        |  39 +
>>>> .../bpf/progs/test_core_reloc_mods.c          |  68 ++
>>>> .../bpf/progs/test_core_reloc_nesting.c       |  48 +
>>>> .../bpf/progs/test_core_reloc_primitives.c    |  50 +
>>>> .../bpf/progs/test_core_reloc_ptr_as_arr.c    |  34 +
>>>> 58 files changed, 2527 insertions(+), 47 deletions(-)
>>>> create mode 100644 tools/testing/selftests/bpf/prog_tests/core_reloc.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_dim.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_val_sz.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_non_array.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_shallow.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_small.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type1.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type2.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors__err_wrong_name.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___bool.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_bitfield.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_16.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_32.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_64.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_8.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___reverse_sign.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___mod_swap.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___typedefs.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___anon_embed.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___dup_compat_types.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_container.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_field.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_dup_incompat_types.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_container.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_field.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_nonstruct_container.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_partial_match_dups.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_too_deep.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___extra_nesting.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___struct_union_mixup.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_enum_def.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_func_proto.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_ptr_type.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_enum.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_int.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_ptr.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr___diff_sz.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/core_reloc_types.h
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_arrays.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_flavors.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ints.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_kernel.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_mods.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_nesting.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_primitives.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ptr_as_arr.c
>>> 
>>> We have created a lot of small files. Would it be cleaner if we can
>>> somehow put these
>>> data in one file (maybe different sections?).
>> 
>> After reading more, I guess you have tried this and end up with current
>> design: keep most struct defines in core_reloc_types.h.
> 
> Yeah, I have all the definition in one header file, but then I need
> individual combinations as separate BTFs, so I essentially "pick"
> desired types using function declarations. Creating those BTFs by hand
> would be a nightmare to create and maintain.
> 
>> 
>>> 
>>> Alternatively, maybe create a folder for these files:
>>>  tools/testing/selftests/bpf/progs/core/
>> 
>> I guess this would still make it cleaner.
> 
> There is nothing too special about core tests to split them. Also it
> would require Makefile changes and would deviate test_progs
> definitions from analogous test_maps, test_verifier, test_btf, etc, so
> I'm not sure about that. I though about putting those btf__* files
> under separate directory, but I'm on the fence there as well, as I'd
> rather have related files to stay together...
> 

I guess we can defer this decision to later. So I am OK with current
approach for now. 

Thanks,
Song


      reply	other threads:[~2019-07-30  5:28 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-24 19:27 [PATCH bpf-next 00/10] CO-RE offset relocations Andrii Nakryiko
2019-07-24 19:27 ` [PATCH bpf-next 01/10] libbpf: add .BTF.ext offset relocation section loading Andrii Nakryiko
2019-07-24 21:42   ` Andrii Nakryiko
2019-07-25  0:00   ` Song Liu
2019-07-25  0:37     ` Andrii Nakryiko
2019-07-25  5:20       ` Song Liu
2019-07-27  5:11         ` Andrii Nakryiko
2019-07-29 20:00           ` Song Liu
2019-07-24 19:27 ` [PATCH bpf-next 02/10] libbpf: implement BPF CO-RE offset relocation algorithm Andrii Nakryiko
2019-07-25 19:32   ` Song Liu
2019-07-27  6:11     ` Andrii Nakryiko
2019-07-27 18:59       ` Song Liu
2019-07-27 19:09         ` Andrii Nakryiko
2019-07-28  0:24           ` Song Liu
2019-07-25 23:18   ` Alexei Starovoitov
2019-07-27  6:25     ` Andrii Nakryiko
2019-07-27 17:00       ` Alexei Starovoitov
2019-07-27 18:24         ` Andrii Nakryiko
2019-07-27 21:29           ` Yonghong Song
2019-07-27 21:36             ` Andrii Nakryiko
2019-07-29 19:56   ` Song Liu
2019-07-24 19:27 ` [PATCH bpf-next 03/10] selftests/bpf: add CO-RE relocs testing setup Andrii Nakryiko
2019-07-29 20:22   ` Song Liu
2019-07-24 19:27 ` [PATCH bpf-next 04/10] selftests/bpf: add CO-RE relocs struct flavors tests Andrii Nakryiko
2019-07-29 20:37   ` Song Liu
2019-07-24 19:27 ` [PATCH bpf-next 05/10] selftests/bpf: add CO-RE relocs nesting tests Andrii Nakryiko
2019-07-29 21:06   ` Song Liu
2019-07-24 19:27 ` [PATCH bpf-next 06/10] selftests/bpf: add CO-RE relocs array tests Andrii Nakryiko
2019-07-25 23:26   ` Alexei Starovoitov
2019-07-26 23:29     ` Andrii Nakryiko
2019-07-24 19:27 ` [PATCH bpf-next 07/10] selftests/bpf: add CO-RE relocs enum/ptr/func_proto tests Andrii Nakryiko
2019-07-29 21:09   ` Song Liu
2019-07-24 19:27 ` [PATCH bpf-next 08/10] selftests/bpf: add CO-RE relocs modifiers/typedef tests Andrii Nakryiko
2019-07-29 21:11   ` Song Liu
2019-07-24 19:27 ` [PATCH bpf-next 09/10] selftest/bpf: add CO-RE relocs ptr-as-array tests Andrii Nakryiko
2019-07-29 21:14   ` Song Liu
2019-07-24 19:27 ` [PATCH bpf-next 10/10] selftests/bpf: add CO-RE relocs ints tests Andrii Nakryiko
2019-07-29 21:21   ` Song Liu
2019-07-29 20:20 ` [PATCH bpf-next 00/10] CO-RE offset relocations Song Liu
2019-07-29 20:36   ` Song Liu
2019-07-29 23:09     ` Andrii Nakryiko
2019-07-30  5:27       ` Song Liu [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=FCAF9AD8-C138-4AA7-AD68-195B01C6A25C@fb.com \
    --to=songliubraving@fb.com \
    --cc=Kernel-team@fb.com \
    --cc=andrii.nakryiko@gmail.com \
    --cc=andriin@fb.com \
    --cc=ast@fb.com \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=liu.song.a23@gmail.com \
    --cc=netdev@vger.kernel.org \
    --cc=yhs@fb.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).