Re: [PATCH v4 bpf-next 5/7] selftests/bpf: replace test_progs and test_maps w/ general rule

[Andrii Nakryiko <andrii.nakryiko@gmail.com>]

On Thu, Oct 17, 2019 at 10:54 AM Alexei Starovoitov <ast@fb.com> wrote:
>
> On 10/17/19 10:50 AM, Andrii Nakryiko wrote:
> > On Tue, Oct 15, 2019 at 11:01 PM Andrii Nakryiko <andriin@fb.com> wrote:
> >>
> >> Define test runner generation meta-rule that codifies dependencies
> >> between test runner, its tests, and its dependent BPF programs. Use that
> >> for defining test_progs and test_maps test-runners. Also additionally define
> >> 2 flavors of test_progs:
> >> - alu32, which builds BPF programs with 32-bit registers codegen;
> >> - bpf_gcc, which build BPF programs using GCC, if it supports BPF target.
> >>
> >> Overall, this is accomplished through $(eval)'ing a set of generic
> >> rules, which defines Makefile targets dynamically at runtime. See
> >> comments explaining the need for 2 $(evals), though.
> >>
> >> For each test runner we have (test_maps and test_progs, currently), and,
> >> optionally, their flavors, the logic of build process is modeled as
> >> follows (using test_progs as an example):
> >> - all BPF objects are in progs/:
> >>    - BPF object's .o file is built into output directory from
> >>      corresponding progs/.c file;
> >>    - all BPF objects in progs/*.c depend on all progs/*.h headers;
> >>    - all BPF objects depend on bpf_*.h helpers from libbpf (but not
> >>      libbpf archive). There is an extra rule to trigger bpf_helper_defs.h
> >>      (re-)build, if it's not present/outdated);
> >>    - build recipe for BPF object can be re-defined per test runner/flavor;
> >> - test files are built from prog_tests/*.c:
> >>    - all such test file objects are built on individual file basis;
> >>    - currently, every single test file depends on all BPF object files;
> >>      this might be improved in follow up patches to do 1-to-1 dependency,
> >>      but allowing to customize this per each individual test;
> >>    - each test runner definition can specify a list of extra .c and .h
> >>      files to be built along test files and test runner binary; all such
> >>      headers are becoming automatic dependency of each test .c file;
> >>    - due to test files sometimes embedding (using .incbin assembly
> >>      directive) contents of some BPF objects at compilation time, which are
> >>      expected to be in CWD of compiler, compilation for test file object does
> >>      cd into test runner's output directory; to support this mode all the
> >>      include paths are turned into absolute paths using $(abspath) make
> >>      function;
> >> - prog_tests/test.h is automatically (re-)generated with an entry for
> >>    each .c file in prog_tests/;
> >> - final test runner binary is linked together from test object files and
> >>    extra object files, linking together libbpf's archive as well;
> >> - it's possible to specify extra "resource" files/targets, which will be
> >>    copied into test runner output directory, if it differes from
> >>    Makefile-wide $(OUTPUT). This is used to ensure btf_dump test cases and
> >>    urandom_read binary is put into a test runner's CWD for tests to find
> >>    them in runtime.
> >>
> >> For flavored test runners, their output directory is a subdirectory of
> >> common Makefile-wide $(OUTPUT) directory with flavor name used as
> >> subdirectory name.
> >>
> >> BPF objects targets might be reused between different test runners, so
> >> extra checks are employed to not double-define them. Similarly, we have
> >> redefinition guards for output directories and test headers.
> >>
> >> test_verifier follows slightly different patterns and is simple enough
> >> to not justify generalizing TEST_RUNNER_DEFINE/TEST_RUNNER_DEFINE_RULES
> >> further to accomodate these differences. Instead, rules for
> >> test_verifier are minimized and simplified, while preserving correctness
> >> of dependencies.
> >>
> >> Signed-off-by: Andrii Nakryiko <andriin@fb.com>
> >> ---
> >
> > BTW, if correctness and DRY-ness argument is not strong enough, these
> > changes makes clean rebuild from scratch about 2x faster for me:
> >
> > BEFORE: `make clean && time make -j50` is 14-15 seconds
> > AFTER: `make clean && time make -j50` is 7-8 seconds
>
> I noticed that too and was about to ask "why?" .. :)

alu32 BPF .o's were dependent on alu32/test_progs for some reason, so
they blocked on all tests be built first, which is completely
backwards and slower. Now all the flavors are built completely in
parallel. Overall CPU usage across all cores increased (because we do
more work compiling binaries), but it's more parallel.