[PATCH bpf-next] selftests/bpf: add general instructions for test execution

[PATCH bpf-next] selftests/bpf: add general instructions for test execution

[Alan Maguire]

Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
are used, pahole is recent (>=1.16) and config matches the specified
config file as closely as possible.  Document all of this in the general
README.rst file.  Also note how to work around timeout failures.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst
index 0f67f1b..b00eebb 100644
--- a/tools/testing/selftests/bpf/README.rst
+++ b/tools/testing/selftests/bpf/README.rst
@@ -1,6 +1,52 @@
 ==================
 BPF Selftest Notes
 ==================
+First verify the built kernel config options match the config options
+specified in the config file in this directory.  Test failures for
+unknown helpers, inability to find BTF etc will be observed otherwise.
+
+To ensure the maximum number of tests pass, it is best to use the latest
+trunk LLVM/clang, i.e.
+
+git clone https://github.com/llvm/llvm-project
+
+Build/install trunk LLVM:
+
+.. code-block:: bash
+  git clone https://github.com/llvm/llvm-project
+  cd llvm-project
+  mkdir build/llvm
+  cd build/llvm
+  cmake ../../llvm/
+  make
+  sudo make install
+  cd ../../
+
+Build/install trunk clang:
+
+.. code-block:: bash
+  mkdir -p build/clang
+  cd build/clang
+  cmake ../../clang
+  make
+  sudo make install
+
+When building the kernel with CONFIG_DEBUG_INFO_BTF, pahole
+version 16 or later is also required for BTF function
+support. pahole can be built from the source at
+
+https://github.com/acmel/dwarves
+
+It is often available in "dwarves/libdwarves" packages also,
+but be aware that versions prior to 1.16 will fail with
+errors that functions cannot be found in BTF.
+
+When running selftests, the default timeout of 45 seconds
+can be exceeded by some tests.  We can override the default
+timeout via a "settings" file; for example:
+
+.. code-block:: bash
+  echo "timeout=120" > tools/testing/selftests/bpf/settings
 
 Additional information about selftest failures are
 documented here.
-- 
1.8.3.1

Re: [PATCH bpf-next] selftests/bpf: add general instructions for test execution

[Andrii Nakryiko]

On Mon, May 18, 2020 at 4:24 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
> are used, pahole is recent (>=1.16) and config matches the specified
> config file as closely as possible.  Document all of this in the general
> README.rst file.  Also note how to work around timeout failures.
>
> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> ---

Awesome, thanks.

Acked-by: Andrii Nakryiko <andriin@fb.com>

>  tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
>

[...]

Re: [PATCH bpf-next] selftests/bpf: add general instructions for test execution

[Alexei Starovoitov]

On Mon, May 18, 2020 at 12:23:10PM +0100, Alan Maguire wrote:
> Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
> are used, pahole is recent (>=1.16) and config matches the specified
> config file as closely as possible.  Document all of this in the general
> README.rst file.  Also note how to work around timeout failures.
> 
> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> ---
>  tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
> 
> diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst
> index 0f67f1b..b00eebb 100644
> --- a/tools/testing/selftests/bpf/README.rst
> +++ b/tools/testing/selftests/bpf/README.rst
> @@ -1,6 +1,52 @@
>  ==================
>  BPF Selftest Notes
>  ==================
> +First verify the built kernel config options match the config options
> +specified in the config file in this directory.  Test failures for
> +unknown helpers, inability to find BTF etc will be observed otherwise.
> +
> +To ensure the maximum number of tests pass, it is best to use the latest
> +trunk LLVM/clang, i.e.
> +
> +git clone https://github.com/llvm/llvm-project
> +
> +Build/install trunk LLVM:
> +
> +.. code-block:: bash
> +  git clone https://github.com/llvm/llvm-project
> +  cd llvm-project
> +  mkdir build/llvm
> +  cd build/llvm
> +  cmake ../../llvm/
> +  make
> +  sudo make install
> +  cd ../../
> +
> +Build/install trunk clang:
> +
> +.. code-block:: bash
> +  mkdir -p build/clang
> +  cd build/clang
> +  cmake ../../clang
> +  make
> +  sudo make install
> +

these instructions are obsolete and partially incorrect.
May be refer to Documentation/bpf/bpf_devel_QA.rst instead?

> +When building the kernel with CONFIG_DEBUG_INFO_BTF, pahole
> +version 16 or later is also required for BTF function
> +support. pahole can be built from the source at
> +
> +https://github.com/acmel/dwarves
> +
> +It is often available in "dwarves/libdwarves" packages also,
> +but be aware that versions prior to 1.16 will fail with
> +errors that functions cannot be found in BTF.
> +
> +When running selftests, the default timeout of 45 seconds
> +can be exceeded by some tests.  We can override the default
> +timeout via a "settings" file; for example:
> +
> +.. code-block:: bash
> +  echo "timeout=120" > tools/testing/selftests/bpf/settings

Is it really the case?
I've never seen anything like this.

Re: [PATCH bpf-next] selftests/bpf: add general instructions for test execution

[Alan Maguire]

On Tue, 19 May 2020, Alexei Starovoitov wrote:

> On Mon, May 18, 2020 at 12:23:10PM +0100, Alan Maguire wrote:
> > Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
> > are used, pahole is recent (>=1.16) and config matches the specified
> > config file as closely as possible.  Document all of this in the general
> > README.rst file.  Also note how to work around timeout failures.
> > 
> > Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> > ---
> >  tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
> >  1 file changed, 46 insertions(+)
> > 
> > diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst
> > index 0f67f1b..b00eebb 100644
> > --- a/tools/testing/selftests/bpf/README.rst
> > +++ b/tools/testing/selftests/bpf/README.rst
> > @@ -1,6 +1,52 @@
> >  ==================
> >  BPF Selftest Notes
> >  ==================
> > +First verify the built kernel config options match the config options
> > +specified in the config file in this directory.  Test failures for
> > +unknown helpers, inability to find BTF etc will be observed otherwise.
> > +
> > +To ensure the maximum number of tests pass, it is best to use the latest
> > +trunk LLVM/clang, i.e.
> > +
> > +git clone https://github.com/llvm/llvm-project
> > +
> > +Build/install trunk LLVM:
> > +
> > +.. code-block:: bash
> > +  git clone https://github.com/llvm/llvm-project
> > +  cd llvm-project
> > +  mkdir build/llvm
> > +  cd build/llvm
> > +  cmake ../../llvm/
> > +  make
> > +  sudo make install
> > +  cd ../../
> > +
> > +Build/install trunk clang:
> > +
> > +.. code-block:: bash
> > +  mkdir -p build/clang
> > +  cd build/clang
> > +  cmake ../../clang
> > +  make
> > +  sudo make install
> > +
> 
> these instructions are obsolete and partially incorrect.
> May be refer to Documentation/bpf/bpf_devel_QA.rst instead?
>

Sure; looks like there are up-to-date sections there on
running BPF selftests and building LLVM manually.  Perhaps
I should add the notes about pahole etc there too?
I should also have noted that without an up-to-date iproute2
failures will be observed also.
  
> > +When building the kernel with CONFIG_DEBUG_INFO_BTF, pahole
> > +version 16 or later is also required for BTF function
> > +support. pahole can be built from the source at
> > +
> > +https://github.com/acmel/dwarves
> > +
> > +It is often available in "dwarves/libdwarves" packages also,
> > +but be aware that versions prior to 1.16 will fail with
> > +errors that functions cannot be found in BTF.
> > +
> > +When running selftests, the default timeout of 45 seconds
> > +can be exceeded by some tests.  We can override the default
> > +timeout via a "settings" file; for example:
> > +
> > +.. code-block:: bash
> > +  echo "timeout=120" > tools/testing/selftests/bpf/settings
> 
> Is it really the case?
> I've never seen anything like this.
> 

When running via "make run_tests" on baremetal systems I
see test timeouts pretty consistently; e.g. from a bpf tree test
run yesterday:

not ok 6 selftests: bpf: test_progs # TIMEOUT
not ok 31 selftests: bpf: test_tunnel.sh # TIMEOUT
not ok 38 selftests: bpf: test_lwt_ip_encap.sh # TIMEOUT
not ok 40 selftests: bpf: test_tc_tunnel.sh # TIMEOUT
not ok 42 selftests: bpf: test_xdping.sh # TIMEOUT
not ok 43 selftests: bpf: test_bpftool_build.sh # TIMEOUT

These will only occur if running via "make run_tests",
so running tests individually would not trigger these
failures.

Alan

Re: [PATCH bpf-next] selftests/bpf: add general instructions for test execution

[Alexei Starovoitov]

On Tue, May 19, 2020 at 10:34:25PM +0100, Alan Maguire wrote:
> On Tue, 19 May 2020, Alexei Starovoitov wrote:
> 
> > On Mon, May 18, 2020 at 12:23:10PM +0100, Alan Maguire wrote:
> > > Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
> > > are used, pahole is recent (>=1.16) and config matches the specified
> > > config file as closely as possible.  Document all of this in the general
> > > README.rst file.  Also note how to work around timeout failures.
> > > 
> > > Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> > > ---
> > >  tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
> > >  1 file changed, 46 insertions(+)
> > > 
> > > diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst
> > > index 0f67f1b..b00eebb 100644
> > > --- a/tools/testing/selftests/bpf/README.rst
> > > +++ b/tools/testing/selftests/bpf/README.rst
> > > @@ -1,6 +1,52 @@
> > >  ==================
> > >  BPF Selftest Notes
> > >  ==================
> > > +First verify the built kernel config options match the config options
> > > +specified in the config file in this directory.  Test failures for
> > > +unknown helpers, inability to find BTF etc will be observed otherwise.
> > > +
> > > +To ensure the maximum number of tests pass, it is best to use the latest
> > > +trunk LLVM/clang, i.e.
> > > +
> > > +git clone https://github.com/llvm/llvm-project
> > > +
> > > +Build/install trunk LLVM:
> > > +
> > > +.. code-block:: bash
> > > +  git clone https://github.com/llvm/llvm-project
> > > +  cd llvm-project
> > > +  mkdir build/llvm
> > > +  cd build/llvm
> > > +  cmake ../../llvm/
> > > +  make
> > > +  sudo make install
> > > +  cd ../../
> > > +
> > > +Build/install trunk clang:
> > > +
> > > +.. code-block:: bash
> > > +  mkdir -p build/clang
> > > +  cd build/clang
> > > +  cmake ../../clang
> > > +  make
> > > +  sudo make install
> > > +
> > 
> > these instructions are obsolete and partially incorrect.
> > May be refer to Documentation/bpf/bpf_devel_QA.rst instead?
> >
> 
> Sure; looks like there are up-to-date sections there on
> running BPF selftests and building LLVM manually.  Perhaps
> I should add the notes about pahole etc there too?

yes. please.
Could you mention distros that have fresh pahole?
Otherwise users will have an impression that pahole needs
to be build from scratch as well. Which is not the case.

> I should also have noted that without an up-to-date iproute2
> failures will be observed also.

Would be good to highlight which tests will fail with old iproute2.
I'm not sure which version is necessary.
What is 'up-to-date' ?
The tests I run before applying work for me and I rebuild iproute2
every year or so :)

> > > +When building the kernel with CONFIG_DEBUG_INFO_BTF, pahole
> > > +version 16 or later is also required for BTF function
> > > +support. pahole can be built from the source at
> > > +
> > > +https://github.com/acmel/dwarves
> > > +
> > > +It is often available in "dwarves/libdwarves" packages also,
> > > +but be aware that versions prior to 1.16 will fail with
> > > +errors that functions cannot be found in BTF.
> > > +
> > > +When running selftests, the default timeout of 45 seconds
> > > +can be exceeded by some tests.  We can override the default
> > > +timeout via a "settings" file; for example:
> > > +
> > > +.. code-block:: bash
> > > +  echo "timeout=120" > tools/testing/selftests/bpf/settings
> > 
> > Is it really the case?
> > I've never seen anything like this.
> > 
> 
> When running via "make run_tests" on baremetal systems I
> see test timeouts pretty consistently; e.g. from a bpf tree test
> run yesterday:
> 
> not ok 6 selftests: bpf: test_progs # TIMEOUT
> not ok 31 selftests: bpf: test_tunnel.sh # TIMEOUT
> not ok 38 selftests: bpf: test_lwt_ip_encap.sh # TIMEOUT
> not ok 40 selftests: bpf: test_tc_tunnel.sh # TIMEOUT
> not ok 42 selftests: bpf: test_xdping.sh # TIMEOUT
> not ok 43 selftests: bpf: test_bpftool_build.sh # TIMEOUT
> 
> These will only occur if running via "make run_tests",
> so running tests individually would not trigger these
> failures.

If timeout is necessary it's better to fix it in the git
instead of requiring users to tweak their environment.