Linux-kselftest Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH bpf-next] selftests/bpf: add general instructions for test execution
@ 2020-05-18 11:23 Alan Maguire
  2020-05-18 23:48 ` Andrii Nakryiko
  2020-05-19 15:50 ` Alexei Starovoitov
  0 siblings, 2 replies; 5+ messages in thread
From: Alan Maguire @ 2020-05-18 11:23 UTC (permalink / raw)
  To: shuah, ast, daniel, andriin, bpf
  Cc: kafai, songliubraving, yhs, john.fastabend, kpsingh,
	linux-kselftest, netdev, Alan Maguire

Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
are used, pahole is recent (>=1.16) and config matches the specified
config file as closely as possible.  Document all of this in the general
README.rst file.  Also note how to work around timeout failures.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst
index 0f67f1b..b00eebb 100644
--- a/tools/testing/selftests/bpf/README.rst
+++ b/tools/testing/selftests/bpf/README.rst
@@ -1,6 +1,52 @@
 ==================
 BPF Selftest Notes
 ==================
+First verify the built kernel config options match the config options
+specified in the config file in this directory.  Test failures for
+unknown helpers, inability to find BTF etc will be observed otherwise.
+
+To ensure the maximum number of tests pass, it is best to use the latest
+trunk LLVM/clang, i.e.
+
+git clone https://github.com/llvm/llvm-project
+
+Build/install trunk LLVM:
+
+.. code-block:: bash
+  git clone https://github.com/llvm/llvm-project
+  cd llvm-project
+  mkdir build/llvm
+  cd build/llvm
+  cmake ../../llvm/
+  make
+  sudo make install
+  cd ../../
+
+Build/install trunk clang:
+
+.. code-block:: bash
+  mkdir -p build/clang
+  cd build/clang
+  cmake ../../clang
+  make
+  sudo make install
+
+When building the kernel with CONFIG_DEBUG_INFO_BTF, pahole
+version 16 or later is also required for BTF function
+support. pahole can be built from the source at
+
+https://github.com/acmel/dwarves
+
+It is often available in "dwarves/libdwarves" packages also,
+but be aware that versions prior to 1.16 will fail with
+errors that functions cannot be found in BTF.
+
+When running selftests, the default timeout of 45 seconds
+can be exceeded by some tests.  We can override the default
+timeout via a "settings" file; for example:
+
+.. code-block:: bash
+  echo "timeout=120" > tools/testing/selftests/bpf/settings
 
 Additional information about selftest failures are
 documented here.
-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH bpf-next] selftests/bpf: add general instructions for test execution
  2020-05-18 11:23 [PATCH bpf-next] selftests/bpf: add general instructions for test execution Alan Maguire
@ 2020-05-18 23:48 ` Andrii Nakryiko
  2020-05-19 15:50 ` Alexei Starovoitov
  1 sibling, 0 replies; 5+ messages in thread
From: Andrii Nakryiko @ 2020-05-18 23:48 UTC (permalink / raw)
  To: Alan Maguire
  Cc: Shuah Khan, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	bpf, Martin Lau, Song Liu, Yonghong Song, john fastabend,
	KP Singh, open list:KERNEL SELFTEST FRAMEWORK, Networking

On Mon, May 18, 2020 at 4:24 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
> are used, pahole is recent (>=1.16) and config matches the specified
> config file as closely as possible.  Document all of this in the general
> README.rst file.  Also note how to work around timeout failures.
>
> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> ---

Awesome, thanks.

Acked-by: Andrii Nakryiko <andriin@fb.com>

>  tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
>

[...]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH bpf-next] selftests/bpf: add general instructions for test execution
  2020-05-18 11:23 [PATCH bpf-next] selftests/bpf: add general instructions for test execution Alan Maguire
  2020-05-18 23:48 ` Andrii Nakryiko
@ 2020-05-19 15:50 ` Alexei Starovoitov
  2020-05-19 21:34   ` Alan Maguire
  1 sibling, 1 reply; 5+ messages in thread
From: Alexei Starovoitov @ 2020-05-19 15:50 UTC (permalink / raw)
  To: Alan Maguire
  Cc: shuah, ast, daniel, andriin, bpf, kafai, songliubraving, yhs,
	john.fastabend, kpsingh, linux-kselftest, netdev

On Mon, May 18, 2020 at 12:23:10PM +0100, Alan Maguire wrote:
> Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
> are used, pahole is recent (>=1.16) and config matches the specified
> config file as closely as possible.  Document all of this in the general
> README.rst file.  Also note how to work around timeout failures.
> 
> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> ---
>  tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
> 
> diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst
> index 0f67f1b..b00eebb 100644
> --- a/tools/testing/selftests/bpf/README.rst
> +++ b/tools/testing/selftests/bpf/README.rst
> @@ -1,6 +1,52 @@
>  ==================
>  BPF Selftest Notes
>  ==================
> +First verify the built kernel config options match the config options
> +specified in the config file in this directory.  Test failures for
> +unknown helpers, inability to find BTF etc will be observed otherwise.
> +
> +To ensure the maximum number of tests pass, it is best to use the latest
> +trunk LLVM/clang, i.e.
> +
> +git clone https://github.com/llvm/llvm-project
> +
> +Build/install trunk LLVM:
> +
> +.. code-block:: bash
> +  git clone https://github.com/llvm/llvm-project
> +  cd llvm-project
> +  mkdir build/llvm
> +  cd build/llvm
> +  cmake ../../llvm/
> +  make
> +  sudo make install
> +  cd ../../
> +
> +Build/install trunk clang:
> +
> +.. code-block:: bash
> +  mkdir -p build/clang
> +  cd build/clang
> +  cmake ../../clang
> +  make
> +  sudo make install
> +

these instructions are obsolete and partially incorrect.
May be refer to Documentation/bpf/bpf_devel_QA.rst instead?

> +When building the kernel with CONFIG_DEBUG_INFO_BTF, pahole
> +version 16 or later is also required for BTF function
> +support. pahole can be built from the source at
> +
> +https://github.com/acmel/dwarves
> +
> +It is often available in "dwarves/libdwarves" packages also,
> +but be aware that versions prior to 1.16 will fail with
> +errors that functions cannot be found in BTF.
> +
> +When running selftests, the default timeout of 45 seconds
> +can be exceeded by some tests.  We can override the default
> +timeout via a "settings" file; for example:
> +
> +.. code-block:: bash
> +  echo "timeout=120" > tools/testing/selftests/bpf/settings

Is it really the case?
I've never seen anything like this.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH bpf-next] selftests/bpf: add general instructions for test execution
  2020-05-19 15:50 ` Alexei Starovoitov
@ 2020-05-19 21:34   ` Alan Maguire
  2020-05-20  5:26     ` Alexei Starovoitov
  0 siblings, 1 reply; 5+ messages in thread
From: Alan Maguire @ 2020-05-19 21:34 UTC (permalink / raw)
  To: Alexei Starovoitov
  Cc: Alan Maguire, shuah, ast, daniel, andriin, bpf, kafai,
	songliubraving, yhs, john.fastabend, kpsingh, linux-kselftest,
	netdev

On Tue, 19 May 2020, Alexei Starovoitov wrote:

> On Mon, May 18, 2020 at 12:23:10PM +0100, Alan Maguire wrote:
> > Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
> > are used, pahole is recent (>=1.16) and config matches the specified
> > config file as closely as possible.  Document all of this in the general
> > README.rst file.  Also note how to work around timeout failures.
> > 
> > Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> > ---
> >  tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
> >  1 file changed, 46 insertions(+)
> > 
> > diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst
> > index 0f67f1b..b00eebb 100644
> > --- a/tools/testing/selftests/bpf/README.rst
> > +++ b/tools/testing/selftests/bpf/README.rst
> > @@ -1,6 +1,52 @@
> >  ==================
> >  BPF Selftest Notes
> >  ==================
> > +First verify the built kernel config options match the config options
> > +specified in the config file in this directory.  Test failures for
> > +unknown helpers, inability to find BTF etc will be observed otherwise.
> > +
> > +To ensure the maximum number of tests pass, it is best to use the latest
> > +trunk LLVM/clang, i.e.
> > +
> > +git clone https://github.com/llvm/llvm-project
> > +
> > +Build/install trunk LLVM:
> > +
> > +.. code-block:: bash
> > +  git clone https://github.com/llvm/llvm-project
> > +  cd llvm-project
> > +  mkdir build/llvm
> > +  cd build/llvm
> > +  cmake ../../llvm/
> > +  make
> > +  sudo make install
> > +  cd ../../
> > +
> > +Build/install trunk clang:
> > +
> > +.. code-block:: bash
> > +  mkdir -p build/clang
> > +  cd build/clang
> > +  cmake ../../clang
> > +  make
> > +  sudo make install
> > +
> 
> these instructions are obsolete and partially incorrect.
> May be refer to Documentation/bpf/bpf_devel_QA.rst instead?
>

Sure; looks like there are up-to-date sections there on
running BPF selftests and building LLVM manually.  Perhaps
I should add the notes about pahole etc there too?
I should also have noted that without an up-to-date iproute2
failures will be observed also.
  
> > +When building the kernel with CONFIG_DEBUG_INFO_BTF, pahole
> > +version 16 or later is also required for BTF function
> > +support. pahole can be built from the source at
> > +
> > +https://github.com/acmel/dwarves
> > +
> > +It is often available in "dwarves/libdwarves" packages also,
> > +but be aware that versions prior to 1.16 will fail with
> > +errors that functions cannot be found in BTF.
> > +
> > +When running selftests, the default timeout of 45 seconds
> > +can be exceeded by some tests.  We can override the default
> > +timeout via a "settings" file; for example:
> > +
> > +.. code-block:: bash
> > +  echo "timeout=120" > tools/testing/selftests/bpf/settings
> 
> Is it really the case?
> I've never seen anything like this.
> 

When running via "make run_tests" on baremetal systems I
see test timeouts pretty consistently; e.g. from a bpf tree test
run yesterday:

not ok 6 selftests: bpf: test_progs # TIMEOUT
not ok 31 selftests: bpf: test_tunnel.sh # TIMEOUT
not ok 38 selftests: bpf: test_lwt_ip_encap.sh # TIMEOUT
not ok 40 selftests: bpf: test_tc_tunnel.sh # TIMEOUT
not ok 42 selftests: bpf: test_xdping.sh # TIMEOUT
not ok 43 selftests: bpf: test_bpftool_build.sh # TIMEOUT

These will only occur if running via "make run_tests",
so running tests individually would not trigger these
failures.

Alan

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH bpf-next] selftests/bpf: add general instructions for test execution
  2020-05-19 21:34   ` Alan Maguire
@ 2020-05-20  5:26     ` Alexei Starovoitov
  0 siblings, 0 replies; 5+ messages in thread
From: Alexei Starovoitov @ 2020-05-20  5:26 UTC (permalink / raw)
  To: Alan Maguire
  Cc: shuah, ast, daniel, andriin, bpf, kafai, songliubraving, yhs,
	john.fastabend, kpsingh, linux-kselftest, netdev

On Tue, May 19, 2020 at 10:34:25PM +0100, Alan Maguire wrote:
> On Tue, 19 May 2020, Alexei Starovoitov wrote:
> 
> > On Mon, May 18, 2020 at 12:23:10PM +0100, Alan Maguire wrote:
> > > Getting a clean BPF selftests run involves ensuring latest trunk LLVM/clang
> > > are used, pahole is recent (>=1.16) and config matches the specified
> > > config file as closely as possible.  Document all of this in the general
> > > README.rst file.  Also note how to work around timeout failures.
> > > 
> > > Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> > > ---
> > >  tools/testing/selftests/bpf/README.rst | 46 ++++++++++++++++++++++++++++++++++
> > >  1 file changed, 46 insertions(+)
> > > 
> > > diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst
> > > index 0f67f1b..b00eebb 100644
> > > --- a/tools/testing/selftests/bpf/README.rst
> > > +++ b/tools/testing/selftests/bpf/README.rst
> > > @@ -1,6 +1,52 @@
> > >  ==================
> > >  BPF Selftest Notes
> > >  ==================
> > > +First verify the built kernel config options match the config options
> > > +specified in the config file in this directory.  Test failures for
> > > +unknown helpers, inability to find BTF etc will be observed otherwise.
> > > +
> > > +To ensure the maximum number of tests pass, it is best to use the latest
> > > +trunk LLVM/clang, i.e.
> > > +
> > > +git clone https://github.com/llvm/llvm-project
> > > +
> > > +Build/install trunk LLVM:
> > > +
> > > +.. code-block:: bash
> > > +  git clone https://github.com/llvm/llvm-project
> > > +  cd llvm-project
> > > +  mkdir build/llvm
> > > +  cd build/llvm
> > > +  cmake ../../llvm/
> > > +  make
> > > +  sudo make install
> > > +  cd ../../
> > > +
> > > +Build/install trunk clang:
> > > +
> > > +.. code-block:: bash
> > > +  mkdir -p build/clang
> > > +  cd build/clang
> > > +  cmake ../../clang
> > > +  make
> > > +  sudo make install
> > > +
> > 
> > these instructions are obsolete and partially incorrect.
> > May be refer to Documentation/bpf/bpf_devel_QA.rst instead?
> >
> 
> Sure; looks like there are up-to-date sections there on
> running BPF selftests and building LLVM manually.  Perhaps
> I should add the notes about pahole etc there too?

yes. please.
Could you mention distros that have fresh pahole?
Otherwise users will have an impression that pahole needs
to be build from scratch as well. Which is not the case.

> I should also have noted that without an up-to-date iproute2
> failures will be observed also.

Would be good to highlight which tests will fail with old iproute2.
I'm not sure which version is necessary.
What is 'up-to-date' ?
The tests I run before applying work for me and I rebuild iproute2
every year or so :)

> > > +When building the kernel with CONFIG_DEBUG_INFO_BTF, pahole
> > > +version 16 or later is also required for BTF function
> > > +support. pahole can be built from the source at
> > > +
> > > +https://github.com/acmel/dwarves
> > > +
> > > +It is often available in "dwarves/libdwarves" packages also,
> > > +but be aware that versions prior to 1.16 will fail with
> > > +errors that functions cannot be found in BTF.
> > > +
> > > +When running selftests, the default timeout of 45 seconds
> > > +can be exceeded by some tests.  We can override the default
> > > +timeout via a "settings" file; for example:
> > > +
> > > +.. code-block:: bash
> > > +  echo "timeout=120" > tools/testing/selftests/bpf/settings
> > 
> > Is it really the case?
> > I've never seen anything like this.
> > 
> 
> When running via "make run_tests" on baremetal systems I
> see test timeouts pretty consistently; e.g. from a bpf tree test
> run yesterday:
> 
> not ok 6 selftests: bpf: test_progs # TIMEOUT
> not ok 31 selftests: bpf: test_tunnel.sh # TIMEOUT
> not ok 38 selftests: bpf: test_lwt_ip_encap.sh # TIMEOUT
> not ok 40 selftests: bpf: test_tc_tunnel.sh # TIMEOUT
> not ok 42 selftests: bpf: test_xdping.sh # TIMEOUT
> not ok 43 selftests: bpf: test_bpftool_build.sh # TIMEOUT
> 
> These will only occur if running via "make run_tests",
> so running tests individually would not trigger these
> failures.

If timeout is necessary it's better to fix it in the git
instead of requiring users to tweak their environment.

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, back to index

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-18 11:23 [PATCH bpf-next] selftests/bpf: add general instructions for test execution Alan Maguire
2020-05-18 23:48 ` Andrii Nakryiko
2020-05-19 15:50 ` Alexei Starovoitov
2020-05-19 21:34   ` Alan Maguire
2020-05-20  5:26     ` Alexei Starovoitov

Linux-kselftest Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-kselftest/0 linux-kselftest/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-kselftest linux-kselftest/ https://lore.kernel.org/linux-kselftest \
		linux-kselftest@vger.kernel.org
	public-inbox-index linux-kselftest

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kselftest


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git