selftests: bpf: test_libbpf.sh failed at file test_l4lb.o

selftests: bpf: test_libbpf.sh failed at file test_l4lb.o

[Naresh Kamboju]

selftests: bpf test_libbpf.sh failed running Linux -next kernel
20190618 and 20190619.

Here is the log from x86_64,
# selftests bpf test_libbpf.sh
bpf: test_libbpf.sh_ #
# [0] libbpf BTF is required, but is missing or corrupted.
libbpf: BTF_is #
# test_libbpf failed at file test_l4lb.o
failed: at_file #
# selftests test_libbpf [FAILED]
test_libbpf: [FAILED]_ #
[FAIL] 29 selftests bpf test_libbpf.sh
selftests: bpf_test_libbpf.sh [FAIL]

Full test log,
https://qa-reports.linaro.org/lkft/linux-next-oe/build/next-20190619/testrun/781777/log

Test results comparison,
https://qa-reports.linaro.org/lkft/linux-next-oe/tests/kselftest/bpf_test_libbpf.sh

Good linux -next tag: next-20190617
Bad linux -next tag: next-20190618
git branch     master
git commit    1c6b40509daf5190b1fd2c758649f7df1da4827b
git repo
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git

Best regards
Naresh Kamboju

Re: selftests: bpf: test_libbpf.sh failed at file test_l4lb.o

[Andrii Nakryiko]

On Thu, Jun 20, 2019 at 1:08 AM Naresh Kamboju
<naresh.kamboju@linaro.org> wrote:
>
> selftests: bpf test_libbpf.sh failed running Linux -next kernel
> 20190618 and 20190619.
>
> Here is the log from x86_64,
> # selftests bpf test_libbpf.sh
> bpf: test_libbpf.sh_ #
> # [0] libbpf BTF is required, but is missing or corrupted.

You need at least clang-9.0.0 (not yet released) to run some of these
tests successfully, as they rely on Clang's support for
BTF_KIND_VAR/BTF_KIND_DATASEC.

> libbpf: BTF_is #
> # test_libbpf failed at file test_l4lb.o
> failed: at_file #
> # selftests test_libbpf [FAILED]
> test_libbpf: [FAILED]_ #
> [FAIL] 29 selftests bpf test_libbpf.sh
> selftests: bpf_test_libbpf.sh [FAIL]
>
> Full test log,
> https://qa-reports.linaro.org/lkft/linux-next-oe/build/next-20190619/testrun/781777/log
>
> Test results comparison,
> https://qa-reports.linaro.org/lkft/linux-next-oe/tests/kselftest/bpf_test_libbpf.sh
>
> Good linux -next tag: next-20190617
> Bad linux -next tag: next-20190618
> git branch     master
> git commit    1c6b40509daf5190b1fd2c758649f7df1da4827b
> git repo
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
>
> Best regards
> Naresh Kamboju

Re: selftests: bpf: test_libbpf.sh failed at file test_l4lb.o

[Dan Rue]

On Thu, Jun 20, 2019 at 10:17:04PM -0700, Andrii Nakryiko wrote:
> On Thu, Jun 20, 2019 at 1:08 AM Naresh Kamboju
> <naresh.kamboju@linaro.org> wrote:
> >
> > selftests: bpf test_libbpf.sh failed running Linux -next kernel
> > 20190618 and 20190619.
> >
> > Here is the log from x86_64,
> > # selftests bpf test_libbpf.sh
> > bpf: test_libbpf.sh_ #
> > # [0] libbpf BTF is required, but is missing or corrupted.
> 
> You need at least clang-9.0.0 (not yet released) to run some of these
> tests successfully, as they rely on Clang's support for
> BTF_KIND_VAR/BTF_KIND_DATASEC.

Can there be a runtime check for BTF that emits a skip instead of a fail
in such a case?

Thanks,
Dan

> 
> > libbpf: BTF_is #
> > # test_libbpf failed at file test_l4lb.o
> > failed: at_file #
> > # selftests test_libbpf [FAILED]
> > test_libbpf: [FAILED]_ #
> > [FAIL] 29 selftests bpf test_libbpf.sh
> > selftests: bpf_test_libbpf.sh [FAIL]
> >
> > Full test log,
> > https://qa-reports.linaro.org/lkft/linux-next-oe/build/next-20190619/testrun/781777/log
> >
> > Test results comparison,
> > https://qa-reports.linaro.org/lkft/linux-next-oe/tests/kselftest/bpf_test_libbpf.sh
> >
> > Good linux -next tag: next-20190617
> > Bad linux -next tag: next-20190618
> > git branch     master
> > git commit    1c6b40509daf5190b1fd2c758649f7df1da4827b
> > git repo
> > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> >
> > Best regards
> > Naresh Kamboju

-- 
Linaro - Kernel Validation

Re: selftests: bpf: test_libbpf.sh failed at file test_l4lb.o

[Andrii Nakryiko]

On Fri, Jun 21, 2019 at 9:17 AM Dan Rue <dan.rue@linaro.org> wrote:
>
> On Thu, Jun 20, 2019 at 10:17:04PM -0700, Andrii Nakryiko wrote:
> > On Thu, Jun 20, 2019 at 1:08 AM Naresh Kamboju
> > <naresh.kamboju@linaro.org> wrote:
> > >
> > > selftests: bpf test_libbpf.sh failed running Linux -next kernel
> > > 20190618 and 20190619.
> > >
> > > Here is the log from x86_64,
> > > # selftests bpf test_libbpf.sh
> > > bpf: test_libbpf.sh_ #
> > > # [0] libbpf BTF is required, but is missing or corrupted.
> >
> > You need at least clang-9.0.0 (not yet released) to run some of these
> > tests successfully, as they rely on Clang's support for
> > BTF_KIND_VAR/BTF_KIND_DATASEC.
>
> Can there be a runtime check for BTF that emits a skip instead of a fail
> in such a case?

I'm not sure how to do this simply and minimally intrusively. The best
I can come up with is setting some envvar from Makefile and checking
for that in each inidividual test, which honestly sounds a bit gross.

How hard is it for you guys to upgrade compiler used to run these test?

>
> Thanks,
> Dan
>
> >
> > > libbpf: BTF_is #
> > > # test_libbpf failed at file test_l4lb.o
> > > failed: at_file #
> > > # selftests test_libbpf [FAILED]
> > > test_libbpf: [FAILED]_ #
> > > [FAIL] 29 selftests bpf test_libbpf.sh
> > > selftests: bpf_test_libbpf.sh [FAIL]
> > >
> > > Full test log,
> > > https://qa-reports.linaro.org/lkft/linux-next-oe/build/next-20190619/testrun/781777/log
> > >
> > > Test results comparison,
> > > https://qa-reports.linaro.org/lkft/linux-next-oe/tests/kselftest/bpf_test_libbpf.sh
> > >
> > > Good linux -next tag: next-20190617
> > > Bad linux -next tag: next-20190618
> > > git branch     master
> > > git commit    1c6b40509daf5190b1fd2c758649f7df1da4827b
> > > git repo
> > > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> > >
> > > Best regards
> > > Naresh Kamboju
>
> --
> Linaro - Kernel Validation

Re: selftests: bpf: test_libbpf.sh failed at file test_l4lb.o

[Dan Rue]

On Mon, Jun 24, 2019 at 11:32:25AM -0700, Andrii Nakryiko wrote:
> On Fri, Jun 21, 2019 at 9:17 AM Dan Rue <dan.rue@linaro.org> wrote:
> >
> > On Thu, Jun 20, 2019 at 10:17:04PM -0700, Andrii Nakryiko wrote:
> > > On Thu, Jun 20, 2019 at 1:08 AM Naresh Kamboju
> > > <naresh.kamboju@linaro.org> wrote:
> > > >
> > > > selftests: bpf test_libbpf.sh failed running Linux -next kernel
> > > > 20190618 and 20190619.
> > > >
> > > > Here is the log from x86_64,
> > > > # selftests bpf test_libbpf.sh
> > > > bpf: test_libbpf.sh_ #
> > > > # [0] libbpf BTF is required, but is missing or corrupted.
> > >
> > > You need at least clang-9.0.0 (not yet released) to run some of these
> > > tests successfully, as they rely on Clang's support for
> > > BTF_KIND_VAR/BTF_KIND_DATASEC.
> >
> > Can there be a runtime check for BTF that emits a skip instead of a fail
> > in such a case?
> 
> I'm not sure how to do this simply and minimally intrusively. The best
> I can come up with is setting some envvar from Makefile and checking
> for that in each inidividual test, which honestly sounds a bit gross.
> 
> How hard is it for you guys to upgrade compiler used to run these test?

We should be able to run kselftest with any compiler that Linux
supports, so that we can test with the toolchain that users actually run
with.

I would say if it's not possible to check at runtime, and it requires
clang 9.0, that this test should not be enabled by default.

Maybe something could be done in Makefile for that? Only add it to
TEST_GEN_PROGS if the toolchain feature exists, otherwise add it to
TEST_GEN_PROGS_EXTENDED. I don't know if this is a good idea.. but from
kselftest.rst:

   TEST_PROGS, TEST_GEN_PROGS mean it is the executable tested by
   default.
   ...
   TEST_PROGS_EXTENDED, TEST_GEN_PROGS_EXTENDED mean it is the
   executable which is not tested by default.

Dan

> 
> >
> > Thanks,
> > Dan
> >
> > >
> > > > libbpf: BTF_is #
> > > > # test_libbpf failed at file test_l4lb.o
> > > > failed: at_file #
> > > > # selftests test_libbpf [FAILED]
> > > > test_libbpf: [FAILED]_ #
> > > > [FAIL] 29 selftests bpf test_libbpf.sh
> > > > selftests: bpf_test_libbpf.sh [FAIL]
> > > >
> > > > Full test log,
> > > > https://qa-reports.linaro.org/lkft/linux-next-oe/build/next-20190619/testrun/781777/log
> > > >
> > > > Test results comparison,
> > > > https://qa-reports.linaro.org/lkft/linux-next-oe/tests/kselftest/bpf_test_libbpf.sh
> > > >
> > > > Good linux -next tag: next-20190617
> > > > Bad linux -next tag: next-20190618
> > > > git branch     master
> > > > git commit    1c6b40509daf5190b1fd2c758649f7df1da4827b
> > > > git repo
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
> > > >
> > > > Best regards
> > > > Naresh Kamboju
> >
> > --
> > Linaro - Kernel Validation

-- 
Linaro - Kernel Validation

Re: selftests: bpf: test_libbpf.sh failed at file test_l4lb.o

[Alexei Starovoitov]

On Mon, Jun 24, 2019 at 12:53 PM Dan Rue <dan.rue@linaro.org> wrote:
>
> I would say if it's not possible to check at runtime, and it requires
> clang 9.0, that this test should not be enabled by default.

The latest clang is the requirement.
If environment has old clang or no clang at all these tests will be failing.

Re: selftests: bpf: test_libbpf.sh failed at file test_l4lb.o

[Dan Rue]

On Mon, Jun 24, 2019 at 12:58:15PM -0700, Alexei Starovoitov wrote:
> On Mon, Jun 24, 2019 at 12:53 PM Dan Rue <dan.rue@linaro.org> wrote:
> >
> > I would say if it's not possible to check at runtime, and it requires
> > clang 9.0, that this test should not be enabled by default.
> 
> The latest clang is the requirement.
> If environment has old clang or no clang at all these tests will be failing.

Hi Alexei!

I'm not certain if I'm interpreting you as you intended, but it sounds
like you're telling me that if the test build environment does not use
'latest clang' (i guess latest as of today?), that these tests will
fail, and that is how it is going to be. If I have that wrong, please
correct me and disregard the rest of my message.

Please understand where we are coming from. We (and many others) run
thousands of tests from a lot of test frameworks, and so our environment
often has mutually exclusive requirements when it comes to things like
toolchain selection.

We believe, strongly, that a test should not emit a "fail" for a missing
requirement. Fail is a serious thing, and should be reserved for an
actual issue that needs to be investigated, reported, and fixed.

This is how we treat test failures - we investigate, report, and fix
them when possible. When they're not real failures, we waste our time
(and yours, in this case).

By adding the tests to TEST_GEN_PROGS, you're adding them to the general
test set that those of us running test farms try to run continuously
across a wide range of hardware environments and kernel branches.

My suggestion is that if you do not want us running them, don't add them
to TEST_GEN_PROGS. I thought the suggestion of testing for adequate
clang support and adding them conditionally at build-time was an idea
worth consideration.

Thanks,
Dan

-- 
Linaro - Kernel Validation

Re: selftests: bpf: test_libbpf.sh failed at file test_l4lb.o

[Alexei Starovoitov]

On Tue, Jun 25, 2019 at 8:32 AM Dan Rue <dan.rue@linaro.org> wrote:
>
> On Mon, Jun 24, 2019 at 12:58:15PM -0700, Alexei Starovoitov wrote:
> > On Mon, Jun 24, 2019 at 12:53 PM Dan Rue <dan.rue@linaro.org> wrote:
> > >
> > > I would say if it's not possible to check at runtime, and it requires
> > > clang 9.0, that this test should not be enabled by default.
> >
> > The latest clang is the requirement.
> > If environment has old clang or no clang at all these tests will be failing.
>
> Hi Alexei!
>
> I'm not certain if I'm interpreting you as you intended, but it sounds
> like you're telling me that if the test build environment does not use
> 'latest clang' (i guess latest as of today?), that these tests will
> fail, and that is how it is going to be. If I have that wrong, please
> correct me and disregard the rest of my message.
>
> Please understand where we are coming from. We (and many others) run
> thousands of tests from a lot of test frameworks, and so our environment
> often has mutually exclusive requirements when it comes to things like
> toolchain selection.
>
> We believe, strongly, that a test should not emit a "fail" for a missing
> requirement. Fail is a serious thing, and should be reserved for an
> actual issue that needs to be investigated, reported, and fixed.
>
> This is how we treat test failures - we investigate, report, and fix
> them when possible. When they're not real failures, we waste our time
> (and yours, in this case).
>
> By adding the tests to TEST_GEN_PROGS, you're adding them to the general
> test set that those of us running test farms try to run continuously
> across a wide range of hardware environments and kernel branches.

you run the latest selftests/bpf on the latest kernel, right?
If not than selftests/bpf is not for your setup.

In the past people argued that selftests/bpf should check
features of the kernel and skip when features are not found.
My answer to that was always the same: such changes to selftests
for older kernels need to live out of tree.
selftests/bpf are one to one to the latest kernel.
Often kernel commit X will break selftests and they're fixed
in the commit X+1.
clang, pahole, bpftool, iproute2 provide those features for the kernel.
In other words new kernel features rely on new clang and
other tools and selftests are testing those latest kernel features.
Without new clang many new features cannot be tested exhaustively.
datasec and btf are just few examples.
Hence if your test farm cannot install the latest clang, pahole, etc then
I recommend not to run selftest/bpf.