Kunit get 0 tests when run defconfig in Debian9

Kunit get 0 tests when run defconfig in Debian9

[kernel test robot]

Hi, all.


I met a strange problem that kunit run 0 tests in Debian9.


Env:

```

root@lkp-hsw-ep4 ~/linus# head /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian

root@lkp-hsw-ep4 ~/linus# uname -r
5.6.0

```


Reproduce: Just run `./tools/testing/kunit/kunit.py run --defconfig`

```

root@lkp-hsw-ep4 ~/linus# ./tools/testing/kunit/kunit.py run --defconfig
[13:41:30] Building KUnit Kernel ...
[13:41:33] Starting KUnit Kernel ...
[13:41:33] ============================================================
[13:41:33] Testing complete. 0 tests run. 0 failed. 0 crashed.
[13:41:33] Elapsed time: 2.619s total, 0.002s configuring, 2.610s 
building, 0.007s running

root@lkp-hsw-ep4 ~/linus# cat .kunitconfig
CONFIG_KUNIT=y
CONFIG_KUNIT_TEST=y
CONFIG_KUNIT_EXAMPLE_TEST=y

```


As doc said, looks like everything worked correctly.  But kunit run 0 tests.

```

If everything worked correctly, you should see the following:

Generating .config ...
Building KUnit Kernel ...
Starting KUnit Kernel ...

```



-- 
Best regards.
Liu Yiding

Re: Kunit get 0 tests when run defconfig in Debian9

[David Gow]

Thanks for checking KUnit out — it's a shame it hasn't worked
out-of-the-box here.

I'm not sure off the top of my head what could be wrong here: as you
note, there are no obvious errors. My suspicion is that it's either
some strange incompatibility with UML, or something is not quite right
with the config being generated and used.

A few questions/suggestions:
- Are there any test results visible when running
./tools/testing/kunit/kunit.py run --raw_output  (older versions of
the KUnit parser may not work well if the kernel log is decorated,
e.g. with timestamps, etc — this should be largely fixed for 5.7).
- If not, does running the built UML kernel manually work, and if so
does it print any test results? This should be as simple as running
"./vmlinux". On some older host kernels, UML may not boot if built
statically and KASLR is in its strictest mode. This shouldn't be the
case here, but running "setarch `uname -m` -R ./vmlinux" may help if
some weirder issues show up.
- Does the .config actually have CONFIG_KUNIT_TEST=y and
CONFIG_KUNIT_EXAMPLE_TEST=y? kunit.py will keep its .config around, so
you can check it after running kunit.py run. It'll also preserve any
config changes you make that don't conflict with the .kunitconfig, so
you could try (for example), also enabling CONFIG_LIST_KUNIT_TEST to
see if it's a problem specific to those tests.
- Finally, running 'make ARCH=um defconfig' before running kunit.py
run may help clean up any residual config that could be causing
problems.
- Otherwise, I'm not sure what could be going wrong. Are you running
on x86_64? UML doesn't support non-x86 architectures, and while the
32-bit stuff worked last I checked, it's not tested as regularly at
present.

Of course, it's also possible to run tests without UML or the kunit.py
script as well, by either building them into a kernel, and reading the
results from the log on startup, or building the tests as modules,
modprobing them, and reading the results from
/sys/kernel/debug/kunit/<test-suite>/results. That's obviously a bit
less convenient than the script, but may fit better into existing test
infrastructure, and can sidestep any UML issues.

Hope that helps!

Cheers,
-- David

On Thu, May 21, 2020 at 1:57 PM kernel test robot <yidingx.liu@intel.com> wrote:
>
> Hi, all.
>
>
> I met a strange problem that kunit run 0 tests in Debian9.
>
>
> Env:
>
> ```
>
> root@lkp-hsw-ep4 ~/linus# head /etc/os-release
> PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
> NAME="Debian GNU/Linux"
> VERSION_ID="9"
> VERSION="9 (stretch)"
> ID=debian
>
> root@lkp-hsw-ep4 ~/linus# uname -r
> 5.6.0
>
> ```
>
>
> Reproduce: Just run `./tools/testing/kunit/kunit.py run --defconfig`
>
> ```
>
> root@lkp-hsw-ep4 ~/linus# ./tools/testing/kunit/kunit.py run --defconfig
> [13:41:30] Building KUnit Kernel ...
> [13:41:33] Starting KUnit Kernel ...
> [13:41:33] ============================================================
> [13:41:33] Testing complete. 0 tests run. 0 failed. 0 crashed.
> [13:41:33] Elapsed time: 2.619s total, 0.002s configuring, 2.610s
> building, 0.007s running
>
> root@lkp-hsw-ep4 ~/linus# cat .kunitconfig
> CONFIG_KUNIT=y
> CONFIG_KUNIT_TEST=y
> CONFIG_KUNIT_EXAMPLE_TEST=y
>
> ```
>
>
> As doc said, looks like everything worked correctly.  But kunit run 0 tests.
>
> ```
>
> If everything worked correctly, you should see the following:
>
> Generating .config ...
> Building KUnit Kernel ...
> Starting KUnit Kernel ...
>
> ```
>
>
>
> --
> Best regards.
> Liu Yiding
>
> --
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/ef994ce9-ccd8-e9c9-9838-049a62a3a561%40intel.com.

Re: Kunit get 0 tests when run defconfig in Debian9

[Alan Maguire]

[-- Attachment #1: Type: text/plain, Size: 4508 bytes --]

On Fri, 22 May 2020, David Gow wrote:

> Thanks for checking KUnit out — it's a shame it hasn't worked
> out-of-the-box here.
> 
> I'm not sure off the top of my head what could be wrong here: as you
> note, there are no obvious errors. My suspicion is that it's either
> some strange incompatibility with UML, or something is not quite right
> with the config being generated and used.
> 
> A few questions/suggestions:
> - Are there any test results visible when running
> ./tools/testing/kunit/kunit.py run --raw_output  (older versions of
> the KUnit parser may not work well if the kernel log is decorated,
> e.g. with timestamps, etc — this should be largely fixed for 5.7).
> - If not, does running the built UML kernel manually work, and if so
> does it print any test results? This should be as simple as running
> "./vmlinux". On some older host kernels, UML may not boot if built
> statically and KASLR is in its strictest mode. This shouldn't be the
> case here, but running "setarch `uname -m` -R ./vmlinux" may help if
> some weirder issues show up.
> - Does the .config actually have CONFIG_KUNIT_TEST=y and
> CONFIG_KUNIT_EXAMPLE_TEST=y? kunit.py will keep its .config around, so
> you can check it after running kunit.py run. It'll also preserve any
> config changes you make that don't conflict with the .kunitconfig, so
> you could try (for example), also enabling CONFIG_LIST_KUNIT_TEST to
> see if it's a problem specific to those tests.
> - Finally, running 'make ARCH=um defconfig' before running kunit.py
> run may help clean up any residual config that could be causing
> problems.
> - Otherwise, I'm not sure what could be going wrong. Are you running
> on x86_64? UML doesn't support non-x86 architectures, and while the
> 32-bit stuff worked last I checked, it's not tested as regularly at
> present.
> 
> Of course, it's also possible to run tests without UML or the kunit.py
> script as well, by either building them into a kernel, and reading the
> results from the log on startup, or building the tests as modules,
> modprobing them, and reading the results from
> /sys/kernel/debug/kunit/<test-suite>/results. That's obviously a bit
> less convenient than the script, but may fit better into existing test
> infrastructure, and can sidestep any UML issues.
>

This is all great advice; I wonder if we should add a "Troubleshooting"
section to the KUnit documentation's start.rst file? A slightly-edited
(to be more general) version of the above would be really helpful there
I think. Thanks!

Alan

 
> Hope that helps!
> 
> Cheers,
> -- David
> 
> On Thu, May 21, 2020 at 1:57 PM kernel test robot <yidingx.liu@intel.com> wrote:
> >
> > Hi, all.
> >
> >
> > I met a strange problem that kunit run 0 tests in Debian9.
> >
> >
> > Env:
> >
> > ```
> >
> > root@lkp-hsw-ep4 ~/linus# head /etc/os-release
> > PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
> > NAME="Debian GNU/Linux"
> > VERSION_ID="9"
> > VERSION="9 (stretch)"
> > ID=debian
> >
> > root@lkp-hsw-ep4 ~/linus# uname -r
> > 5.6.0
> >
> > ```
> >
> >
> > Reproduce: Just run `./tools/testing/kunit/kunit.py run --defconfig`
> >
> > ```
> >
> > root@lkp-hsw-ep4 ~/linus# ./tools/testing/kunit/kunit.py run --defconfig
> > [13:41:30] Building KUnit Kernel ...
> > [13:41:33] Starting KUnit Kernel ...
> > [13:41:33] ============================================================
> > [13:41:33] Testing complete. 0 tests run. 0 failed. 0 crashed.
> > [13:41:33] Elapsed time: 2.619s total, 0.002s configuring, 2.610s
> > building, 0.007s running
> >
> > root@lkp-hsw-ep4 ~/linus# cat .kunitconfig
> > CONFIG_KUNIT=y
> > CONFIG_KUNIT_TEST=y
> > CONFIG_KUNIT_EXAMPLE_TEST=y
> >
> > ```
> >
> >
> > As doc said, looks like everything worked correctly.  But kunit run 0 tests.
> >
> > ```
> >
> > If everything worked correctly, you should see the following:
> >
> > Generating .config ...
> > Building KUnit Kernel ...
> > Starting KUnit Kernel ...
> >
> > ```
> >
> >
> >
> > --
> > Best regards.
> > Liu Yiding
> >
> > --
> > You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/ef994ce9-ccd8-e9c9-9838-049a62a3a561%40intel.com.
> 

Re: Kunit get 0 tests when run defconfig in Debian9

[Liu Yiding]

Hi, David.


Sorry for the delay and really appericated for your help !!!

On 5/22/20 10:33 AM, David Gow wrote:
> A few questions/suggestions:
> - Are there any test results visible when running
> ./tools/testing/kunit/kunit.py run --raw_output  (older versions of
> the KUnit parser may not work well if the kernel log is decorated,
> e.g. with timestamps, etc — this should be largely fixed for 5.7).


--raw_output is really helpful.

The root cause of missing tests is that tempdir doesn't exist.

I create that file and kunit runs well now. :-P

```

root@vm-snb-195 ~/linus# ./tools/testing/kunit/kunit.py run --raw_output 
--defconfig
Generating .config ...
[09:35:37] Building KUnit Kernel ...
arch/um/os-Linux/signal.c: In function ‘sig_handler_common’:
arch/um/os-Linux/signal.c:51:1: warning: the frame size of 2960 bytes is 
larger than 2048 bytes [-Wframe-larger-than=]
  }
  ^
arch/um/os-Linux/signal.c: In function ‘timer_real_alarm_handler’:
arch/um/os-Linux/signal.c:95:1: warning: the frame size of 2960 bytes is 
larger than 2048 bytes [-Wframe-larger-than=]
  }
  ^
[09:39:48] Starting KUnit Kernel ...
Core dump limits :

     soft - 0

     hard - 0

Checking that ptrace can change system call numbers...OK

Checking syscall emulation patch for ptrace...OK

Checking advanced syscall emulation patch for ptrace...OK

Checking environment variables for a tempdir.../lkp/lkp/src/tmp

Checking if /lkp/lkp/src/tmp is on tmpfs...No such file or directory

Warning: tempdir /lkp/lkp/src/tmp is not on tmpfs

[09:39:48] Elapsed time: 257.808s total, 6.515s configuring, 251.274s 
building, 0.018s running

```

-- 
Best regards.
Liu Yiding