Running kunit_tool on unclean trees

Running kunit_tool on unclean trees

[Marcelo Schmitt]

Hi, a friend and I were chasing bug 205219 [1] listed in Bugzilla.
We step into something a little bit different when trying to reproduce
the buggy behavior. In our try, compilation failed with a message form
make asking us to clean the source tree. We couldn't run kunit_tool
after compiling the kernel for x86, as described by Ted in the
discussion pointed out by the bug report.

Steps to reproduce:

0) Run kunit_tool
$ ./tools/testing/kunit/kunit.py run
Works fine with a clean tree.

1) Compile the kernel for some architecture (we did it for x86_64).

2) Run kunit_tool again
$ ./tools/testing/kunit/kunit.py run
Fails with a message form make asking us to clean the source tree.

Removing the clean source tree check from the top-level Makefile gives
us a similar error to what was described in the bug report. We see that
after running `git clean -fdx` kunit_tool runs nicely again. However,
this is not a real solution since some kernel binaries are erased by git.

We also had a look into the commit messages of Masahiro Yamada but
couldn't quite grasp why the check for the tree to be clean was added.
We could invest more time in this issue but actually don't know how to
proceed. We'd be glad to receive any comment about it. We could also try
something else if it's a too hard issue for beginners.

[1]: https://bugzilla.kernel.org/show_bug.cgi?id=205219


Best Regards,

Marcelo

Re: Running kunit_tool on unclean trees

[Masahiro Yamada]

On Sun, Apr 25, 2021 at 4:05 AM Marcelo Schmitt
<marcelo.schmitt1@gmail.com> wrote:
>
> Hi, a friend and I were chasing bug 205219 [1] listed in Bugzilla.
> We step into something a little bit different when trying to reproduce
> the buggy behavior. In our try, compilation failed with a message form
> make asking us to clean the source tree. We couldn't run kunit_tool
> after compiling the kernel for x86, as described by Ted in the
> discussion pointed out by the bug report.
>
> Steps to reproduce:
>
> 0) Run kunit_tool
> $ ./tools/testing/kunit/kunit.py run
> Works fine with a clean tree.
>
> 1) Compile the kernel for some architecture (we did it for x86_64).
>
> 2) Run kunit_tool again
> $ ./tools/testing/kunit/kunit.py run
> Fails with a message form make asking us to clean the source tree.

This is probably because
tools/testing/kunit/kunit_kernel.py
runs make with O= option.



> Removing the clean source tree check from the top-level Makefile gives
> us a similar error to what was described in the bug report. We see that
> after running `git clean -fdx` kunit_tool runs nicely again. However,
> this is not a real solution since some kernel binaries are erased by git.
>
> We also had a look into the commit messages of Masahiro Yamada but
> couldn't quite grasp why the check for the tree to be clean was added.
> We could invest more time in this issue but actually don't know how to
> proceed. We'd be glad to receive any comment about it. We could also try
> something else if it's a too hard issue for beginners.

I think you are talking about the following error message.

***
*** The source tree is not clean, please run 'make mrproper'
*** in /home/masahiro/ref/linux
***



Kbuild checks if the source tree is clean
before starting the out-of-tree build
because the out-of-tree build relies on VPATH.

This check has existed for a long time. (at least more than a decade)

If Kbuild started the O= build with a dirty source tree,
some stale generated source files would have been remaining.
(some *.c and *.h are generated by scripts)

Then, Kbuild would wrongly use stale source files in srctree
instead of generating new ones in objtree.





> [1]: https://bugzilla.kernel.org/show_bug.cgi?id=205219
>
>
> Best Regards,
>
> Marcelo



--
Best Regards
Masahiro Yamada

Re: Running kunit_tool on unclean trees

[David Gow]

On Mon, Apr 26, 2021 at 3:06 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> On Sun, Apr 25, 2021 at 4:05 AM Marcelo Schmitt
> <marcelo.schmitt1@gmail.com> wrote:
> >
> > Hi, a friend and I were chasing bug 205219 [1] listed in Bugzilla.
> > We step into something a little bit different when trying to reproduce
> > the buggy behavior. In our try, compilation failed with a message form
> > make asking us to clean the source tree. We couldn't run kunit_tool
> > after compiling the kernel for x86, as described by Ted in the
> > discussion pointed out by the bug report.
> >
> > Steps to reproduce:
> >
> > 0) Run kunit_tool
> > $ ./tools/testing/kunit/kunit.py run
> > Works fine with a clean tree.
> >
> > 1) Compile the kernel for some architecture (we did it for x86_64).
> >
> > 2) Run kunit_tool again
> > $ ./tools/testing/kunit/kunit.py run
> > Fails with a message form make asking us to clean the source tree.
>
> This is probably because
> tools/testing/kunit/kunit_kernel.py
> runs make with O= option.
>
>
>

Yes: KUnit's tooling has started setting O=.kunit by default[1]. If
you'd rather build KUnit in-tree, you can use the "kunit.py run
--build_dir=." option, which should work.

-- David

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/tools/testing/kunit?id=ddbd60c779b4ddaa87173a160ce90146933fb8f9

Re: Running kunit_tool on unclean trees

[Marcelo Schmitt]

On 04/26, Masahiro Yamada wrote:
> On Sun, Apr 25, 2021 at 4:05 AM Marcelo Schmitt
> <marcelo.schmitt1@gmail.com> wrote:
> >
> > Hi, a friend and I were chasing bug 205219 [1] listed in Bugzilla.
> > We step into something a little bit different when trying to reproduce
> > the buggy behavior. In our try, compilation failed with a message form
> > make asking us to clean the source tree. We couldn't run kunit_tool
> > after compiling the kernel for x86, as described by Ted in the
> > discussion pointed out by the bug report.
> >
> > Steps to reproduce:
> >
> > 0) Run kunit_tool
> > $ ./tools/testing/kunit/kunit.py run
> > Works fine with a clean tree.
> >
> > 1) Compile the kernel for some architecture (we did it for x86_64).
> >
> > 2) Run kunit_tool again
> > $ ./tools/testing/kunit/kunit.py run
> > Fails with a message form make asking us to clean the source tree.
> 
> This is probably because
> tools/testing/kunit/kunit_kernel.py
> runs make with O= option.
> 
> 
> 
> > Removing the clean source tree check from the top-level Makefile gives
> > us a similar error to what was described in the bug report. We see that
> > after running `git clean -fdx` kunit_tool runs nicely again. However,
> > this is not a real solution since some kernel binaries are erased by git.
> >
> > We also had a look into the commit messages of Masahiro Yamada but
> > couldn't quite grasp why the check for the tree to be clean was added.
> > We could invest more time in this issue but actually don't know how to
> > proceed. We'd be glad to receive any comment about it. We could also try
> > something else if it's a too hard issue for beginners.
> 
> I think you are talking about the following error message.
> 
> ***
> *** The source tree is not clean, please run 'make mrproper'
> *** in /home/masahiro/ref/linux
> ***
> 

Yes, I wanted to mean that message.

> 
> 
> Kbuild checks if the source tree is clean
> before starting the out-of-tree build
> because the out-of-tree build relies on VPATH.
> 
> This check has existed for a long time. (at least more than a decade)
> 
> If Kbuild started the O= build with a dirty source tree,
> some stale generated source files would have been remaining.
> (some *.c and *.h are generated by scripts)
> 
> Then, Kbuild would wrongly use stale source files in srctree
> instead of generating new ones in objtree.
> 

I see, we better try not to break stuff.
Well, I can't think of any means of fixing bug 205219 so we're probably
going to start looking forward to work on fixes for another one.


Thanks,

Marcelo

> 
> 
> 
> 
> > [1]: https://bugzilla.kernel.org/show_bug.cgi?id=205219
> >
> >
> > Best Regards,
> >
> > Marcelo
> 
> 
> 
> --
> Best Regards
> Masahiro Yamada
> 
> -- 
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/CAK7LNATsbkhYHk6NCZJCDrtT0NFfBwe_n9GRSrEvURaXaW%2Bgfg%40mail.gmail.com.