ptest results

ptest results

[Saul Wold]

Bjorn:

I ran ptest-runner finally I saw that there were failures, I am 
wondering if these are expected?

http://pastebin.com/PAH3gJT0

The bash packages did not seem to get split out correct, the ptest seem 
to land in ${PN} not in ${PN}-ptest, you might want to look into this also.

With your patch, the ptest related debug files did go into the 
${PN}-dbg, is this what you intend or should they go into a 
${PN}-ptest-dbg package?

Thanks


-- 
     Sau!

Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System

Re: ptest results

[Björn Stenberg]

Saul Wold wrote:
> I ran ptest-runner finally I saw that there were failures, I am
> wondering if these are expected?
>
> http://pastebin.com/PAH3gJT0

Thank you for testing. Some test failures are to be expected. The purpose of ptest is to make the tests build and run. Fixing errors found by the tests is a separate job.

That said, it is not always immediately obvious which errors are due to recipe/framework malfunction and which are actual test failures. You paste contains some errors that are clearly recipe/framework issues:

- bash-ptest appears not have installed itself correctly. It looks like the do_install_ptest() action has not been run. I fail to repeat this. Are you seeing any errors in tmp/work/x86_64-poky-linux/bash/4.2-r6/temp/log.do_install_ptest_base?

- dbus-ptest-ptest has a similar issue. Its do_install_ptest contains a sed expression to remove the dependencies that cause the error you get. Again it appears the action has not been run for some reason. I can't repeat this either.

- Some test cases in dbus-ptest-ptest require /bin/sed, which your image doesn't have installed. The first-glance fix is to add sed as a runtime dependency, but that doesn't work with images built with busybox. I'm not sure what the proper fix is here.

- The references to gcc in the output are due to libtool issues that I have missed cleaning out. This is a bug in the dbus-ptest recipe.

- There is a path issue with some tests spawning binaries or opening files using the host build dir path, and obviously failing. This too is a recipe/testsuite bug.

> The bash packages did not seem to get split out correct, the ptest
> seem to land in ${PN} not in ${PN}-ptest, you might want to look
> into this also.

That would imply the PACKAGES prepend in ptest.bbclass isn't working. I can't repeat this on my end.

> With your patch, the ptest related debug files did go into the
> ${PN}-dbg, is this what you intend or should they go into a
> ${PN}-ptest-dbg package?

My intention, for now, is to put everything from /usr/lib/${PN}/ptest into ${PN}-ptest. I'm not sure it is a good idea to split ptest packages into multiple packages, because each such split risks catching some test data file and move it into the wrong package. There is no rule or pattern to test data files.

-- 
Björn

Re: ptest results

[Björn Stenberg]

Björn Stenberg wrote:
> Your paste contains some errors that are clearly recipe/framework issues:

I forgot to say that I think we should separate between issues with the ptest framework (bbclass changes) and issues with specific package test suites.

In my view it is critical that we get the framework fixes into 1.4 so that this release can be used as a basis for wider ptest development, while improvements for specific test suites are less critical and can wait if necessary.

-- 
Björn