Help needed with ptest

Help needed with ptest

[Richard Purdie]

We now have better data than ever before on the status of the project
from a test results perspective. In particular we can now easily
collect ptest data.

I triggered the tests for qemux86-64 on the autobuilder yesterday, the
results are published and you can see them with:

wget https://autobuilder.yocto.io/pub/non-release/20190127-9/testresults/qemux86-64-ptest/testresults.json 

To show the raw logs you can quickly hack to make then more readable:

cat testresults.json  | sed 's/\\n/\n/g'  > testresults1.json 

and there are numerous worrying things when you read them such as if
you grep for "TIMEOUT:".

I also have a suspicion that some ptest runners are exiting without
cleaning up all processes resulting in corrupted output later in the
logs too, I saw something like that locally, I think for perl. More
investigation is needed for that.

There are multiple areas that could do with help:

a) investigate/fix the timeouts
b) improve the test results tool Ee Peng has posted to include ptest
processing tools:
 i) view the logs
 ii) compare the results
 iii) highlight timeouts 
 iv) summarise durations
c) Split logs into individual ptest runs instead of one block
d) record the timings for the individual ptest blocks
e) Add timeout status for tests into the json so we can explicitly test
for it.

I'd welcome help from anyone in any of the above areas, we probably
need to create feature enhancement bugs for them.

Cheers,

Richard

Re: Help needed with ptest

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 2159 bytes --]

Hmm... do we want to expand on the "running ptests" section in the
documentation to include some stuff about viewing and analyzing results?
We currently have this section here ...
https://yoctoproject.org/docs/2.7/mega-manual/mega-manual.html#testing-packages-with-ptest

Scott

On Mon, Jan 28, 2019 at 3:39 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> We now have better data than ever before on the status of the project
> from a test results perspective. In particular we can now easily
> collect ptest data.
>
> I triggered the tests for qemux86-64 on the autobuilder yesterday, the
> results are published and you can see them with:
>
> wget
> https://autobuilder.yocto.io/pub/non-release/20190127-9/testresults/qemux86-64-ptest/testresults.json
>
> To show the raw logs you can quickly hack to make then more readable:
>
> cat testresults.json  | sed 's/\\n/\n/g'  > testresults1.json
>
> and there are numerous worrying things when you read them such as if
> you grep for "TIMEOUT:".
>
> I also have a suspicion that some ptest runners are exiting without
> cleaning up all processes resulting in corrupted output later in the
> logs too, I saw something like that locally, I think for perl. More
> investigation is needed for that.
>
> There are multiple areas that could do with help:
>
> a) investigate/fix the timeouts
> b) improve the test results tool Ee Peng has posted to include ptest
> processing tools:
>  i) view the logs
>  ii) compare the results
>  iii) highlight timeouts
>  iv) summarise durations
> c) Split logs into individual ptest runs instead of one block
> d) record the timings for the individual ptest blocks
> e) Add timeout status for tests into the json so we can explicitly test
> for it.
>
> I'd welcome help from anyone in any of the above areas, we probably
> need to create feature enhancement bugs for them.
>
> Cheers,
>
> Richard
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3095 bytes --]

Re: Help needed with ptest

[Richard Purdie]

On Mon, 2019-01-28 at 11:39 +0000, Richard Purdie wrote:
> There are multiple areas that could do with help:
> 
> a) investigate/fix the timeouts
> b) improve the test results tool Ee Peng has posted to include ptest
> processing tools:
>  i) view the logs
>  ii) compare the results
>  iii) highlight timeouts 
>  iv) summarise durations
> c) Split logs into individual ptest runs instead of one block
> d) record the timings for the individual ptest blocks
> e) Add timeout status for tests into the json so we can explicitly
> test for it.

I've now got code to at least help with c/d/e above.

Just to update and document a few things:

a) openssl ptest is broken, it runs but doesn't report the results in
any form that works for us

b) there is a bug in ptest-runner which intermixes stdout and stderr.
Anibal has a patch which should merge shortly

c) lines can get corrupted when logs didn't end with a newline meaning
you'd see DURATION: at the end of a log line. Anibal knows how to fix
and should merge shortly

d) running with 256B memory is too low. This shows as the OOM killer
activating in dmesg. It doesn't show in the ptest runner logs. We
should add checking the logs to the ptest runtime test and increase the
memory for sato-sdk-ptest

e) adding more memory meant the test run took longer but completed more
tests

f) timing data for ptest for qemux86-64 "core-image-sato-sdk-ptest -c
testimage":
5656s for 256MB no SMP
6386s for 1024MB and -smp 4
6803s for 1024MB and no SMP

Cheers,

Richard