[PATCH 0/2] travis: Improve log browseability

[PATCH 0/2] travis: Improve log browseability

[Wainer dos Santos Moschetta]

While I was working on [1] I needed to check what dependencies
were missing on Travis environment to build a kernel. The test was
coded so that a build failure would cancel the test. As a result,
and because all tests pass'ed (or cancel'ed) I could not see
the Avocado logs (consequentely not able to check the missing
packages). That was the main reason for the Patch 01 in this
series. Also it seems pointless to print the log of the tests
that pass.

The patch 02 is a tentative to improve the browsing of build vs
test logs. Honestly I was expecting that Travis was able to
put the output of steps each on toggled text boxes (see the
'before_script' section, each command output has a
toggle). But that isn't the behavior of the 'script'
section. At least it prints the end of the build command
('The command "BUILD_RC=0 && make -j3 || BUILD_RC=$?" exited with 0.')
which can be grep'ed.

Git:
- Tree: https://github.com/wainersm/qemu
- Brach: travis_improved_browse

CI:
- Travis (FAIL): https://travis-ci.org/wainersm/qemu/builds/630948777
  QEMU keep failing with --without-default-devices

References:
[1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg663432.html

Wainer dos Santos Moschetta (2):
  travis.yml: avocado: Print logs of non-pass tests only
  travis.yml: Detach build and test steps

 .travis.yml | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

-- 
2.23.0

[PATCH 1/2] travis.yml: avocado: Print logs of non-pass tests only

[Wainer dos Santos Moschetta]

The acceptance tests build on Travis is configured to print
the entire Avocado's job log in case any test fail. Usually one is
interested on failed tests only though. So this change the Travis
configuration in order to show the log of tests which status is
different from 'PASS' and 'SKIP' only. Note that 'CANCEL'-ed tests
will have the log printed too because it can help to debug some
condition on CI environment which is not being fulfilled.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
---
 .travis.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 376b7d6dfa..e7fdcb238c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -262,8 +262,8 @@ matrix:
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
         - TEST_CMD="make check-acceptance"
-      after_failure:
-        - cat tests/results/latest/job.log
+      after_script:
+        - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
       addons:
         apt:
           packages:
-- 
2.23.0

[PATCH 2/2] travis.yml: Detach build and test steps

[Wainer dos Santos Moschetta]

Currently build and test commands are a single step in a
Travis's `script` block. In order to see the output
of the tests one needs to scroll down the log to find where
the build messages ended and the limit is not clear. If
they were in different steps then Travis would print the
result build command, which can be easily grep'ed.

So this change is made to detach those commands
to ease the visualization of the output.

Note that all steps on the `script` block is executed regardless
if one previous has failed. To overcome it, let's save the
return code of the build then check whether succeed or failed on
the test step.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
---
 .travis.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index e7fdcb238c..fcc9de368b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -94,7 +94,8 @@ before_script:
   - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
   - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
 script:
-  - make -j3 && travis_retry ${TEST_CMD}
+  - BUILD_RC=0 && make -j3 || BUILD_RC=$?
+  - if [ "$BUILD_RC" -eq 0 ] ; then travis_retry ${TEST_CMD} ; else $(exit $BUILD_RC); fi
 after_script:
   - if command -v ccache ; then ccache --show-stats ; fi
 
-- 
2.23.0

Re: [PATCH 1/2] travis.yml: avocado: Print logs of non-pass tests only

[Philippe Mathieu-Daudé]

On 12/30/19 7:43 PM, Wainer dos Santos Moschetta wrote:
> The acceptance tests build on Travis is configured to print
> the entire Avocado's job log in case any test fail. Usually one is
> interested on failed tests only though. So this change the Travis
> configuration in order to show the log of tests which status is
> different from 'PASS' and 'SKIP' only. Note that 'CANCEL'-ed tests
> will have the log printed too because it can help to debug some
> condition on CI environment which is not being fulfilled.

Nice!

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Tested with a broken test:
https://travis-ci.org/philmd/qemu/jobs/631341382

Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> ---
>   .travis.yml | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 376b7d6dfa..e7fdcb238c 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -262,8 +262,8 @@ matrix:
>       - env:
>           - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
>           - TEST_CMD="make check-acceptance"
> -      after_failure:
> -        - cat tests/results/latest/job.log
> +      after_script:
> +        - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
>         addons:
>           apt:
>             packages:
> 

Re: [PATCH 2/2] travis.yml: Detach build and test steps

[Philippe Mathieu-Daudé]

On 12/30/19 7:43 PM, Wainer dos Santos Moschetta wrote:
> Currently build and test commands are a single step in a
> Travis's `script` block. In order to see the output
> of the tests one needs to scroll down the log to find where
> the build messages ended and the limit is not clear. If
> they were in different steps then Travis would print the
> result build command, which can be easily grep'ed.
> 
> So this change is made to detach those commands
> to ease the visualization of the output.
> 
> Note that all steps on the `script` block is executed regardless
> if one previous has failed. To overcome it, let's save the
> return code of the build then check whether succeed or failed on
> the test step.

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Tested with a broken test:
https://travis-ci.org/philmd/qemu/jobs/631341382

Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> 
> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> ---
>   .travis.yml | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index e7fdcb238c..fcc9de368b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -94,7 +94,8 @@ before_script:
>     - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
>     - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
>   script:
> -  - make -j3 && travis_retry ${TEST_CMD}
> +  - BUILD_RC=0 && make -j3 || BUILD_RC=$?
> +  - if [ "$BUILD_RC" -eq 0 ] ; then travis_retry ${TEST_CMD} ; else $(exit $BUILD_RC); fi
>   after_script:
>     - if command -v ccache ; then ccache --show-stats ; fi
>   
> 

Re: [PATCH 1/2] travis.yml: avocado: Print logs of non-pass tests only

[Alex Bennée]

Wainer dos Santos Moschetta <wainersm@redhat.com> writes:

> The acceptance tests build on Travis is configured to print
> the entire Avocado's job log in case any test fail. Usually one is
> interested on failed tests only though. So this change the Travis
> configuration in order to show the log of tests which status is
> different from 'PASS' and 'SKIP' only. Note that 'CANCEL'-ed tests
> will have the log printed too because it can help to debug some
> condition on CI environment which is not being fulfilled.
>
> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> ---
>  .travis.yml | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 376b7d6dfa..e7fdcb238c 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -262,8 +262,8 @@ matrix:
>      - env:
>          - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
>          - TEST_CMD="make check-acceptance"
> -      after_failure:
> -        - cat tests/results/latest/job.log
> +      after_script:
> +        - python3 -c 'import json; r =
> json.load(open("tests/results/latest/results.json"));
> [print(t["logfile"]) for t in r["tests"] if t["status"] not in
> ("PASS", "SKIP")]' | xargs cat

Shame there is no scriptlet for this sort of thing packaged with
avocado. Anyway I'll give it a spin and see if any failures come up.


>        addons:
>          apt:
>            packages:


-- 
Alex Bennée

Re: [PATCH 1/2] travis.yml: avocado: Print logs of non-pass tests only

[Wainer dos Santos Moschetta]

On 1/7/20 9:20 AM, Alex Bennée wrote:
> Wainer dos Santos Moschetta <wainersm@redhat.com> writes:
>
>> The acceptance tests build on Travis is configured to print
>> the entire Avocado's job log in case any test fail. Usually one is
>> interested on failed tests only though. So this change the Travis
>> configuration in order to show the log of tests which status is
>> different from 'PASS' and 'SKIP' only. Note that 'CANCEL'-ed tests
>> will have the log printed too because it can help to debug some
>> condition on CI environment which is not being fulfilled.
>>
>> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>> ---
>>   .travis.yml | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 376b7d6dfa..e7fdcb238c 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -262,8 +262,8 @@ matrix:
>>       - env:
>>           - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
>>           - TEST_CMD="make check-acceptance"
>> -      after_failure:
>> -        - cat tests/results/latest/job.log
>> +      after_script:
>> +        - python3 -c 'import json; r =
>> json.load(open("tests/results/latest/results.json"));
>> [print(t["logfile"]) for t in r["tests"] if t["status"] not in
>> ("PASS", "SKIP")]' | xargs cat
> Shame there is no scriptlet for this sort of thing packaged with
> avocado. Anyway I'll give it a spin and see if any failures come up.

Good point. I'm working on such as scriptlet ('plugin' on Avocado 
jargon) but I think we can stick with this change.

Thanks!

- Wainer

>
>>         addons:
>>           apt:
>>             packages:
>