[Qemu-devel] [PATCH 0/4] docker: Update Travis-CI image to run acceptance tests

[Qemu-devel] [PATCH 0/4] docker: Update Travis-CI image to run acceptance tests

[Philippe Mathieu-Daudé]

This series adds a 'test-acceptance' Docker target, so we can
run 'make docker-test-acceptance@travis' similarly to how the
Travis-CI service runs its builds.

This is particularly useful to debug Travis-CI failures locally.

Regards,

Phil.

Philippe Mathieu-Daudé (4):
  tests/docker: Use one package per line to improve readability
  tests/docker: update our Travis image to run acceptance tests locally
  tests/docker: Add test-acceptance runner
  tests/docker: Use Travis-CI default environment variables

 tests/docker/dockerfiles/travis.docker | 22 ++++++++++++++++++----
 tests/docker/test-acceptance           | 21 +++++++++++++++++++++
 2 files changed, 39 insertions(+), 4 deletions(-)
 create mode 100755 tests/docker/test-acceptance

-- 
2.20.1

[Qemu-devel] [PATCH 1/4] tests/docker: Use one package per line to improve readability

[Philippe Mathieu-Daudé]

Use one package per line to improve readability. This also
helps while reviewing patches.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/docker/dockerfiles/travis.docker | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
index e72dc85ca7..35714664a1 100644
--- a/tests/docker/dockerfiles/travis.docker
+++ b/tests/docker/dockerfiles/travis.docker
@@ -5,7 +5,15 @@ ENV LC_ALL en_US.UTF-8
 RUN sed -i "s/# deb-src/deb-src/" /etc/apt/sources.list
 RUN apt-get update
 RUN apt-get -y build-dep qemu
-RUN apt-get -y install device-tree-compiler python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools gcovr
+RUN apt-get -y install \
+    device-tree-compiler \
+    dh-autoreconf \
+    gcovr \
+    gdb strace \
+    lsof \
+    net-tools \
+    python2.7 \
+    python-yaml
 # Travis tools require PhantomJS / Neo4j / Maven accessible
 # in their PATH (QEMU build won't access them).
 ENV PATH /usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-- 
2.20.1

[Qemu-devel] [PATCH 2/4] tests/docker: update our Travis image to run acceptance tests locally

[Philippe Mathieu-Daudé]

Since commit aa983ff67c3, Travis CI runs acceptance tests using
the Avocado framework. Since Avocado requires Python 3, update
our Docker image to be able to run these tests locally.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/docker/dockerfiles/travis.docker | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
index 35714664a1..b5c5036534 100644
--- a/tests/docker/dockerfiles/travis.docker
+++ b/tests/docker/dockerfiles/travis.docker
@@ -13,6 +13,8 @@ RUN apt-get -y install \
     lsof \
     net-tools \
     python2.7 \
+    python3-pip \
+    python3.5-venv \
     python-yaml
 # Travis tools require PhantomJS / Neo4j / Maven accessible
 # in their PATH (QEMU build won't access them).
-- 
2.20.1

[Qemu-devel] [PATCH 3/4] tests/docker: Add test-acceptance runner

[Philippe Mathieu-Daudé]

Add a runner script to be able to run acceptance tests within
Docker images. We can now reproduce Travis CI builds locally (and
debug  them!).

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/docker/test-acceptance | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100755 tests/docker/test-acceptance

diff --git a/tests/docker/test-acceptance b/tests/docker/test-acceptance
new file mode 100755
index 0000000000..84edaa676c
--- /dev/null
+++ b/tests/docker/test-acceptance
@@ -0,0 +1,21 @@
+#!/bin/bash -e
+#
+# Compile default Travis-CI target and run Avocado acceptance tests
+#
+# Copyright (c) 2019 Red Hat Inc.
+#
+# Authors:
+#  Philippe Mathieu-Daudé <philmd@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+cd "$BUILD_DIR"
+
+DEF_TARGET_LIST="x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
+TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
+build_qemu
+check_qemu check-acceptance
-- 
2.20.1

[Qemu-devel] [PATCH 4/4] tests/docker: Use Travis-CI default environment variables

[Philippe Mathieu-Daudé]

Update to environment variables of our Docker image to closely
reproduce the builds run by the Travis-CI service.

Default variables from:
https://docs.travis-ci.com/user/environment-variables/#default-environment-variables

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/docker/dockerfiles/travis.docker | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
index b5c5036534..a0e8152317 100644
--- a/tests/docker/dockerfiles/travis.docker
+++ b/tests/docker/dockerfiles/travis.docker
@@ -1,7 +1,11 @@
 FROM travisci/ci-sardonyx:packer-1546978056-2c98a19
-ENV DEBIAN_FRONTEND noninteractive
-ENV LANG en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
+ENV CI=true \
+    TRAVIS=true \
+    CONTINUOUS_INTEGRATION=true \
+    DEBIAN_FRONTEND=noninteractive \
+    LANG=en_US.UTF-8 \
+    LC_ALL=en_US.UTF-8 \
+    TRAVIS_OS_NAME=linux
 RUN sed -i "s/# deb-src/deb-src/" /etc/apt/sources.list
 RUN apt-get update
 RUN apt-get -y build-dep qemu
-- 
2.20.1

Re: [Qemu-devel] [PATCH 1/4] tests/docker: Use one package per line to improve readability

[Cleber Rosa]

On Mon, Aug 19, 2019 at 01:18:24AM +0200, Philippe Mathieu-Daudé wrote:
> Use one package per line to improve readability. This also
> helps while reviewing patches.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/docker/dockerfiles/travis.docker | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
> index e72dc85ca7..35714664a1 100644
> --- a/tests/docker/dockerfiles/travis.docker
> +++ b/tests/docker/dockerfiles/travis.docker
> @@ -5,7 +5,15 @@ ENV LC_ALL en_US.UTF-8
>  RUN sed -i "s/# deb-src/deb-src/" /etc/apt/sources.list
>  RUN apt-get update
>  RUN apt-get -y build-dep qemu
> -RUN apt-get -y install device-tree-compiler python2.7 python-yaml dh-autoreconf gdb strace lsof net-tools gcovr
> +RUN apt-get -y install \
> +    device-tree-compiler \
> +    dh-autoreconf \
> +    gcovr \
> +    gdb strace \

Two in a single line here.

> +    lsof \
> +    net-tools \
> +    python2.7 \
> +    python-yaml
>  # Travis tools require PhantomJS / Neo4j / Maven accessible
>  # in their PATH (QEMU build won't access them).
>  ENV PATH /usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> -- 
> 2.20.1
> 
> 

Re: [Qemu-devel] [PATCH 2/4] tests/docker: update our Travis image to run acceptance tests locally

[Cleber Rosa]

On Mon, Aug 19, 2019 at 01:18:25AM +0200, Philippe Mathieu-Daudé wrote:
> Since commit aa983ff67c3, Travis CI runs acceptance tests using
> the Avocado framework. Since Avocado requires Python 3, update
> our Docker image to be able to run these tests locally.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/docker/dockerfiles/travis.docker | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/tests/docker/dockerfiles/travis.docker b/tests/docker/dockerfiles/travis.docker
> index 35714664a1..b5c5036534 100644
> --- a/tests/docker/dockerfiles/travis.docker
> +++ b/tests/docker/dockerfiles/travis.docker
> @@ -13,6 +13,8 @@ RUN apt-get -y install \
>      lsof \
>      net-tools \
>      python2.7 \
> +    python3-pip \
> +    python3.5-venv \
>      python-yaml
>  # Travis tools require PhantomJS / Neo4j / Maven accessible
>  # in their PATH (QEMU build won't access them).
> -- 
> 2.20.1
> 
> 

Tested-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>

Re: [Qemu-devel] [PATCH 3/4] tests/docker: Add test-acceptance runner

[Cleber Rosa]

On Mon, Aug 19, 2019 at 01:18:26AM +0200, Philippe Mathieu-Daudé wrote:
> Add a runner script to be able to run acceptance tests within
> Docker images. We can now reproduce Travis CI builds locally (and
> debug  them!).
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/docker/test-acceptance | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
>  create mode 100755 tests/docker/test-acceptance
> 
> diff --git a/tests/docker/test-acceptance b/tests/docker/test-acceptance
> new file mode 100755
> index 0000000000..84edaa676c
> --- /dev/null
> +++ b/tests/docker/test-acceptance
> @@ -0,0 +1,21 @@
> +#!/bin/bash -e
> +#
> +# Compile default Travis-CI target and run Avocado acceptance tests
> +#
> +# Copyright (c) 2019 Red Hat Inc.
> +#
> +# Authors:
> +#  Philippe Mathieu-Daudé <philmd@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +. common.rc
> +
> +cd "$BUILD_DIR"
> +
> +DEF_TARGET_LIST="x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
> +build_qemu
> +check_qemu check-acceptance
> -- 
> 2.20.1
> 
> 

I'm currently seeing test errors when running in a container:

  MKDIR   /tmp/qemu-test/build/tests/results
  AVOCADO tests/acceptance
JOB ID     : fe56cc0b2d1adbc0b5bb5828902e113d596edccf
JOB LOG    : /tmp/qemu-test/build/tests/results/job-2019-08-19T22.13-fe56cc0/job.log
 (01/27) /tmp/qemu-test/src/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  ERROR: join() argument must be str or bytes, not 'NoneType' (0.05 s)
Interrupting job (failfast).
RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 26 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 0.26 s
/tmp/qemu-test/src/tests/Makefile.include:1158: recipe for target 'check-acceptance' failed
make: *** [check-acceptance] Error 9

That being said, I'm not running it under docker, but under podman,
although I'm not convinced yet that is the defining issue.  I'll try
to identify what's going here.

Cheers,
- Cleber.

Re: [Qemu-devel] [PATCH 3/4] tests/docker: Add test-acceptance runner

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> On Mon, Aug 19, 2019 at 01:18:26AM +0200, Philippe Mathieu-Daudé wrote:
>> Add a runner script to be able to run acceptance tests within
>> Docker images. We can now reproduce Travis CI builds locally (and
>> debug  them!).
>> 
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  tests/docker/test-acceptance | 21 +++++++++++++++++++++
>>  1 file changed, 21 insertions(+)
>>  create mode 100755 tests/docker/test-acceptance
>> 
>> diff --git a/tests/docker/test-acceptance b/tests/docker/test-acceptance
>> new file mode 100755
>> index 0000000000..84edaa676c
>> --- /dev/null
>> +++ b/tests/docker/test-acceptance
>> @@ -0,0 +1,21 @@
>> +#!/bin/bash -e
>> +#
>> +# Compile default Travis-CI target and run Avocado acceptance tests
>> +#
>> +# Copyright (c) 2019 Red Hat Inc.
>> +#
>> +# Authors:
>> +#  Philippe Mathieu-Daudé <philmd@redhat.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2
>> +# or (at your option) any later version. See the COPYING file in
>> +# the top-level directory.
>> +
>> +. common.rc
>> +
>> +cd "$BUILD_DIR"
>> +
>> +DEF_TARGET_LIST="x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
>> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
>> +build_qemu
>> +check_qemu check-acceptance
>> -- 
>> 2.20.1
>> 
>> 
>
> I'm currently seeing test errors when running in a container:
>
>   MKDIR   /tmp/qemu-test/build/tests/results
>   AVOCADO tests/acceptance
> JOB ID     : fe56cc0b2d1adbc0b5bb5828902e113d596edccf
> JOB LOG    : /tmp/qemu-test/build/tests/results/job-2019-08-19T22.13-fe56cc0/job.log
>  (01/27) /tmp/qemu-test/src/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  ERROR: join() argument must be str or bytes, not 'NoneType' (0.05 s)
> Interrupting job (failfast).
> RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 26 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 0.26 s
> /tmp/qemu-test/src/tests/Makefile.include:1158: recipe for target 'check-acceptance' failed
> make: *** [check-acceptance] Error 9
>
> That being said, I'm not running it under docker, but under podman,
> although I'm not convinced yet that is the defining issue.  I'll try
> to identify what's going here.

Was there a conclusion to the discussion about this series?

-- 
Alex Bennée

Re: [Qemu-devel] [PATCH 3/4] tests/docker: Add test-acceptance runner

[Philippe Mathieu-Daudé]

On 12/3/19 2:41 PM, Alex Bennée wrote:
> Cleber Rosa <crosa@redhat.com> writes:
>> On Mon, Aug 19, 2019 at 01:18:26AM +0200, Philippe Mathieu-Daudé wrote:
>>> Add a runner script to be able to run acceptance tests within
>>> Docker images. We can now reproduce Travis CI builds locally (and
>>> debug  them!).
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>> ---
>>>   tests/docker/test-acceptance | 21 +++++++++++++++++++++
>>>   1 file changed, 21 insertions(+)
>>>   create mode 100755 tests/docker/test-acceptance
>>>
>>> diff --git a/tests/docker/test-acceptance b/tests/docker/test-acceptance
>>> new file mode 100755
>>> index 0000000000..84edaa676c
>>> --- /dev/null
>>> +++ b/tests/docker/test-acceptance
>>> @@ -0,0 +1,21 @@
>>> +#!/bin/bash -e
>>> +#
>>> +# Compile default Travis-CI target and run Avocado acceptance tests
>>> +#
>>> +# Copyright (c) 2019 Red Hat Inc.
>>> +#
>>> +# Authors:
>>> +#  Philippe Mathieu-Daudé <philmd@redhat.com>
>>> +#
>>> +# This work is licensed under the terms of the GNU GPL, version 2
>>> +# or (at your option) any later version. See the COPYING file in
>>> +# the top-level directory.
>>> +
>>> +. common.rc
>>> +
>>> +cd "$BUILD_DIR"
>>> +
>>> +DEF_TARGET_LIST="x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
>>> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
>>> +build_qemu
>>> +check_qemu check-acceptance
>>> -- 
>>> 2.20.1
>>>
>>>
>>
>> I'm currently seeing test errors when running in a container:
>>
>>    MKDIR   /tmp/qemu-test/build/tests/results
>>    AVOCADO tests/acceptance
>> JOB ID     : fe56cc0b2d1adbc0b5bb5828902e113d596edccf
>> JOB LOG    : /tmp/qemu-test/build/tests/results/job-2019-08-19T22.13-fe56cc0/job.log
>>   (01/27) /tmp/qemu-test/src/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  ERROR: join() argument must be str or bytes, not 'NoneType' (0.05 s)
>> Interrupting job (failfast).
>> RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 26 | WARN 0 | INTERRUPT 0 | CANCEL 0
>> JOB TIME   : 0.26 s
>> /tmp/qemu-test/src/tests/Makefile.include:1158: recipe for target 'check-acceptance' failed
>> make: *** [check-acceptance] Error 9
>>
>> That being said, I'm not running it under docker, but under podman,
>> although I'm not convinced yet that is the defining issue.  I'll try
>> to identify what's going here.
> 
> Was there a conclusion to the discussion about this series?

No sure if there was a conclusion...

tests/docker/ is listed as 'Maintained' but the listed maintainers only 
use Docker. If Podman is supported, then maybe we need another 
maintainer for Podman? We are not testing it so far.