* [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test @ 2018-05-18 18:34 Alistair Francis 2018-05-21 3:16 ` Fam Zheng 0 siblings, 1 reply; 11+ messages in thread From: Alistair Francis @ 2018-05-18 18:34 UTC (permalink / raw) To: qemu-devel; +Cc: alistair.francis, alex.bennee, f4bug, alistair23, famz Avocado is not trivial to setup on non-Fedora systems. To simplfying future testing add a docker test image that runs Avocado tests. Signed-off-by: Alistair Francis <alistair.francis@wdc.com> --- v2: - Add a seperate fedora-avocado Docker image - Move the avocado vt-bootstrap into the Docker file tests/docker/Makefile.include | 1 + .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ tests/docker/test-avocado | 28 +++++++++++++++++++ 3 files changed, 54 insertions(+) create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker create mode 100755 tests/docker/test-avocado diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include index ef1a3e62eb..0e3d108dde 100644 --- a/tests/docker/Makefile.include +++ b/tests/docker/Makefile.include @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 docker-image-debian-s390x-cross: docker-image-debian9 docker-image-debian-win32-cross: docker-image-debian8-mxe docker-image-debian-win64-cross: docker-image-debian8-mxe +docker-image-fedora-avocado: docker-image-fedora docker-image-travis: NOUSER=1 # Expand all the pre-requistes for each docker image and test combination diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker new file mode 100644 index 0000000000..55b19eebbf --- /dev/null +++ b/tests/docker/dockerfiles/fedora-avocado.docker @@ -0,0 +1,25 @@ +FROM qemu:fedora + +ENV PACKAGES \ + libvirt-devel \ + nc \ + python-avocado \ + python2-devel python3-devel \ + qemu-kvm \ + tcpdump \ + xz +ENV PIP_PACKAGES \ + avocado-qemu \ + avocado-framework-plugin-runner-remote \ + avocado-framework-plugin-runner-vm \ + avocado-framework-plugin-vt + +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 + +RUN dnf install -y $PACKAGES +RUN pip install $PIP_PACKAGES +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu + +RUN rpm -q $PACKAGES | sort > /packages.txt + +ENV FEATURES mingw clang pyyaml asan avocado diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado new file mode 100755 index 0000000000..40474db2ce --- /dev/null +++ b/tests/docker/test-avocado @@ -0,0 +1,28 @@ +#!/bin/bash -e +# +# Avocado tests on Fedora, as these are a real pain on Debian systems +# +# Copyright (c) 2018 Western Digital. +# +# Authors: +# Alistair Francis <alistair.francis@wdc.com> +# +# This work is licensed under the terms of the GNU GPL, version 2 +# or (at your option) any later version. See the COPYING file in +# the top-level directory. +# +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado + +. common.rc + +requires avocado + +cd "$BUILD_DIR" + +DEF_TARGET_LIST="x86_64-softmmu" +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ +build_qemu +install_qemu + +export PATH="${PATH}:$(pwd)" +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 -- 2.17.0 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-18 18:34 [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test Alistair Francis @ 2018-05-21 3:16 ` Fam Zheng 2018-05-21 17:26 ` Philippe Mathieu-Daudé 2018-05-21 22:33 ` Alistair Francis 0 siblings, 2 replies; 11+ messages in thread From: Fam Zheng @ 2018-05-21 3:16 UTC (permalink / raw) To: Alistair Francis; +Cc: qemu-devel, alex.bennee, f4bug, alistair23, crosa On Fri, 05/18 11:34, Alistair Francis wrote: > Avocado is not trivial to setup on non-Fedora systems. To simplfying > future testing add a docker test image that runs Avocado tests. > > Signed-off-by: Alistair Francis <alistair.francis@wdc.com> > --- > v2: > - Add a seperate fedora-avocado Docker image > - Move the avocado vt-bootstrap into the Docker file > > tests/docker/Makefile.include | 1 + > .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ > tests/docker/test-avocado | 28 +++++++++++++++++++ > 3 files changed, 54 insertions(+) > create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker > create mode 100755 tests/docker/test-avocado > > diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include > index ef1a3e62eb..0e3d108dde 100644 > --- a/tests/docker/Makefile.include > +++ b/tests/docker/Makefile.include > @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 > docker-image-debian-s390x-cross: docker-image-debian9 > docker-image-debian-win32-cross: docker-image-debian8-mxe > docker-image-debian-win64-cross: docker-image-debian8-mxe > +docker-image-fedora-avocado: docker-image-fedora > docker-image-travis: NOUSER=1 > > # Expand all the pre-requistes for each docker image and test combination > diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker > new file mode 100644 > index 0000000000..55b19eebbf > --- /dev/null > +++ b/tests/docker/dockerfiles/fedora-avocado.docker > @@ -0,0 +1,25 @@ > +FROM qemu:fedora > + > +ENV PACKAGES \ > + libvirt-devel \ > + nc \ > + python-avocado \ > + python2-devel python3-devel \ > + qemu-kvm \ > + tcpdump \ > + xz > +ENV PIP_PACKAGES \ > + avocado-qemu \ > + avocado-framework-plugin-runner-remote \ > + avocado-framework-plugin-runner-vm \ > + avocado-framework-plugin-vt > + > +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 I think this is inherited from qemu:fedora, no? > + > +RUN dnf install -y $PACKAGES > +RUN pip install $PIP_PACKAGES > +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu > + > +RUN rpm -q $PACKAGES | sort > /packages.txt Can you keep the parent image's list with ">>" or appending to the old $PACKAGES in the above "ENV" directive? > + > +ENV FEATURES mingw clang pyyaml asan avocado Similarly, is it possible to append to the parent list instead of overriding? > diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado > new file mode 100755 > index 0000000000..40474db2ce > --- /dev/null > +++ b/tests/docker/test-avocado > @@ -0,0 +1,28 @@ > +#!/bin/bash -e > +# > +# Avocado tests on Fedora, as these are a real pain on Debian systems Shouldn't pip packages work just well on Debian too? What are the pain? (Cc'ing Cleber who may want to know this). Fam > +# > +# Copyright (c) 2018 Western Digital. > +# > +# Authors: > +# Alistair Francis <alistair.francis@wdc.com> > +# > +# This work is licensed under the terms of the GNU GPL, version 2 > +# or (at your option) any later version. See the COPYING file in > +# the top-level directory. > +# > +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado > + > +. common.rc > + > +requires avocado > + > +cd "$BUILD_DIR" > + > +DEF_TARGET_LIST="x86_64-softmmu" > +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ > +build_qemu > +install_qemu > + > +export PATH="${PATH}:$(pwd)" > +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 > -- > 2.17.0 > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-21 3:16 ` Fam Zheng @ 2018-05-21 17:26 ` Philippe Mathieu-Daudé 2018-05-21 22:37 ` Alistair Francis 2018-05-21 22:33 ` Alistair Francis 1 sibling, 1 reply; 11+ messages in thread From: Philippe Mathieu-Daudé @ 2018-05-21 17:26 UTC (permalink / raw) To: Fam Zheng, Alistair Francis; +Cc: qemu-devel, alex.bennee, alistair23, crosa Hi Alistair, Fam, On 05/21/2018 12:16 AM, Fam Zheng wrote: > On Fri, 05/18 11:34, Alistair Francis wrote: >> Avocado is not trivial to setup on non-Fedora systems. To simplfying >> future testing add a docker test image that runs Avocado tests. Can you add an entry in the "make docker" help menu? >> >> Signed-off-by: Alistair Francis <alistair.francis@wdc.com> >> --- >> v2: >> - Add a seperate fedora-avocado Docker image >> - Move the avocado vt-bootstrap into the Docker file >> >> tests/docker/Makefile.include | 1 + >> .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ >> tests/docker/test-avocado | 28 +++++++++++++++++++ >> 3 files changed, 54 insertions(+) >> create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker >> create mode 100755 tests/docker/test-avocado >> >> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >> index ef1a3e62eb..0e3d108dde 100644 >> --- a/tests/docker/Makefile.include >> +++ b/tests/docker/Makefile.include >> @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >> docker-image-debian-s390x-cross: docker-image-debian9 >> docker-image-debian-win32-cross: docker-image-debian8-mxe >> docker-image-debian-win64-cross: docker-image-debian8-mxe >> +docker-image-fedora-avocado: docker-image-fedora >> docker-image-travis: NOUSER=1 >> >> # Expand all the pre-requistes for each docker image and test combination >> diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker >> new file mode 100644 >> index 0000000000..55b19eebbf >> --- /dev/null >> +++ b/tests/docker/dockerfiles/fedora-avocado.docker >> @@ -0,0 +1,25 @@ >> +FROM qemu:fedora >> + >> +ENV PACKAGES \ >> + libvirt-devel \ >> + nc \ >> + python-avocado \ >> + python2-devel python3-devel \ >> + qemu-kvm \ >> + tcpdump \ >> + xz >> +ENV PIP_PACKAGES \ >> + avocado-qemu \ >> + avocado-framework-plugin-runner-remote \ >> + avocado-framework-plugin-runner-vm \ >> + avocado-framework-plugin-vt >> + >> +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 > > I think this is inherited from qemu:fedora, no? Yes. > >> + >> +RUN dnf install -y $PACKAGES >> +RUN pip install $PIP_PACKAGES >> +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >> + >> +RUN rpm -q $PACKAGES | sort > /packages.txt > > Can you keep the parent image's list with ">>" or appending to the old $PACKAGES > in the above "ENV" directive? Appending looks cleaner to me. > >> + >> +ENV FEATURES mingw clang pyyaml asan avocado > > Similarly, is it possible to append to the parent list instead of overriding? > >> diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado >> new file mode 100755 >> index 0000000000..40474db2ce >> --- /dev/null >> +++ b/tests/docker/test-avocado >> @@ -0,0 +1,28 @@ >> +#!/bin/bash -e >> +# >> +# Avocado tests on Fedora, as these are a real pain on Debian systems > > Shouldn't pip packages work just well on Debian too? What are the pain? > (Cc'ing Cleber who may want to know this). Avocado isn't packaged (yet?) on Debian. > > Fam > >> +# >> +# Copyright (c) 2018 Western Digital. >> +# >> +# Authors: >> +# Alistair Francis <alistair.francis@wdc.com> >> +# >> +# This work is licensed under the terms of the GNU GPL, version 2 >> +# or (at your option) any later version. See the COPYING file in >> +# the top-level directory. >> +# >> +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado >> + >> +. common.rc >> + >> +requires avocado >> + >> +cd "$BUILD_DIR" >> + >> +DEF_TARGET_LIST="x86_64-softmmu" >> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ >> +build_qemu >> +install_qemu >> + >> +export PATH="${PATH}:$(pwd)" >> +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 This failed when testing (I suppose due to too old corporate proxy...): Step 7/11 : RUN avocado vt-bootstrap --yes-to-all --vt-type qemu ---> Running in 008e494971c7 [...] 8 - Verifying (and possibly downloading) guest image Verifying expected SHA1 sum from http://avocado-project.org/data/assets/jeos/27/SHA1SUM_JEOS_27_64 Failed to get SHA1 from file: HTTP Error 403: Forbidden file type or location: http://avocado-project.org/data/assets/jeos/27/SHA1SUM_JEOS_27_64 File /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz not found Check your internet connection: HTTP Error 403: Forbidden file type or location: http://avocado-project.org/data/assets/jeos/27/jeos-27-64.qcow2.xz Uncompressing /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz -> /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2 Bootstrap command failed Command: xz -cd /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz > /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2 stderr output: xz: /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz: File format not recognized The command '/bin/sh -c avocado vt-bootstrap --yes-to-all --vt-type qemu' returned a non-zero code: 1 Traceback (most recent call last): File "./tests/docker/docker.py", line 407, in <module> sys.exit(main()) File "./tests/docker/docker.py", line 404, in main return args.cmdobj.run(args, argv) File "./tests/docker/docker.py", line 326, in run extra_files_cksum=cksum) File "./tests/docker/docker.py", line 209, in build_image quiet=quiet) File "./tests/docker/docker.py", line 147, in _do_check return subprocess.check_call(self._command + cmd, **kwargs) File "/usr/lib/python2.7/subprocess.py", line 190, in check_call raise CalledProcessError(retcode, cmd) make: *** [tests/docker/Makefile.include:37: docker-image-fedora-avocado] Error 1 We said we should use some userdir volume to cache the data/assets/, 1/ to avoid to keep downloading this file again and again, 2/ to share it between different docker instances, 3/ so I can download it via another way and put it there (proxy). BTW Fam (unrelated) when building the intermediary Fedora image I hit: [...] (562/563): man-pages-4.12-1.fc27.noarch.rpm 270 kB/s | 5.9 MB 00:22 (563/563): glibc-common-2.26-27.fc27.x86_64.rpm 62 kB/s | 789 kB 00:12 -------------------------------------------------------------------------------- Total 10 kB/s | 425 MB 709:02 Running transaction check Transaction check succeeded. [...] Error: Transaction check error: installing package glibc-common-2.26-27.fc27.x86_64 needs 4MB on the / filesystem installing package dconf-0.26.1-1.fc27.x86_64 needs 2016MB on the / filesystem installing package man-pages-4.12-1.fc27.noarch needs 2018MB on the / filesystem Error Summary ------------- Disk Requirements: At least 2018MB more space needed on the / filesystem. subprocess.CalledProcessError: Command '['docker', 'build', '-t', 'qemu:fedora', '-f', '/tmp/docker_buildd0ajZ0/tmp1jgnS6.docker', '/tmp/docker_buildd0ajZ0']' returned non-zero exit status 1 make: *** [tests/docker/Makefile.include:37: docker-image-fedora] Error 1 I wonder if we should add an estimated size of image, to avoid (like in my case, wait another night to test this due to my ****** internet connection). Regards, Phil. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-21 17:26 ` Philippe Mathieu-Daudé @ 2018-05-21 22:37 ` Alistair Francis 2018-05-21 23:29 ` Philippe Mathieu-Daudé 0 siblings, 1 reply; 11+ messages in thread From: Alistair Francis @ 2018-05-21 22:37 UTC (permalink / raw) To: Philippe Mathieu-Daudé Cc: Fam Zheng, Alistair Francis, qemu-devel@nongnu.org Developers, Alex Bennée, Cleber Rosa On Mon, May 21, 2018 at 10:26 AM, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote: > Hi Alistair, Fam, > > On 05/21/2018 12:16 AM, Fam Zheng wrote: >> On Fri, 05/18 11:34, Alistair Francis wrote: >>> Avocado is not trivial to setup on non-Fedora systems. To simplfying >>> future testing add a docker test image that runs Avocado tests. > > Can you add an entry in the "make docker" help menu? The one in tests/docker/Makefile.include? It seems like it's mostly auto generated. What do you think I should add? > >>> >>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com> >>> --- >>> v2: >>> - Add a seperate fedora-avocado Docker image >>> - Move the avocado vt-bootstrap into the Docker file >>> >>> tests/docker/Makefile.include | 1 + >>> .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ >>> tests/docker/test-avocado | 28 +++++++++++++++++++ >>> 3 files changed, 54 insertions(+) >>> create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker >>> create mode 100755 tests/docker/test-avocado >>> >>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >>> index ef1a3e62eb..0e3d108dde 100644 >>> --- a/tests/docker/Makefile.include >>> +++ b/tests/docker/Makefile.include >>> @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >>> docker-image-debian-s390x-cross: docker-image-debian9 >>> docker-image-debian-win32-cross: docker-image-debian8-mxe >>> docker-image-debian-win64-cross: docker-image-debian8-mxe >>> +docker-image-fedora-avocado: docker-image-fedora >>> docker-image-travis: NOUSER=1 >>> >>> # Expand all the pre-requistes for each docker image and test combination >>> diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker >>> new file mode 100644 >>> index 0000000000..55b19eebbf >>> --- /dev/null >>> +++ b/tests/docker/dockerfiles/fedora-avocado.docker >>> @@ -0,0 +1,25 @@ >>> +FROM qemu:fedora >>> + >>> +ENV PACKAGES \ >>> + libvirt-devel \ >>> + nc \ >>> + python-avocado \ >>> + python2-devel python3-devel \ >>> + qemu-kvm \ >>> + tcpdump \ >>> + xz >>> +ENV PIP_PACKAGES \ >>> + avocado-qemu \ >>> + avocado-framework-plugin-runner-remote \ >>> + avocado-framework-plugin-runner-vm \ >>> + avocado-framework-plugin-vt >>> + >>> +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 >> >> I think this is inherited from qemu:fedora, no? > > Yes. > >> >>> + >>> +RUN dnf install -y $PACKAGES >>> +RUN pip install $PIP_PACKAGES >>> +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >>> + >>> +RUN rpm -q $PACKAGES | sort > /packages.txt >> >> Can you keep the parent image's list with ">>" or appending to the old $PACKAGES >> in the above "ENV" directive? > > Appending looks cleaner to me. > >> >>> + >>> +ENV FEATURES mingw clang pyyaml asan avocado >> >> Similarly, is it possible to append to the parent list instead of overriding? >> >>> diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado >>> new file mode 100755 >>> index 0000000000..40474db2ce >>> --- /dev/null >>> +++ b/tests/docker/test-avocado >>> @@ -0,0 +1,28 @@ >>> +#!/bin/bash -e >>> +# >>> +# Avocado tests on Fedora, as these are a real pain on Debian systems >> >> Shouldn't pip packages work just well on Debian too? What are the pain? >> (Cc'ing Cleber who may want to know this). > > Avocado isn't packaged (yet?) on Debian. > >> >> Fam >> >>> +# >>> +# Copyright (c) 2018 Western Digital. >>> +# >>> +# Authors: >>> +# Alistair Francis <alistair.francis@wdc.com> >>> +# >>> +# This work is licensed under the terms of the GNU GPL, version 2 >>> +# or (at your option) any later version. See the COPYING file in >>> +# the top-level directory. >>> +# >>> +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado >>> + >>> +. common.rc >>> + >>> +requires avocado >>> + >>> +cd "$BUILD_DIR" >>> + >>> +DEF_TARGET_LIST="x86_64-softmmu" >>> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ >>> +build_qemu >>> +install_qemu >>> + >>> +export PATH="${PATH}:$(pwd)" >>> +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 > > This failed when testing (I suppose due to too old corporate proxy...): > > Step 7/11 : RUN avocado vt-bootstrap --yes-to-all --vt-type qemu > ---> Running in 008e494971c7 > [...] > 8 - Verifying (and possibly downloading) guest image > Verifying expected SHA1 sum from > http://avocado-project.org/data/assets/jeos/27/SHA1SUM_JEOS_27_64 > Failed to get SHA1 from file: HTTP Error 403: Forbidden file type or > location: http://avocado-project.org/data/assets/jeos/27/SHA1SUM_JEOS_27_64 > File /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz not > found > Check your internet connection: HTTP Error 403: Forbidden file type or > location: http://avocado-project.org/data/assets/jeos/27/jeos-27-64.qcow2.xz > Uncompressing > /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz -> > /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2 > Bootstrap command failed > Command: xz -cd > /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz > > /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2 > stderr output: > xz: /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz: > File format not recognized > > The command '/bin/sh -c avocado vt-bootstrap --yes-to-all --vt-type > qemu' returned a non-zero code: 1 > Traceback (most recent call last): > File "./tests/docker/docker.py", line 407, in <module> > sys.exit(main()) > File "./tests/docker/docker.py", line 404, in main > return args.cmdobj.run(args, argv) > File "./tests/docker/docker.py", line 326, in run > extra_files_cksum=cksum) > File "./tests/docker/docker.py", line 209, in build_image > quiet=quiet) > File "./tests/docker/docker.py", line 147, in _do_check > return subprocess.check_call(self._command + cmd, **kwargs) > File "/usr/lib/python2.7/subprocess.py", line 190, in check_call > raise CalledProcessError(retcode, cmd) > make: *** [tests/docker/Makefile.include:37: > docker-image-fedora-avocado] Error 1 Yep, this is a proxy problem. I had the same problem, I had to open the page in my browser to click through the proxy warning before it would work. On the plus side we worked with Palo Alto Networks to unblock the link. Hopefully you can do the same with whatever your company uses. > > We said we should use some userdir volume to cache the data/assets/, > 1/ to avoid to keep downloading this file again and again, > 2/ to share it between different docker instances, > 3/ so I can download it via another way and put it there (proxy). Ok! How do you think this should be done? Alistair > > BTW Fam (unrelated) when building the intermediary Fedora image I hit: > > [...] > (562/563): man-pages-4.12-1.fc27.noarch.rpm 270 kB/s | 5.9 MB > 00:22 > (563/563): glibc-common-2.26-27.fc27.x86_64.rpm 62 kB/s | 789 kB > 00:12 > -------------------------------------------------------------------------------- > Total 10 kB/s | 425 MB > 709:02 > Running transaction check > Transaction check succeeded. > [...] > Error: Transaction check error: > installing package glibc-common-2.26-27.fc27.x86_64 needs 4MB on the / > filesystem > installing package dconf-0.26.1-1.fc27.x86_64 needs 2016MB on the / > filesystem > installing package man-pages-4.12-1.fc27.noarch needs 2018MB on the / > filesystem > Error Summary > ------------- > Disk Requirements: > At least 2018MB more space needed on the / filesystem. > subprocess.CalledProcessError: Command '['docker', 'build', '-t', > 'qemu:fedora', '-f', '/tmp/docker_buildd0ajZ0/tmp1jgnS6.docker', > '/tmp/docker_buildd0ajZ0']' returned non-zero exit status 1 > make: *** [tests/docker/Makefile.include:37: docker-image-fedora] Error 1 > > I wonder if we should add an estimated size of image, to avoid (like in > my case, wait another night to test this due to my ****** internet > connection). > > Regards, > > Phil. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-21 22:37 ` Alistair Francis @ 2018-05-21 23:29 ` Philippe Mathieu-Daudé 0 siblings, 0 replies; 11+ messages in thread From: Philippe Mathieu-Daudé @ 2018-05-21 23:29 UTC (permalink / raw) To: Alistair Francis Cc: Fam Zheng, Alistair Francis, qemu-devel@nongnu.org Developers, Alex Bennée, Cleber Rosa On 05/21/2018 07:37 PM, Alistair Francis wrote: > On Mon, May 21, 2018 at 10:26 AM, Philippe Mathieu-Daudé > <f4bug@amsat.org> wrote: >> Hi Alistair, Fam, >> >> On 05/21/2018 12:16 AM, Fam Zheng wrote: >>> On Fri, 05/18 11:34, Alistair Francis wrote: >>>> Avocado is not trivial to setup on non-Fedora systems. To simplfying >>>> future testing add a docker test image that runs Avocado tests. >> >> Can you add an entry in the "make docker" help menu? > > The one in tests/docker/Makefile.include? It seems like it's mostly > auto generated. What do you think I should add? Ah you right, that's fine then: $ make docker [...] Available tests: test-block test-debug test-clang test-build test-full test-avocado test-mingw test-quick > >> >>>> >>>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com> >>>> --- >>>> v2: >>>> - Add a seperate fedora-avocado Docker image >>>> - Move the avocado vt-bootstrap into the Docker file >>>> >>>> tests/docker/Makefile.include | 1 + >>>> .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ >>>> tests/docker/test-avocado | 28 +++++++++++++++++++ >>>> 3 files changed, 54 insertions(+) >>>> create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker >>>> create mode 100755 tests/docker/test-avocado >>>> >>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >>>> index ef1a3e62eb..0e3d108dde 100644 >>>> --- a/tests/docker/Makefile.include >>>> +++ b/tests/docker/Makefile.include >>>> @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >>>> docker-image-debian-s390x-cross: docker-image-debian9 >>>> docker-image-debian-win32-cross: docker-image-debian8-mxe >>>> docker-image-debian-win64-cross: docker-image-debian8-mxe >>>> +docker-image-fedora-avocado: docker-image-fedora >>>> docker-image-travis: NOUSER=1 >>>> >>>> # Expand all the pre-requistes for each docker image and test combination >>>> diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker >>>> new file mode 100644 >>>> index 0000000000..55b19eebbf >>>> --- /dev/null >>>> +++ b/tests/docker/dockerfiles/fedora-avocado.docker >>>> @@ -0,0 +1,25 @@ >>>> +FROM qemu:fedora >>>> + >>>> +ENV PACKAGES \ >>>> + libvirt-devel \ >>>> + nc \ >>>> + python-avocado \ >>>> + python2-devel python3-devel \ >>>> + qemu-kvm \ >>>> + tcpdump \ >>>> + xz >>>> +ENV PIP_PACKAGES \ >>>> + avocado-qemu \ >>>> + avocado-framework-plugin-runner-remote \ >>>> + avocado-framework-plugin-runner-vm \ >>>> + avocado-framework-plugin-vt >>>> + >>>> +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 >>> >>> I think this is inherited from qemu:fedora, no? >> >> Yes. >> >>> >>>> + >>>> +RUN dnf install -y $PACKAGES >>>> +RUN pip install $PIP_PACKAGES >>>> +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >>>> + >>>> +RUN rpm -q $PACKAGES | sort > /packages.txt >>> >>> Can you keep the parent image's list with ">>" or appending to the old $PACKAGES >>> in the above "ENV" directive? >> >> Appending looks cleaner to me. >> >>> >>>> + >>>> +ENV FEATURES mingw clang pyyaml asan avocado >>> >>> Similarly, is it possible to append to the parent list instead of overriding? >>> >>>> diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado >>>> new file mode 100755 >>>> index 0000000000..40474db2ce >>>> --- /dev/null >>>> +++ b/tests/docker/test-avocado >>>> @@ -0,0 +1,28 @@ >>>> +#!/bin/bash -e >>>> +# >>>> +# Avocado tests on Fedora, as these are a real pain on Debian systems >>> >>> Shouldn't pip packages work just well on Debian too? What are the pain? >>> (Cc'ing Cleber who may want to know this). >> >> Avocado isn't packaged (yet?) on Debian. >> >>> >>> Fam >>> >>>> +# >>>> +# Copyright (c) 2018 Western Digital. >>>> +# >>>> +# Authors: >>>> +# Alistair Francis <alistair.francis@wdc.com> >>>> +# >>>> +# This work is licensed under the terms of the GNU GPL, version 2 >>>> +# or (at your option) any later version. See the COPYING file in >>>> +# the top-level directory. >>>> +# >>>> +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado >>>> + >>>> +. common.rc >>>> + >>>> +requires avocado >>>> + >>>> +cd "$BUILD_DIR" >>>> + >>>> +DEF_TARGET_LIST="x86_64-softmmu" >>>> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ >>>> +build_qemu >>>> +install_qemu >>>> + >>>> +export PATH="${PATH}:$(pwd)" >>>> +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 >> >> This failed when testing (I suppose due to too old corporate proxy...): >> >> Step 7/11 : RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >> ---> Running in 008e494971c7 >> [...] >> 8 - Verifying (and possibly downloading) guest image >> Verifying expected SHA1 sum from >> http://avocado-project.org/data/assets/jeos/27/SHA1SUM_JEOS_27_64 >> Failed to get SHA1 from file: HTTP Error 403: Forbidden file type or >> location: http://avocado-project.org/data/assets/jeos/27/SHA1SUM_JEOS_27_64 >> File /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz not >> found >> Check your internet connection: HTTP Error 403: Forbidden file type or >> location: http://avocado-project.org/data/assets/jeos/27/jeos-27-64.qcow2.xz >> Uncompressing >> /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz -> >> /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2 >> Bootstrap command failed >> Command: xz -cd >> /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz > >> /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2 >> stderr output: >> xz: /var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2.xz: >> File format not recognized >> >> The command '/bin/sh -c avocado vt-bootstrap --yes-to-all --vt-type >> qemu' returned a non-zero code: 1 >> Traceback (most recent call last): >> File "./tests/docker/docker.py", line 407, in <module> >> sys.exit(main()) >> File "./tests/docker/docker.py", line 404, in main >> return args.cmdobj.run(args, argv) >> File "./tests/docker/docker.py", line 326, in run >> extra_files_cksum=cksum) >> File "./tests/docker/docker.py", line 209, in build_image >> quiet=quiet) >> File "./tests/docker/docker.py", line 147, in _do_check >> return subprocess.check_call(self._command + cmd, **kwargs) >> File "/usr/lib/python2.7/subprocess.py", line 190, in check_call >> raise CalledProcessError(retcode, cmd) >> make: *** [tests/docker/Makefile.include:37: >> docker-image-fedora-avocado] Error 1 > > Yep, this is a proxy problem. > > I had the same problem, I had to open the page in my browser to click > through the proxy warning before it would work. > > On the plus side we worked with Palo Alto Networks to unblock the > link. Hopefully you can do the same with whatever your company uses. > >> >> We said we should use some userdir volume to cache the data/assets/, >> 1/ to avoid to keep downloading this file again and again, >> 2/ to share it between different docker instances, >> 3/ so I can download it via another way and put it there (proxy). > > Ok! How do you think this should be done? Look at tests/vm/basevm.py _download_with_cache(), Fam uses: ... cache_dir = os.path.expanduser("~/.cache/qemu-vm/download") if not os.path.exists(cache_dir): os.makedirs(cache_dir) fname = os.path.join(cache_dir, hashlib.sha1(url).hexdigest()) ... Avocado already has a caching API: http://avocado-framework.readthedocs.io/en/57.0/WritingTests.html#fetching-asset-files We could add something like this in tests/docker/Makefile.include: docker-image-fedora-avocado: EXTRA_VOLUMES += ~/.cache/qemu-vm/download:/avocado/data/cache (we already use $DOCKER_CCACHE_DIR). This is a follow up patch anyway. > > Alistair > >> >> BTW Fam (unrelated) when building the intermediary Fedora image I hit: >> >> [...] >> (562/563): man-pages-4.12-1.fc27.noarch.rpm 270 kB/s | 5.9 MB >> 00:22 >> (563/563): glibc-common-2.26-27.fc27.x86_64.rpm 62 kB/s | 789 kB >> 00:12 >> -------------------------------------------------------------------------------- >> Total 10 kB/s | 425 MB >> 709:02 >> Running transaction check >> Transaction check succeeded. >> [...] >> Error: Transaction check error: >> installing package glibc-common-2.26-27.fc27.x86_64 needs 4MB on the / >> filesystem >> installing package dconf-0.26.1-1.fc27.x86_64 needs 2016MB on the / >> filesystem >> installing package man-pages-4.12-1.fc27.noarch needs 2018MB on the / >> filesystem >> Error Summary >> ------------- >> Disk Requirements: >> At least 2018MB more space needed on the / filesystem. >> subprocess.CalledProcessError: Command '['docker', 'build', '-t', >> 'qemu:fedora', '-f', '/tmp/docker_buildd0ajZ0/tmp1jgnS6.docker', >> '/tmp/docker_buildd0ajZ0']' returned non-zero exit status 1 >> make: *** [tests/docker/Makefile.include:37: docker-image-fedora] Error 1 >> >> I wonder if we should add an estimated size of image, to avoid (like in >> my case, wait another night to test this due to my ****** internet >> connection). >> >> Regards, >> >> Phil. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-21 3:16 ` Fam Zheng 2018-05-21 17:26 ` Philippe Mathieu-Daudé @ 2018-05-21 22:33 ` Alistair Francis 2018-05-21 23:32 ` Philippe Mathieu-Daudé 1 sibling, 1 reply; 11+ messages in thread From: Alistair Francis @ 2018-05-21 22:33 UTC (permalink / raw) To: Fam Zheng Cc: Alistair Francis, qemu-devel@nongnu.org Developers, Alex Bennée, Philippe Mathieu-Daudé, Cleber Rosa On Sun, May 20, 2018 at 8:16 PM, Fam Zheng <famz@redhat.com> wrote: > On Fri, 05/18 11:34, Alistair Francis wrote: >> Avocado is not trivial to setup on non-Fedora systems. To simplfying >> future testing add a docker test image that runs Avocado tests. >> >> Signed-off-by: Alistair Francis <alistair.francis@wdc.com> >> --- >> v2: >> - Add a seperate fedora-avocado Docker image >> - Move the avocado vt-bootstrap into the Docker file >> >> tests/docker/Makefile.include | 1 + >> .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ >> tests/docker/test-avocado | 28 +++++++++++++++++++ >> 3 files changed, 54 insertions(+) >> create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker >> create mode 100755 tests/docker/test-avocado >> >> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >> index ef1a3e62eb..0e3d108dde 100644 >> --- a/tests/docker/Makefile.include >> +++ b/tests/docker/Makefile.include >> @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >> docker-image-debian-s390x-cross: docker-image-debian9 >> docker-image-debian-win32-cross: docker-image-debian8-mxe >> docker-image-debian-win64-cross: docker-image-debian8-mxe >> +docker-image-fedora-avocado: docker-image-fedora >> docker-image-travis: NOUSER=1 >> >> # Expand all the pre-requistes for each docker image and test combination >> diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker >> new file mode 100644 >> index 0000000000..55b19eebbf >> --- /dev/null >> +++ b/tests/docker/dockerfiles/fedora-avocado.docker >> @@ -0,0 +1,25 @@ >> +FROM qemu:fedora >> + >> +ENV PACKAGES \ >> + libvirt-devel \ >> + nc \ >> + python-avocado \ >> + python2-devel python3-devel \ >> + qemu-kvm \ >> + tcpdump \ >> + xz >> +ENV PIP_PACKAGES \ >> + avocado-qemu \ >> + avocado-framework-plugin-runner-remote \ >> + avocado-framework-plugin-runner-vm \ >> + avocado-framework-plugin-vt >> + >> +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 > > I think this is inherited from qemu:fedora, no? It is, I have removed it. > >> + >> +RUN dnf install -y $PACKAGES >> +RUN pip install $PIP_PACKAGES >> +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >> + >> +RUN rpm -q $PACKAGES | sort > /packages.txt > > Can you keep the parent image's list with ">>" or appending to the old $PACKAGES > in the above "ENV" directive? > >> + >> +ENV FEATURES mingw clang pyyaml asan avocado > > Similarly, is it possible to append to the parent list instead of overriding? I have changed both of these to append to the original variables. > >> diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado >> new file mode 100755 >> index 0000000000..40474db2ce >> --- /dev/null >> +++ b/tests/docker/test-avocado >> @@ -0,0 +1,28 @@ >> +#!/bin/bash -e >> +# >> +# Avocado tests on Fedora, as these are a real pain on Debian systems > > Shouldn't pip packages work just well on Debian too? What are the pain? > (Cc'ing Cleber who may want to know this). There is no debian package at the moment. Alistair > > Fam > >> +# >> +# Copyright (c) 2018 Western Digital. >> +# >> +# Authors: >> +# Alistair Francis <alistair.francis@wdc.com> >> +# >> +# This work is licensed under the terms of the GNU GPL, version 2 >> +# or (at your option) any later version. See the COPYING file in >> +# the top-level directory. >> +# >> +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado >> + >> +. common.rc >> + >> +requires avocado >> + >> +cd "$BUILD_DIR" >> + >> +DEF_TARGET_LIST="x86_64-softmmu" >> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ >> +build_qemu >> +install_qemu >> + >> +export PATH="${PATH}:$(pwd)" >> +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 >> -- >> 2.17.0 >> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-21 22:33 ` Alistair Francis @ 2018-05-21 23:32 ` Philippe Mathieu-Daudé 2018-05-22 19:50 ` Cleber Rosa 0 siblings, 1 reply; 11+ messages in thread From: Philippe Mathieu-Daudé @ 2018-05-21 23:32 UTC (permalink / raw) To: Alistair Francis, Fam Zheng Cc: Alistair Francis, qemu-devel@nongnu.org Developers, Alex Bennée, Cleber Rosa On 05/21/2018 07:33 PM, Alistair Francis wrote: > On Sun, May 20, 2018 at 8:16 PM, Fam Zheng <famz@redhat.com> wrote: >> On Fri, 05/18 11:34, Alistair Francis wrote: >>> Avocado is not trivial to setup on non-Fedora systems. To simplfying >>> future testing add a docker test image that runs Avocado tests. >>> >>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com> >>> --- >>> v2: >>> - Add a seperate fedora-avocado Docker image >>> - Move the avocado vt-bootstrap into the Docker file >>> >>> tests/docker/Makefile.include | 1 + >>> .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ >>> tests/docker/test-avocado | 28 +++++++++++++++++++ >>> 3 files changed, 54 insertions(+) >>> create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker >>> create mode 100755 tests/docker/test-avocado >>> >>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >>> index ef1a3e62eb..0e3d108dde 100644 >>> --- a/tests/docker/Makefile.include >>> +++ b/tests/docker/Makefile.include >>> @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >>> docker-image-debian-s390x-cross: docker-image-debian9 >>> docker-image-debian-win32-cross: docker-image-debian8-mxe >>> docker-image-debian-win64-cross: docker-image-debian8-mxe >>> +docker-image-fedora-avocado: docker-image-fedora >>> docker-image-travis: NOUSER=1 >>> >>> # Expand all the pre-requistes for each docker image and test combination >>> diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker >>> new file mode 100644 >>> index 0000000000..55b19eebbf >>> --- /dev/null >>> +++ b/tests/docker/dockerfiles/fedora-avocado.docker >>> @@ -0,0 +1,25 @@ >>> +FROM qemu:fedora >>> + >>> +ENV PACKAGES \ >>> + libvirt-devel \ >>> + nc \ >>> + python-avocado \ >>> + python2-devel python3-devel \ >>> + qemu-kvm \ >>> + tcpdump \ >>> + xz >>> +ENV PIP_PACKAGES \ >>> + avocado-qemu \ >>> + avocado-framework-plugin-runner-remote \ >>> + avocado-framework-plugin-runner-vm \ >>> + avocado-framework-plugin-vt >>> + >>> +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 >> >> I think this is inherited from qemu:fedora, no? > > It is, I have removed it. > >> >>> + >>> +RUN dnf install -y $PACKAGES >>> +RUN pip install $PIP_PACKAGES >>> +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >>> + >>> +RUN rpm -q $PACKAGES | sort > /packages.txt >> >> Can you keep the parent image's list with ">>" or appending to the old $PACKAGES >> in the above "ENV" directive? >> >>> + >>> +ENV FEATURES mingw clang pyyaml asan avocado >> >> Similarly, is it possible to append to the parent list instead of overriding? > > I have changed both of these to append to the original variables. Feel free to add my R-b then: Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> I'll wait your respin to try again, hoping I got my proxy issues solved. >> >>> diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado >>> new file mode 100755 >>> index 0000000000..40474db2ce >>> --- /dev/null >>> +++ b/tests/docker/test-avocado >>> @@ -0,0 +1,28 @@ >>> +#!/bin/bash -e >>> +# >>> +# Avocado tests on Fedora, as these are a real pain on Debian systems >> >> Shouldn't pip packages work just well on Debian too? What are the pain? >> (Cc'ing Cleber who may want to know this). > > There is no debian package at the moment. > > Alistair > >> >> Fam >> >>> +# >>> +# Copyright (c) 2018 Western Digital. >>> +# >>> +# Authors: >>> +# Alistair Francis <alistair.francis@wdc.com> >>> +# >>> +# This work is licensed under the terms of the GNU GPL, version 2 >>> +# or (at your option) any later version. See the COPYING file in >>> +# the top-level directory. >>> +# >>> +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado >>> + >>> +. common.rc >>> + >>> +requires avocado >>> + >>> +cd "$BUILD_DIR" >>> + >>> +DEF_TARGET_LIST="x86_64-softmmu" >>> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ >>> +build_qemu >>> +install_qemu >>> + >>> +export PATH="${PATH}:$(pwd)" >>> +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 >>> -- >>> 2.17.0 >>> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-21 23:32 ` Philippe Mathieu-Daudé @ 2018-05-22 19:50 ` Cleber Rosa 2018-05-22 20:14 ` Lucas Meneghel Rodrigues 0 siblings, 1 reply; 11+ messages in thread From: Cleber Rosa @ 2018-05-22 19:50 UTC (permalink / raw) To: Philippe Mathieu-Daudé, Alistair Francis, Fam Zheng Cc: Alex Bennée, Alistair Francis, qemu-devel@nongnu.org Developers, Amador Pahim, Lucas Meneghel Rodrigues On 05/21/2018 07:32 PM, Philippe Mathieu-Daudé wrote: > On 05/21/2018 07:33 PM, Alistair Francis wrote: >> On Sun, May 20, 2018 at 8:16 PM, Fam Zheng <famz@redhat.com> wrote: >>> On Fri, 05/18 11:34, Alistair Francis wrote: >>>> Avocado is not trivial to setup on non-Fedora systems. To simplfying >>>> future testing add a docker test image that runs Avocado tests. >>>> >>>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com> >>>> --- >>>> v2: >>>> - Add a seperate fedora-avocado Docker image >>>> - Move the avocado vt-bootstrap into the Docker file >>>> >>>> tests/docker/Makefile.include | 1 + >>>> .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ >>>> tests/docker/test-avocado | 28 +++++++++++++++++++ >>>> 3 files changed, 54 insertions(+) >>>> create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker >>>> create mode 100755 tests/docker/test-avocado >>>> >>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >>>> index ef1a3e62eb..0e3d108dde 100644 >>>> --- a/tests/docker/Makefile.include >>>> +++ b/tests/docker/Makefile.include >>>> @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >>>> docker-image-debian-s390x-cross: docker-image-debian9 >>>> docker-image-debian-win32-cross: docker-image-debian8-mxe >>>> docker-image-debian-win64-cross: docker-image-debian8-mxe >>>> +docker-image-fedora-avocado: docker-image-fedora >>>> docker-image-travis: NOUSER=1 >>>> >>>> # Expand all the pre-requistes for each docker image and test combination >>>> diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker >>>> new file mode 100644 >>>> index 0000000000..55b19eebbf >>>> --- /dev/null >>>> +++ b/tests/docker/dockerfiles/fedora-avocado.docker >>>> @@ -0,0 +1,25 @@ >>>> +FROM qemu:fedora >>>> + >>>> +ENV PACKAGES \ >>>> + libvirt-devel \ >>>> + nc \ >>>> + python-avocado \ python-avocado is a "Provides" of python2-avocado, but that may may go away soon (as soon as Fedora 29). I'd recommend using python2-avocado, which is the proper package name available on Fedora. >>>> + python2-devel python3-devel \ >>>> + qemu-kvm \ >>>> + tcpdump \ >>>> + xz >>>> +ENV PIP_PACKAGES \ >>>> + avocado-qemu \ +CC: Amador Pahim This pip package was created by Amador Pahim, but was intended for demo purposes only. Upon finding about this, he (cc'ed here) said that the package is going to be removed to avoid further confusion. BTW, the intention is to have this supporting code for QEMU tests in the QEMU tree itself. Myself and Eduardo are working on a new version of that supporting code. Also, based on what I see on the test-avocado file, this code is not needed at this point. >>>> + avocado-framework-plugin-runner-remote \ >>>> + avocado-framework-plugin-runner-vm \ These two plugins are only needed if you intend to run tests on remote machines. Given "test-avocado" is not making use of those features, there's no need to install them. As a side note, plugins are also available on Fedora as RPM packages, so if I failed to understand the requirements for these plugins here, they could still be installed as the python2-avocado-plugins-runner-remote and python2-avocado-plugins-runner-vm RPM packages. >>>> + avocado-framework-plugin-vt >>>> + +CC: Lucas Rodrigues This is a pip package created by Lucas Rodrigues, and this code is certainly used in the "test-avocado" file. He told me he's using it to run Avocado-VT tests in a venv on Ubuntu, Amazon Linux, etc, and he's willing to maintain and update it. >>>> +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 OK, "python3-avocado" is not yet available on Fedora, because it still tracks the 52.x (LTS) series. So, if Python 3 is the version chosen, it maybe a good idea to remove "python-avocado" from "PACKAGES" completely and use "avocado-framework" instead on "PIP_PACKAGES". >>> >>> I think this is inherited from qemu:fedora, no? >> >> It is, I have removed it. >> >>> >>>> + >>>> +RUN dnf install -y $PACKAGES >>>> +RUN pip install $PIP_PACKAGES >>>> +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >>>> + >>>> +RUN rpm -q $PACKAGES | sort > /packages.txt >>> >>> Can you keep the parent image's list with ">>" or appending to the old $PACKAGES >>> in the above "ENV" directive? >>> >>>> + >>>> +ENV FEATURES mingw clang pyyaml asan avocado >>> >>> Similarly, is it possible to append to the parent list instead of overriding? >> >> I have changed both of these to append to the original variables. > > Feel free to add my R-b then: > Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> > > I'll wait your respin to try again, hoping I got my proxy issues solved. > >>> >>>> diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado >>>> new file mode 100755 >>>> index 0000000000..40474db2ce >>>> --- /dev/null >>>> +++ b/tests/docker/test-avocado >>>> @@ -0,0 +1,28 @@ >>>> +#!/bin/bash -e >>>> +# >>>> +# Avocado tests on Fedora, as these are a real pain on Debian systems >>> >>> Shouldn't pip packages work just well on Debian too? What are the pain? >>> (Cc'ing Cleber who may want to know this). >> >> There is no debian package at the moment. >> That's why Fam said *pip* packages. In my experience with pip packages on Debian: * avocado-framework: installation and general usage works well. Found one issue though when running on containers, though, with an easy workaround[1] * avocado-framework-plugins-vt: currently at version 57.x, and when used outside of venvs, doesn't get the "test-providers" directory right[2]. Same happened with latest master (using pip install -e git+https://github.com/avocado-framework/avocado-vt.git#egg=avocado-plugins-vt) [1] - https://trello.com/c/yEH8ixps/1328-avocado-crashes-on-systems-without-less [2] - https://trello.com/c/0BZZE4CR/1329-avocado-vt-bootstrap-from-pip-package-fails-to-find-test-providers-dir - Cleber. >> Alistair >> >>> >>> Fam >>> >>>> +# >>>> +# Copyright (c) 2018 Western Digital. >>>> +# >>>> +# Authors: >>>> +# Alistair Francis <alistair.francis@wdc.com> >>>> +# >>>> +# This work is licensed under the terms of the GNU GPL, version 2 >>>> +# or (at your option) any later version. See the COPYING file in >>>> +# the top-level directory. >>>> +# >>>> +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado >>>> + >>>> +. common.rc >>>> + >>>> +requires avocado >>>> + >>>> +cd "$BUILD_DIR" >>>> + >>>> +DEF_TARGET_LIST="x86_64-softmmu" >>>> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ >>>> +build_qemu >>>> +install_qemu >>>> + >>>> +export PATH="${PATH}:$(pwd)" >>>> +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 >>>> -- >>>> 2.17.0 >>>> > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-22 19:50 ` Cleber Rosa @ 2018-05-22 20:14 ` Lucas Meneghel Rodrigues 2018-05-23 9:35 ` Lucas Meneghel Rodrigues 0 siblings, 1 reply; 11+ messages in thread From: Lucas Meneghel Rodrigues @ 2018-05-22 20:14 UTC (permalink / raw) To: Cleber Rosa Cc: Philippe Mathieu-Daudé, Alistair Francis, Fam Zheng, Alex Bennée, Alistair Francis, qemu-devel@nongnu.org Developers, Amador Pahim On Tue, May 22, 2018 at 9:50 PM, Cleber Rosa <crosa@redhat.com> wrote: > > > On 05/21/2018 07:32 PM, Philippe Mathieu-Daudé wrote: >> On 05/21/2018 07:33 PM, Alistair Francis wrote: >>> On Sun, May 20, 2018 at 8:16 PM, Fam Zheng <famz@redhat.com> wrote: >>>> On Fri, 05/18 11:34, Alistair Francis wrote: >>>>> Avocado is not trivial to setup on non-Fedora systems. To simplfying >>>>> future testing add a docker test image that runs Avocado tests. >>>>> >>>>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com> >>>>> --- >>>>> v2: >>>>> - Add a seperate fedora-avocado Docker image >>>>> - Move the avocado vt-bootstrap into the Docker file >>>>> >>>>> tests/docker/Makefile.include | 1 + >>>>> .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ >>>>> tests/docker/test-avocado | 28 +++++++++++++++++++ >>>>> 3 files changed, 54 insertions(+) >>>>> create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker >>>>> create mode 100755 tests/docker/test-avocado >>>>> >>>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >>>>> index ef1a3e62eb..0e3d108dde 100644 >>>>> --- a/tests/docker/Makefile.include >>>>> +++ b/tests/docker/Makefile.include >>>>> @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >>>>> docker-image-debian-s390x-cross: docker-image-debian9 >>>>> docker-image-debian-win32-cross: docker-image-debian8-mxe >>>>> docker-image-debian-win64-cross: docker-image-debian8-mxe >>>>> +docker-image-fedora-avocado: docker-image-fedora >>>>> docker-image-travis: NOUSER=1 >>>>> >>>>> # Expand all the pre-requistes for each docker image and test combination >>>>> diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker >>>>> new file mode 100644 >>>>> index 0000000000..55b19eebbf >>>>> --- /dev/null >>>>> +++ b/tests/docker/dockerfiles/fedora-avocado.docker >>>>> @@ -0,0 +1,25 @@ >>>>> +FROM qemu:fedora >>>>> + >>>>> +ENV PACKAGES \ >>>>> + libvirt-devel \ >>>>> + nc \ >>>>> + python-avocado \ > > python-avocado is a "Provides" of python2-avocado, but that may may go > away soon (as soon as Fedora 29). I'd recommend using python2-avocado, > which is the proper package name available on Fedora. > >>>>> + python2-devel python3-devel \ >>>>> + qemu-kvm \ >>>>> + tcpdump \ >>>>> + xz >>>>> +ENV PIP_PACKAGES \ >>>>> + avocado-qemu \ > > +CC: Amador Pahim > > This pip package was created by Amador Pahim, but was intended for demo > purposes only. Upon finding about this, he (cc'ed here) said that the > package is going to be removed to avoid further confusion. > > BTW, the intention is to have this supporting code for QEMU tests in the > QEMU tree itself. Myself and Eduardo are working on a new version of > that supporting code. Also, based on what I see on the test-avocado > file, this code is not needed at this point. > >>>>> + avocado-framework-plugin-runner-remote \ >>>>> + avocado-framework-plugin-runner-vm \ > > These two plugins are only needed if you intend to run tests on remote > machines. Given "test-avocado" is not making use of those features, > there's no need to install them. > > As a side note, plugins are also available on Fedora as RPM packages, so > if I failed to understand the requirements for these plugins here, they > could still be installed as the python2-avocado-plugins-runner-remote > and python2-avocado-plugins-runner-vm RPM packages. > >>>>> + avocado-framework-plugin-vt >>>>> + > > +CC: Lucas Rodrigues > > This is a pip package created by Lucas Rodrigues, and this code is > certainly used in the "test-avocado" file. He told me he's using it to > run Avocado-VT tests in a venv on Ubuntu, Amazon Linux, etc, and he's > willing to maintain and update it. I'd suggest to look into virtual environments to run avocado-vt. virtual environments are more lightweight than using a docker container to run things and achieve similar functionality. Since I mostly never install python libraries and projects system wide these days, I didn't test this install mode in a docker container. Here's what I'd do on my Ubuntu 16.04 development laptop: $ virtualenv avocado_vt_demo Running virtualenv with interpreter /usr/bin/python2 New python executable in /home/lmr/avocado_vt_demo/bin/python2 Also creating executable in /home/lmr/avocado_vt_demo/bin/python Installing setuptools, pkg_resources, pip, wheel...done. $ source avocado_vt_demo/bin/activate (avocado_vt_demo) $ pip install avocado-framework-plugin-vt ... output trimmed ... Successfully built avocado-framework-plugin-vt autotest aexpect simplejson Installing collected packages: pbr, six, stevedore, avocado-framework, autotest, aexpect, simplejson, netaddr, netifaces, avocado-framework-plugin-vt Successfully installed aexpect-1.4.0 autotest-0.16.4 avocado-framework-61.0 avocado-framework-plugin-vt-57.4 netaddr-0.7.19 netifaces-0.10.7 pbr-4.0.3 simplejson-3.15.0 six-1.11.0 stevedore-1.28.0 (avocado_vt_demo) $ avocado vt-bootstrap --yes-to-all --vt-type qemu Failed to load plugin from module "avocado_vt.plugins.vt_list": EnvironmentError("Bootstrap missing. Execute 'avocado vt-bootstrap' or disable this plugin to get rid of this message",) Failed to load plugin from module "avocado_vt.plugins.vt": EnvironmentError("Bootstrap missing. Execute 'avocado vt-bootstrap' or disable this plugin to get rid of this message",) Running bootstrap for qemu 1 - Checking the mandatory programs and headers /usr/bin/xz OK /usr/sbin/tcpdump OK /bin/nc OK /bin/ip OK /usr/bin/arping OK /usr/bin/gcc OK /usr/include/python2.7/Python.h OK /usr/include/linux/types.h OK /usr/include/linux/unistd.h OK /usr/include/linux/socket.h OK ... output trimmed ... 9 - Checking for modules kvm, kvm-intel Module kvm loaded Module kvm-intel loaded VT-BOOTSTRAP FINISHED You may take a look at the following online docs for more info: - http://avocado-vt.readthedocs.org/ - http://avocado-framework.readthedocs.org/ (avocado_vt_demo) $ avocado run boot JOB ID : 2ffba3965cd327e768bd8dadf4f82d6416a4f293 JOB LOG : /home/lmr/avocado/job-results/job-2018-05-22T22.09-2ffba39/job.log (1/1) io-github-autotest-qemu.boot: PASS (31.31 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 33.25 s I can maintain the pip package mostly in sync with the other avocado releases. Just keep in mind that my bandwidth for maintenance tasks outside of my job @Amazon is limited, so you will have to nudge me if you need something urgently. Cheers, Lucas >>>>> +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 > > OK, "python3-avocado" is not yet available on Fedora, because it still > tracks the 52.x (LTS) series. So, if Python 3 is the version chosen, it > maybe a good idea to remove "python-avocado" from "PACKAGES" completely > and use "avocado-framework" instead on "PIP_PACKAGES". > >>>> >>>> I think this is inherited from qemu:fedora, no? >>> >>> It is, I have removed it. >>> >>>> >>>>> + >>>>> +RUN dnf install -y $PACKAGES >>>>> +RUN pip install $PIP_PACKAGES >>>>> +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >>>>> + >>>>> +RUN rpm -q $PACKAGES | sort > /packages.txt >>>> >>>> Can you keep the parent image's list with ">>" or appending to the old $PACKAGES >>>> in the above "ENV" directive? >>>> >>>>> + >>>>> +ENV FEATURES mingw clang pyyaml asan avocado >>>> >>>> Similarly, is it possible to append to the parent list instead of overriding? >>> >>> I have changed both of these to append to the original variables. >> >> Feel free to add my R-b then: >> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> >> >> I'll wait your respin to try again, hoping I got my proxy issues solved. >> >>>> >>>>> diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado >>>>> new file mode 100755 >>>>> index 0000000000..40474db2ce >>>>> --- /dev/null >>>>> +++ b/tests/docker/test-avocado >>>>> @@ -0,0 +1,28 @@ >>>>> +#!/bin/bash -e >>>>> +# >>>>> +# Avocado tests on Fedora, as these are a real pain on Debian systems >>>> >>>> Shouldn't pip packages work just well on Debian too? What are the pain? >>>> (Cc'ing Cleber who may want to know this). >>> >>> There is no debian package at the moment. >>> > > That's why Fam said *pip* packages. In my experience with pip packages > on Debian: > > * avocado-framework: installation and general usage works well. Found > one issue though when running on containers, though, with an easy > workaround[1] > > * avocado-framework-plugins-vt: currently at version 57.x, and when > used outside of venvs, doesn't get the "test-providers" directory > right[2]. Same happened with latest master (using pip install -e > git+https://github.com/avocado-framework/avocado-vt.git#egg=avocado-plugins-vt) > > [1] - > https://trello.com/c/yEH8ixps/1328-avocado-crashes-on-systems-without-less > > [2] - > https://trello.com/c/0BZZE4CR/1329-avocado-vt-bootstrap-from-pip-package-fails-to-find-test-providers-dir > > - Cleber. > >>> Alistair >>> >>>> >>>> Fam >>>> >>>>> +# >>>>> +# Copyright (c) 2018 Western Digital. >>>>> +# >>>>> +# Authors: >>>>> +# Alistair Francis <alistair.francis@wdc.com> >>>>> +# >>>>> +# This work is licensed under the terms of the GNU GPL, version 2 >>>>> +# or (at your option) any later version. See the COPYING file in >>>>> +# the top-level directory. >>>>> +# >>>>> +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado >>>>> + >>>>> +. common.rc >>>>> + >>>>> +requires avocado >>>>> + >>>>> +cd "$BUILD_DIR" >>>>> + >>>>> +DEF_TARGET_LIST="x86_64-softmmu" >>>>> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ >>>>> +build_qemu >>>>> +install_qemu >>>>> + >>>>> +export PATH="${PATH}:$(pwd)" >>>>> +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 >>>>> -- >>>>> 2.17.0 >>>>> >> -- Lucas ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-22 20:14 ` Lucas Meneghel Rodrigues @ 2018-05-23 9:35 ` Lucas Meneghel Rodrigues 2018-05-23 10:14 ` Cleber Rosa 0 siblings, 1 reply; 11+ messages in thread From: Lucas Meneghel Rodrigues @ 2018-05-23 9:35 UTC (permalink / raw) To: Cleber Rosa Cc: Philippe Mathieu-Daudé, Alistair Francis, Fam Zheng, Alex Bennée, Alistair Francis, qemu-devel@nongnu.org Developers, Amador Pahim Me and Cleber were talking about running avocado in containers, and we verified there are some things we need to fix in avocado to make it work well under container environments. One of the main offenders is our service manager library, that was created to deal with sysvinit and systemd, but Docker containers usually don't have either. The test runner does use the service library to infer a few things about the system (even on simple tests), so fixing is needed. So your docker proposal as of is now depends on us fixing those bugs for it to be functional. Please bear with us meanwhile. On Tue, May 22, 2018 at 10:14 PM, Lucas Meneghel Rodrigues <lookkas@gmail.com> wrote: > On Tue, May 22, 2018 at 9:50 PM, Cleber Rosa <crosa@redhat.com> wrote: >> >> >> On 05/21/2018 07:32 PM, Philippe Mathieu-Daudé wrote: >>> On 05/21/2018 07:33 PM, Alistair Francis wrote: >>>> On Sun, May 20, 2018 at 8:16 PM, Fam Zheng <famz@redhat.com> wrote: >>>>> On Fri, 05/18 11:34, Alistair Francis wrote: >>>>>> Avocado is not trivial to setup on non-Fedora systems. To simplfying >>>>>> future testing add a docker test image that runs Avocado tests. >>>>>> >>>>>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com> >>>>>> --- >>>>>> v2: >>>>>> - Add a seperate fedora-avocado Docker image >>>>>> - Move the avocado vt-bootstrap into the Docker file >>>>>> >>>>>> tests/docker/Makefile.include | 1 + >>>>>> .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ >>>>>> tests/docker/test-avocado | 28 +++++++++++++++++++ >>>>>> 3 files changed, 54 insertions(+) >>>>>> create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker >>>>>> create mode 100755 tests/docker/test-avocado >>>>>> >>>>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >>>>>> index ef1a3e62eb..0e3d108dde 100644 >>>>>> --- a/tests/docker/Makefile.include >>>>>> +++ b/tests/docker/Makefile.include >>>>>> @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >>>>>> docker-image-debian-s390x-cross: docker-image-debian9 >>>>>> docker-image-debian-win32-cross: docker-image-debian8-mxe >>>>>> docker-image-debian-win64-cross: docker-image-debian8-mxe >>>>>> +docker-image-fedora-avocado: docker-image-fedora >>>>>> docker-image-travis: NOUSER=1 >>>>>> >>>>>> # Expand all the pre-requistes for each docker image and test combination >>>>>> diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker >>>>>> new file mode 100644 >>>>>> index 0000000000..55b19eebbf >>>>>> --- /dev/null >>>>>> +++ b/tests/docker/dockerfiles/fedora-avocado.docker >>>>>> @@ -0,0 +1,25 @@ >>>>>> +FROM qemu:fedora >>>>>> + >>>>>> +ENV PACKAGES \ >>>>>> + libvirt-devel \ >>>>>> + nc \ >>>>>> + python-avocado \ >> >> python-avocado is a "Provides" of python2-avocado, but that may may go >> away soon (as soon as Fedora 29). I'd recommend using python2-avocado, >> which is the proper package name available on Fedora. >> >>>>>> + python2-devel python3-devel \ >>>>>> + qemu-kvm \ >>>>>> + tcpdump \ >>>>>> + xz >>>>>> +ENV PIP_PACKAGES \ >>>>>> + avocado-qemu \ >> >> +CC: Amador Pahim >> >> This pip package was created by Amador Pahim, but was intended for demo >> purposes only. Upon finding about this, he (cc'ed here) said that the >> package is going to be removed to avoid further confusion. >> >> BTW, the intention is to have this supporting code for QEMU tests in the >> QEMU tree itself. Myself and Eduardo are working on a new version of >> that supporting code. Also, based on what I see on the test-avocado >> file, this code is not needed at this point. >> >>>>>> + avocado-framework-plugin-runner-remote \ >>>>>> + avocado-framework-plugin-runner-vm \ >> >> These two plugins are only needed if you intend to run tests on remote >> machines. Given "test-avocado" is not making use of those features, >> there's no need to install them. >> >> As a side note, plugins are also available on Fedora as RPM packages, so >> if I failed to understand the requirements for these plugins here, they >> could still be installed as the python2-avocado-plugins-runner-remote >> and python2-avocado-plugins-runner-vm RPM packages. >> >>>>>> + avocado-framework-plugin-vt >>>>>> + >> >> +CC: Lucas Rodrigues >> >> This is a pip package created by Lucas Rodrigues, and this code is >> certainly used in the "test-avocado" file. He told me he's using it to >> run Avocado-VT tests in a venv on Ubuntu, Amazon Linux, etc, and he's >> willing to maintain and update it. > > I'd suggest to look into virtual environments to run avocado-vt. > virtual environments are more lightweight than using a docker > container to run things and achieve similar functionality. Since I > mostly never install python libraries and projects system wide these > days, I didn't test this install mode in a docker container. Here's > what I'd do on my Ubuntu 16.04 development laptop: > > $ virtualenv avocado_vt_demo > Running virtualenv with interpreter /usr/bin/python2 > New python executable in /home/lmr/avocado_vt_demo/bin/python2 > Also creating executable in /home/lmr/avocado_vt_demo/bin/python > Installing setuptools, pkg_resources, pip, wheel...done. > > $ source avocado_vt_demo/bin/activate > > (avocado_vt_demo) $ pip install avocado-framework-plugin-vt > ... output trimmed ... > Successfully built avocado-framework-plugin-vt autotest aexpect simplejson > Installing collected packages: pbr, six, stevedore, avocado-framework, > autotest, aexpect, simplejson, netaddr, netifaces, > avocado-framework-plugin-vt > Successfully installed aexpect-1.4.0 autotest-0.16.4 > avocado-framework-61.0 avocado-framework-plugin-vt-57.4 netaddr-0.7.19 > netifaces-0.10.7 pbr-4.0.3 simplejson-3.15.0 six-1.11.0 > stevedore-1.28.0 > > (avocado_vt_demo) $ avocado vt-bootstrap --yes-to-all --vt-type qemu > Failed to load plugin from module "avocado_vt.plugins.vt_list": > EnvironmentError("Bootstrap missing. Execute 'avocado vt-bootstrap' or > disable this plugin to get rid of this message",) > Failed to load plugin from module "avocado_vt.plugins.vt": > EnvironmentError("Bootstrap missing. Execute 'avocado vt-bootstrap' or > disable this plugin to get rid of this message",) > Running bootstrap for qemu > > 1 - Checking the mandatory programs and headers > /usr/bin/xz OK > /usr/sbin/tcpdump OK > /bin/nc OK > /bin/ip OK > /usr/bin/arping OK > /usr/bin/gcc OK > /usr/include/python2.7/Python.h OK > /usr/include/linux/types.h OK > /usr/include/linux/unistd.h OK > /usr/include/linux/socket.h OK > > ... output trimmed ... > > 9 - Checking for modules kvm, kvm-intel > Module kvm loaded > Module kvm-intel loaded > > VT-BOOTSTRAP FINISHED > You may take a look at the following online docs for more info: > - http://avocado-vt.readthedocs.org/ > - http://avocado-framework.readthedocs.org/ > > (avocado_vt_demo) $ avocado run boot > JOB ID : 2ffba3965cd327e768bd8dadf4f82d6416a4f293 > JOB LOG : /home/lmr/avocado/job-results/job-2018-05-22T22.09-2ffba39/job.log > (1/1) io-github-autotest-qemu.boot: PASS (31.31 s) > RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 > | CANCEL 0 > JOB TIME : 33.25 s > > I can maintain the pip package mostly in sync with the other avocado > releases. Just keep in mind that my bandwidth for maintenance tasks > outside of my job @Amazon is limited, so you will have to nudge me if > you need something urgently. > > Cheers, > > Lucas > >>>>>> +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 >> >> OK, "python3-avocado" is not yet available on Fedora, because it still >> tracks the 52.x (LTS) series. So, if Python 3 is the version chosen, it >> maybe a good idea to remove "python-avocado" from "PACKAGES" completely >> and use "avocado-framework" instead on "PIP_PACKAGES". >> >>>>> >>>>> I think this is inherited from qemu:fedora, no? >>>> >>>> It is, I have removed it. >>>> >>>>> >>>>>> + >>>>>> +RUN dnf install -y $PACKAGES >>>>>> +RUN pip install $PIP_PACKAGES >>>>>> +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >>>>>> + >>>>>> +RUN rpm -q $PACKAGES | sort > /packages.txt >>>>> >>>>> Can you keep the parent image's list with ">>" or appending to the old $PACKAGES >>>>> in the above "ENV" directive? >>>>> >>>>>> + >>>>>> +ENV FEATURES mingw clang pyyaml asan avocado >>>>> >>>>> Similarly, is it possible to append to the parent list instead of overriding? >>>> >>>> I have changed both of these to append to the original variables. >>> >>> Feel free to add my R-b then: >>> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> >>> >>> I'll wait your respin to try again, hoping I got my proxy issues solved. >>> >>>>> >>>>>> diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado >>>>>> new file mode 100755 >>>>>> index 0000000000..40474db2ce >>>>>> --- /dev/null >>>>>> +++ b/tests/docker/test-avocado >>>>>> @@ -0,0 +1,28 @@ >>>>>> +#!/bin/bash -e >>>>>> +# >>>>>> +# Avocado tests on Fedora, as these are a real pain on Debian systems >>>>> >>>>> Shouldn't pip packages work just well on Debian too? What are the pain? >>>>> (Cc'ing Cleber who may want to know this). >>>> >>>> There is no debian package at the moment. >>>> >> >> That's why Fam said *pip* packages. In my experience with pip packages >> on Debian: >> >> * avocado-framework: installation and general usage works well. Found >> one issue though when running on containers, though, with an easy >> workaround[1] >> >> * avocado-framework-plugins-vt: currently at version 57.x, and when >> used outside of venvs, doesn't get the "test-providers" directory >> right[2]. Same happened with latest master (using pip install -e >> git+https://github.com/avocado-framework/avocado-vt.git#egg=avocado-plugins-vt) >> >> [1] - >> https://trello.com/c/yEH8ixps/1328-avocado-crashes-on-systems-without-less >> >> [2] - >> https://trello.com/c/0BZZE4CR/1329-avocado-vt-bootstrap-from-pip-package-fails-to-find-test-providers-dir >> >> - Cleber. >> >>>> Alistair >>>> >>>>> >>>>> Fam >>>>> >>>>>> +# >>>>>> +# Copyright (c) 2018 Western Digital. >>>>>> +# >>>>>> +# Authors: >>>>>> +# Alistair Francis <alistair.francis@wdc.com> >>>>>> +# >>>>>> +# This work is licensed under the terms of the GNU GPL, version 2 >>>>>> +# or (at your option) any later version. See the COPYING file in >>>>>> +# the top-level directory. >>>>>> +# >>>>>> +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado >>>>>> + >>>>>> +. common.rc >>>>>> + >>>>>> +requires avocado >>>>>> + >>>>>> +cd "$BUILD_DIR" >>>>>> + >>>>>> +DEF_TARGET_LIST="x86_64-softmmu" >>>>>> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ >>>>>> +build_qemu >>>>>> +install_qemu >>>>>> + >>>>>> +export PATH="${PATH}:$(pwd)" >>>>>> +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 >>>>>> -- >>>>>> 2.17.0 >>>>>> >>> > > > > -- > Lucas -- Lucas ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test 2018-05-23 9:35 ` Lucas Meneghel Rodrigues @ 2018-05-23 10:14 ` Cleber Rosa 0 siblings, 0 replies; 11+ messages in thread From: Cleber Rosa @ 2018-05-23 10:14 UTC (permalink / raw) To: Lucas Meneghel Rodrigues Cc: Philippe Mathieu-Daudé, Alistair Francis, Fam Zheng, Alex Bennée, Alistair Francis, qemu-devel@nongnu.org Developers, Amador Pahim On 05/23/2018 05:35 AM, Lucas Meneghel Rodrigues wrote: > Me and Cleber were talking about running avocado in containers, and we > verified there are some things we need to fix in avocado to make it > work well under container environments. One of the main offenders is > our service manager library, that was created to deal with sysvinit > and systemd, but Docker containers usually don't have either. The test > runner does use the service library to infer a few things about the > system (even on simple tests), so fixing is needed. > What (I believe) Lucas means is: * if you use Avocado-VT tests, like on this proposal, inside containers, then there's code checking for the libvirtd "service", which will get the "service manager" (init or systemd) wrong * if you run Avocado tests and use the "avocado.utils.service" library on containers, you can hit a similar issue We're tracking these issues here: https://trello.com/c/GX1cdFSp/1330-avocadoutilsservice-adapt-to-service-management-on-containers Now for other Avocado tests, or for the Avocado test runner itself, running on containers, there should be no issues: $ docker pull clebergnu/fedora-avocado $ docker run -t clebergnu/fedora-avocado avocado run /bin/true - Cleber. > So your docker proposal as of is now depends on us fixing those bugs > for it to be functional. Please bear with us meanwhile. > > On Tue, May 22, 2018 at 10:14 PM, Lucas Meneghel Rodrigues > <lookkas@gmail.com> wrote: >> On Tue, May 22, 2018 at 9:50 PM, Cleber Rosa <crosa@redhat.com> wrote: >>> >>> >>> On 05/21/2018 07:32 PM, Philippe Mathieu-Daudé wrote: >>>> On 05/21/2018 07:33 PM, Alistair Francis wrote: >>>>> On Sun, May 20, 2018 at 8:16 PM, Fam Zheng <famz@redhat.com> wrote: >>>>>> On Fri, 05/18 11:34, Alistair Francis wrote: >>>>>>> Avocado is not trivial to setup on non-Fedora systems. To simplfying >>>>>>> future testing add a docker test image that runs Avocado tests. >>>>>>> >>>>>>> Signed-off-by: Alistair Francis <alistair.francis@wdc.com> >>>>>>> --- >>>>>>> v2: >>>>>>> - Add a seperate fedora-avocado Docker image >>>>>>> - Move the avocado vt-bootstrap into the Docker file >>>>>>> >>>>>>> tests/docker/Makefile.include | 1 + >>>>>>> .../docker/dockerfiles/fedora-avocado.docker | 25 +++++++++++++++++ >>>>>>> tests/docker/test-avocado | 28 +++++++++++++++++++ >>>>>>> 3 files changed, 54 insertions(+) >>>>>>> create mode 100644 tests/docker/dockerfiles/fedora-avocado.docker >>>>>>> create mode 100755 tests/docker/test-avocado >>>>>>> >>>>>>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include >>>>>>> index ef1a3e62eb..0e3d108dde 100644 >>>>>>> --- a/tests/docker/Makefile.include >>>>>>> +++ b/tests/docker/Makefile.include >>>>>>> @@ -60,6 +60,7 @@ docker-image-debian-ppc64el-cross: docker-image-debian9 >>>>>>> docker-image-debian-s390x-cross: docker-image-debian9 >>>>>>> docker-image-debian-win32-cross: docker-image-debian8-mxe >>>>>>> docker-image-debian-win64-cross: docker-image-debian8-mxe >>>>>>> +docker-image-fedora-avocado: docker-image-fedora >>>>>>> docker-image-travis: NOUSER=1 >>>>>>> >>>>>>> # Expand all the pre-requistes for each docker image and test combination >>>>>>> diff --git a/tests/docker/dockerfiles/fedora-avocado.docker b/tests/docker/dockerfiles/fedora-avocado.docker >>>>>>> new file mode 100644 >>>>>>> index 0000000000..55b19eebbf >>>>>>> --- /dev/null >>>>>>> +++ b/tests/docker/dockerfiles/fedora-avocado.docker >>>>>>> @@ -0,0 +1,25 @@ >>>>>>> +FROM qemu:fedora >>>>>>> + >>>>>>> +ENV PACKAGES \ >>>>>>> + libvirt-devel \ >>>>>>> + nc \ >>>>>>> + python-avocado \ >>> >>> python-avocado is a "Provides" of python2-avocado, but that may may go >>> away soon (as soon as Fedora 29). I'd recommend using python2-avocado, >>> which is the proper package name available on Fedora. >>> >>>>>>> + python2-devel python3-devel \ >>>>>>> + qemu-kvm \ >>>>>>> + tcpdump \ >>>>>>> + xz >>>>>>> +ENV PIP_PACKAGES \ >>>>>>> + avocado-qemu \ >>> >>> +CC: Amador Pahim >>> >>> This pip package was created by Amador Pahim, but was intended for demo >>> purposes only. Upon finding about this, he (cc'ed here) said that the >>> package is going to be removed to avoid further confusion. >>> >>> BTW, the intention is to have this supporting code for QEMU tests in the >>> QEMU tree itself. Myself and Eduardo are working on a new version of >>> that supporting code. Also, based on what I see on the test-avocado >>> file, this code is not needed at this point. >>> >>>>>>> + avocado-framework-plugin-runner-remote \ >>>>>>> + avocado-framework-plugin-runner-vm \ >>> >>> These two plugins are only needed if you intend to run tests on remote >>> machines. Given "test-avocado" is not making use of those features, >>> there's no need to install them. >>> >>> As a side note, plugins are also available on Fedora as RPM packages, so >>> if I failed to understand the requirements for these plugins here, they >>> could still be installed as the python2-avocado-plugins-runner-remote >>> and python2-avocado-plugins-runner-vm RPM packages. >>> >>>>>>> + avocado-framework-plugin-vt >>>>>>> + >>> >>> +CC: Lucas Rodrigues >>> >>> This is a pip package created by Lucas Rodrigues, and this code is >>> certainly used in the "test-avocado" file. He told me he's using it to >>> run Avocado-VT tests in a venv on Ubuntu, Amazon Linux, etc, and he's >>> willing to maintain and update it. >> >> I'd suggest to look into virtual environments to run avocado-vt. >> virtual environments are more lightweight than using a docker >> container to run things and achieve similar functionality. Since I >> mostly never install python libraries and projects system wide these >> days, I didn't test this install mode in a docker container. Here's >> what I'd do on my Ubuntu 16.04 development laptop: >> >> $ virtualenv avocado_vt_demo >> Running virtualenv with interpreter /usr/bin/python2 >> New python executable in /home/lmr/avocado_vt_demo/bin/python2 >> Also creating executable in /home/lmr/avocado_vt_demo/bin/python >> Installing setuptools, pkg_resources, pip, wheel...done. >> >> $ source avocado_vt_demo/bin/activate >> >> (avocado_vt_demo) $ pip install avocado-framework-plugin-vt >> ... output trimmed ... >> Successfully built avocado-framework-plugin-vt autotest aexpect simplejson >> Installing collected packages: pbr, six, stevedore, avocado-framework, >> autotest, aexpect, simplejson, netaddr, netifaces, >> avocado-framework-plugin-vt >> Successfully installed aexpect-1.4.0 autotest-0.16.4 >> avocado-framework-61.0 avocado-framework-plugin-vt-57.4 netaddr-0.7.19 >> netifaces-0.10.7 pbr-4.0.3 simplejson-3.15.0 six-1.11.0 >> stevedore-1.28.0 >> >> (avocado_vt_demo) $ avocado vt-bootstrap --yes-to-all --vt-type qemu >> Failed to load plugin from module "avocado_vt.plugins.vt_list": >> EnvironmentError("Bootstrap missing. Execute 'avocado vt-bootstrap' or >> disable this plugin to get rid of this message",) >> Failed to load plugin from module "avocado_vt.plugins.vt": >> EnvironmentError("Bootstrap missing. Execute 'avocado vt-bootstrap' or >> disable this plugin to get rid of this message",) >> Running bootstrap for qemu >> >> 1 - Checking the mandatory programs and headers >> /usr/bin/xz OK >> /usr/sbin/tcpdump OK >> /bin/nc OK >> /bin/ip OK >> /usr/bin/arping OK >> /usr/bin/gcc OK >> /usr/include/python2.7/Python.h OK >> /usr/include/linux/types.h OK >> /usr/include/linux/unistd.h OK >> /usr/include/linux/socket.h OK >> >> ... output trimmed ... >> >> 9 - Checking for modules kvm, kvm-intel >> Module kvm loaded >> Module kvm-intel loaded >> >> VT-BOOTSTRAP FINISHED >> You may take a look at the following online docs for more info: >> - http://avocado-vt.readthedocs.org/ >> - http://avocado-framework.readthedocs.org/ >> >> (avocado_vt_demo) $ avocado run boot >> JOB ID : 2ffba3965cd327e768bd8dadf4f82d6416a4f293 >> JOB LOG : /home/lmr/avocado/job-results/job-2018-05-22T22.09-2ffba39/job.log >> (1/1) io-github-autotest-qemu.boot: PASS (31.31 s) >> RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 >> | CANCEL 0 >> JOB TIME : 33.25 s >> >> I can maintain the pip package mostly in sync with the other avocado >> releases. Just keep in mind that my bandwidth for maintenance tasks >> outside of my job @Amazon is limited, so you will have to nudge me if >> you need something urgently. >> >> Cheers, >> >> Lucas >> >>>>>>> +ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3 >>> >>> OK, "python3-avocado" is not yet available on Fedora, because it still >>> tracks the 52.x (LTS) series. So, if Python 3 is the version chosen, it >>> maybe a good idea to remove "python-avocado" from "PACKAGES" completely >>> and use "avocado-framework" instead on "PIP_PACKAGES". >>> >>>>>> >>>>>> I think this is inherited from qemu:fedora, no? >>>>> >>>>> It is, I have removed it. >>>>> >>>>>> >>>>>>> + >>>>>>> +RUN dnf install -y $PACKAGES >>>>>>> +RUN pip install $PIP_PACKAGES >>>>>>> +RUN avocado vt-bootstrap --yes-to-all --vt-type qemu >>>>>>> + >>>>>>> +RUN rpm -q $PACKAGES | sort > /packages.txt >>>>>> >>>>>> Can you keep the parent image's list with ">>" or appending to the old $PACKAGES >>>>>> in the above "ENV" directive? >>>>>> >>>>>>> + >>>>>>> +ENV FEATURES mingw clang pyyaml asan avocado >>>>>> >>>>>> Similarly, is it possible to append to the parent list instead of overriding? >>>>> >>>>> I have changed both of these to append to the original variables. >>>> >>>> Feel free to add my R-b then: >>>> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> >>>> >>>> I'll wait your respin to try again, hoping I got my proxy issues solved. >>>> >>>>>> >>>>>>> diff --git a/tests/docker/test-avocado b/tests/docker/test-avocado >>>>>>> new file mode 100755 >>>>>>> index 0000000000..40474db2ce >>>>>>> --- /dev/null >>>>>>> +++ b/tests/docker/test-avocado >>>>>>> @@ -0,0 +1,28 @@ >>>>>>> +#!/bin/bash -e >>>>>>> +# >>>>>>> +# Avocado tests on Fedora, as these are a real pain on Debian systems >>>>>> >>>>>> Shouldn't pip packages work just well on Debian too? What are the pain? >>>>>> (Cc'ing Cleber who may want to know this). >>>>> >>>>> There is no debian package at the moment. >>>>> >>> >>> That's why Fam said *pip* packages. In my experience with pip packages >>> on Debian: >>> >>> * avocado-framework: installation and general usage works well. Found >>> one issue though when running on containers, though, with an easy >>> workaround[1] >>> >>> * avocado-framework-plugins-vt: currently at version 57.x, and when >>> used outside of venvs, doesn't get the "test-providers" directory >>> right[2]. Same happened with latest master (using pip install -e >>> git+https://github.com/avocado-framework/avocado-vt.git#egg=avocado-plugins-vt) >>> >>> [1] - >>> https://trello.com/c/yEH8ixps/1328-avocado-crashes-on-systems-without-less >>> >>> [2] - >>> https://trello.com/c/0BZZE4CR/1329-avocado-vt-bootstrap-from-pip-package-fails-to-find-test-providers-dir >>> >>> - Cleber. >>> >>>>> Alistair >>>>> >>>>>> >>>>>> Fam >>>>>> >>>>>>> +# >>>>>>> +# Copyright (c) 2018 Western Digital. >>>>>>> +# >>>>>>> +# Authors: >>>>>>> +# Alistair Francis <alistair.francis@wdc.com> >>>>>>> +# >>>>>>> +# This work is licensed under the terms of the GNU GPL, version 2 >>>>>>> +# or (at your option) any later version. See the COPYING file in >>>>>>> +# the top-level directory. >>>>>>> +# >>>>>>> +# Run this test: NOUSER=1 make docker-test-avocado@fedora-avocado >>>>>>> + >>>>>>> +. common.rc >>>>>>> + >>>>>>> +requires avocado >>>>>>> + >>>>>>> +cd "$BUILD_DIR" >>>>>>> + >>>>>>> +DEF_TARGET_LIST="x86_64-softmmu" >>>>>>> +TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \ >>>>>>> +build_qemu >>>>>>> +install_qemu >>>>>>> + >>>>>>> +export PATH="${PATH}:$(pwd)" >>>>>>> +avocado run boot --vt-qemu-bin ./x86_64-softmmu/qemu-system-x86_64 >>>>>>> -- >>>>>>> 2.17.0 >>>>>>> >>>> >> >> >> >> -- >> Lucas > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2018-05-23 10:14 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-05-18 18:34 [Qemu-devel] [PATCH v2 1/1] tests/docker: Add a Avocado Docker test Alistair Francis 2018-05-21 3:16 ` Fam Zheng 2018-05-21 17:26 ` Philippe Mathieu-Daudé 2018-05-21 22:37 ` Alistair Francis 2018-05-21 23:29 ` Philippe Mathieu-Daudé 2018-05-21 22:33 ` Alistair Francis 2018-05-21 23:32 ` Philippe Mathieu-Daudé 2018-05-22 19:50 ` Cleber Rosa 2018-05-22 20:14 ` Lucas Meneghel Rodrigues 2018-05-23 9:35 ` Lucas Meneghel Rodrigues 2018-05-23 10:14 ` Cleber Rosa
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.