[PATCH-for-5.0 0/8] Acceptance tests queue

[PATCH-for-5.0 0/8] Acceptance tests queue

[Philippe Mathieu-Daudé]

I'm planning to send the first 5 patches as
This series contains acceptance test fixes, and Travis-CI
improvements. I plan to send a pull request with the first
6 patches for v5.0-rc2. The last 3 patches helped debugging
painfully the Travis failures, but can not be used because
Avocado 78.0 is not released yet.

Patches 3 and 5 are not reviewed.

Up-to-patch 5 job:
https://travis-ci.org/github/philmd/qemu/jobs/670645531

All patches job:
https://travis-ci.org/github/philmd/qemu/jobs/670645611

Oksana Vohchana (1):
  Acceptance test: Fix to EXEC migration

Philippe Mathieu-Daudé (7):
  tests/acceptance/ppc_prep_40p: Use cdn.netbsd.org hostname
  tests/acceptance/ppc_prep_40p: Use mirror for ftp.software.ibm.com
  tests/acceptance/machine_sparc_leon3: Disable HelenOS test
  .travis.yml: Cache acceptance-test assets
  tests/Makefile: Add fetch-acceptance-assets rule
  .travis.yml: Run fetch-acceptance-assets before check-acceptance
  NOTFORMERGE Kludge for Avocado issue #3661

 .travis.yml                             | 4 +++-
 tests/Makefile.include                  | 7 +++++++
 tests/acceptance/machine_sparc_leon3.py | 4 ++++
 tests/acceptance/migration.py           | 6 +++---
 tests/acceptance/ppc_prep_40p.py        | 7 ++++---
 tests/requirements.txt                  | 2 +-
 6 files changed, 22 insertions(+), 8 deletions(-)

-- 
2.21.1

[PATCH-for-5.0 1/8] Acceptance test: Fix to EXEC migration

[Philippe Mathieu-Daudé]

From: Oksana Vohchana <ovoshcha@redhat.com>

The exec migration test isn't run a whole test scenario.
This patch fixes it

Fixes: 2e768cb682bf
Signed-off-by: Oksana Vohchana <ovoshcha@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20200325113138.20337-1-ovoshcha@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/migration.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/acceptance/migration.py b/tests/acceptance/migration.py
index a8367ca023..0365289cda 100644
--- a/tests/acceptance/migration.py
+++ b/tests/acceptance/migration.py
@@ -70,8 +70,8 @@ def test_migration_with_unix(self):
 
     @skipUnless(find_command('nc', default=False), "'nc' command not found")
     def test_migration_with_exec(self):
-        """
-        The test works for both netcat-traditional and netcat-openbsd packages
-        """
+        """The test works for both netcat-traditional and netcat-openbsd packages."""
         free_port = self._get_free_port()
         dest_uri = 'exec:nc -l localhost %u' % free_port
+        src_uri = 'exec:nc localhost %u' % free_port
+        self.do_migrate(dest_uri, src_uri)
-- 
2.21.1

[PATCH-for-5.0 2/8] tests/acceptance/ppc_prep_40p: Use cdn.netbsd.org hostname

[Philippe Mathieu-Daudé]

Use NetBSD content delivery network to get faster downloads.

Suggested-by: Kamil Rytarowski <kamil@netbsd.org>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20200211134504.9156-1-philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/ppc_prep_40p.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/acceptance/ppc_prep_40p.py
index b27572f212..138064285a 100644
--- a/tests/acceptance/ppc_prep_40p.py
+++ b/tests/acceptance/ppc_prep_40p.py
@@ -34,7 +34,7 @@ def test_factory_firmware_and_netbsd(self):
                     '7020-40p/P12H0456.IMG')
         bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
         bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)
-        drive_url = ('https://ftp.netbsd.org/pub/NetBSD/NetBSD-archive/'
+        drive_url = ('https://cdn.netbsd.org/pub/NetBSD/NetBSD-archive/'
                      'NetBSD-4.0/prep/installation/floppy/generic_com0.fs')
         drive_hash = 'dbcfc09912e71bd5f0d82c7c1ee43082fb596ceb'
         drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash)
@@ -66,7 +66,7 @@ def test_openbios_and_netbsd(self):
         :avocado: tags=arch:ppc
         :avocado: tags=machine:40p
         """
-        drive_url = ('https://ftp.netbsd.org/pub/NetBSD/iso/7.1.2/'
+        drive_url = ('https://cdn.netbsd.org/pub/NetBSD/iso/7.1.2/'
                      'NetBSD-7.1.2-prep.iso')
         drive_hash = 'ac6fa2707d888b36d6fa64de6e7fe48e'
         drive_path = self.fetch_asset(drive_url, asset_hash=drive_hash,
-- 
2.21.1

[PATCH-for-5.0 3/8] tests/acceptance/ppc_prep_40p: Use mirror for ftp.software.ibm.com

[Philippe Mathieu-Daudé]

To avoid regular failures on Travis-CI with ftp.software.ibm.com,
use a mirror.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/ppc_prep_40p.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/acceptance/ppc_prep_40p.py
index 138064285a..1515561249 100644
--- a/tests/acceptance/ppc_prep_40p.py
+++ b/tests/acceptance/ppc_prep_40p.py
@@ -30,7 +30,8 @@ def test_factory_firmware_and_netbsd(self):
         :avocado: tags=machine:40p
         :avocado: tags=slowness:high
         """
-        bios_url = ('ftp://ftp.boulder.ibm.com/rs6000/firmware/'
+        bios_url = ('http://ftpmirror.your.org/pub/misc/'
+                    'ftp.software.ibm.com/rs6000/firmware/'
                     '7020-40p/P12H0456.IMG')
         bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
         bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)
-- 
2.21.1

[PATCH-for-5.0 4/8] tests/acceptance/machine_sparc_leon3: Disable HelenOS test

[Philippe Mathieu-Daudé]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

This test was written/tested around beginning of 2019, but was
extracted from a bigger series and posted end of June 2019 [*].
Unfortunately I did not notice commit 162abf1a8 was merged by
then, which implements the AHB and APB plug and play devices.

HelenOS 0.6 is expecting the PnP registers to be not implemented
by QEMU, then forces the discovered AMBA devices (see [2]).

Before 162abf1a8, the console was displaying:

  HelenOS bootloader, release 0.6.0 (Elastic Horse)
  Built on 2014-12-21 20:17:42 for sparc32
  Copyright (c) 2001-2014 HelenOS project
   0x4000bf20|0x4000bf20: kernel image (496640/128466 bytes)
   0x4002b4f2|0x4002b4f2: ns image (154195/66444 bytes)
   0x4003b87e|0x4003b87e: loader image (153182/66437 bytes)
   0x4004bc03|0x4004bc03: init image (155339/66834 bytes)
   0x4005c115|0x4005c115: locsrv image (162063/70267 bytes)
   0x4006d390|0x4006d390: rd image (152678/65889 bytes)
   0x4007d4f1|0x4007d4f1: vfs image (168480/73394 bytes)
   0x4008f3a3|0x4008f3a3: logger image (158034/68368 bytes)
   0x4009feb3|0x4009feb3: ext4fs image (234510/100301 bytes)
   0x400b8680|0x400b8680: initrd image (8388608/1668901 bytes)
  ABMA devices:
  <1:00c> at 0x80000100 irq 3
  <1:00d> at 0x80000200
  <1:011> at 0x80000300 irq 8
  Memory size: 64 MB

As of this commit, it is now confused:

  ABMA devices:
  <1:3000> at 0x00000000 irq 0
  <1:3000> at 0x00000000 irq 0
  <1:3000> at 0x00000000 irq 0
  <1:3000> at 0x00000000 irq 0
  <1:3000> at 0x00000000 irq 0
  <1:3000> at 0x00000000 irq 0
  <1:3000> at 0x00000000 irq 0
  ...

As this test is not working as expected, simply disable it (by
skipping it) for now.

[1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg627094.html
[2] https://github.com/HelenOS/helenos/blob/0.6.0/boot/arch/sparc32/src/ambapp.c#L75

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200331105048.27989-2-f4bug@amsat.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/acceptance/machine_sparc_leon3.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/acceptance/machine_sparc_leon3.py b/tests/acceptance/machine_sparc_leon3.py
index f77e210ccb..2405cd7a0d 100644
--- a/tests/acceptance/machine_sparc_leon3.py
+++ b/tests/acceptance/machine_sparc_leon3.py
@@ -7,12 +7,16 @@
 
 from avocado_qemu import Test
 from avocado_qemu import wait_for_console_pattern
+from avocado import skip
 
 
 class Leon3Machine(Test):
 
     timeout = 60
 
+    @skip("Test currently broken")
+    # A Window Underflow exception occurs before booting the kernel,
+    # and QEMU exit calling cpu_abort(), which makes this test to fail.
     def test_leon3_helenos_uimage(self):
         """
         :avocado: tags=arch:sparc
-- 
2.21.1

[PATCH-for-5.0 5/8] .travis.yml: Cache acceptance-test assets

[Philippe Mathieu-Daudé]

Keep all acceptance-test assets in the same cache bucket.

As of v5.0.0-rc1, the cache is 2610.11MB:
https://travis-ci.org/github/philmd/qemu/jobs/670558103

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index e0c72210b7..2fd63eceaa 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -318,8 +318,10 @@ jobs:
       env:
         - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
         - TEST_CMD="make check-acceptance"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
       after_script:
         - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
+        - du -chs $HOME/avocado/data/cache
       addons:
         apt:
           packages:
-- 
2.21.1

[PATCH-for-5.1 6/8] tests/Makefile: Add fetch-acceptance-assets rule

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/Makefile.include | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 51de676298..90f457593c 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -906,6 +906,13 @@ get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
+fetch-acceptance-assets: check-venv
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            --show=$(if $(DEBUG),avocado.test,$(AVOCADO_SHOW)) assets fetch \
+            tests/acceptance/*py, \
+            "AVOCADO", "tests/acceptance")
+
 check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
-- 
2.21.1

[PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 2fd63eceaa..c6b32da447 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -317,7 +317,7 @@ jobs:
       dist: bionic
       env:
         - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets check-acceptance DEBUG=1"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
       after_script:
         - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
-- 
2.21.1

[NOTFORMERGE PATCH 8/8] Kludge for Avocado issue #3661

[Philippe Mathieu-Daudé]

Use a feature from unreleased Avocado v78.0.
See https://github.com/avocado-framework/avocado/issues/3661

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/requirements.txt b/tests/requirements.txt
index f9c84b4ba1..d625b32dbb 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,5 +1,5 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==76.0
+-e git+https://github.com/avocado-framework/avocado.git@f9b4dc7c58a6424eb8d0ed6781a1d76ae3a5ab06#egg=avocado-framework
 pycdlib==1.9.0
-- 
2.21.1

Re: [PATCH-for-5.0 3/8] tests/acceptance/ppc_prep_40p: Use mirror for ftp.software.ibm.com

[Wainer dos Santos Moschetta]

On 4/3/20 2:29 PM, Philippe Mathieu-Daudé wrote:
> To avoid regular failures on Travis-CI with ftp.software.ibm.com,
> use a mirror.
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   tests/acceptance/ppc_prep_40p.py | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/acceptance/ppc_prep_40p.py
> index 138064285a..1515561249 100644
> --- a/tests/acceptance/ppc_prep_40p.py
> +++ b/tests/acceptance/ppc_prep_40p.py
> @@ -30,7 +30,8 @@ def test_factory_firmware_and_netbsd(self):
>           :avocado: tags=machine:40p
>           :avocado: tags=slowness:high
>           """
> -        bios_url = ('ftp://ftp.boulder.ibm.com/rs6000/firmware/'
> +        bios_url = ('http://ftpmirror.your.org/pub/misc/'
> +                    'ftp.software.ibm.com/rs6000/firmware/'
>                       '7020-40p/P12H0456.IMG')

I checked the bios file can be downloaded from the mirror. I cannot 
judge the mirror is more reliable than ftp.boulder.ibm.com or not, though.

Code-wise, looks good. So:

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

>           bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
>           bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)

Re: [PATCH-for-5.0 3/8] tests/acceptance/ppc_prep_40p: Use mirror for ftp.software.ibm.com

[Willian Rampazzo]

On Mon, Apr 6, 2020 at 10:55 AM Wainer dos Santos Moschetta
<wainersm@redhat.com> wrote:
>
>
> On 4/3/20 2:29 PM, Philippe Mathieu-Daudé wrote:
> > To avoid regular failures on Travis-CI with ftp.software.ibm.com,
> > use a mirror.
> >
> > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> > ---
> >   tests/acceptance/ppc_prep_40p.py | 3 ++-
> >   1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/tests/acceptance/ppc_prep_40p.py b/tests/acceptance/ppc_prep_40p.py
> > index 138064285a..1515561249 100644
> > --- a/tests/acceptance/ppc_prep_40p.py
> > +++ b/tests/acceptance/ppc_prep_40p.py
> > @@ -30,7 +30,8 @@ def test_factory_firmware_and_netbsd(self):
> >           :avocado: tags=machine:40p
> >           :avocado: tags=slowness:high
> >           """
> > -        bios_url = ('ftp://ftp.boulder.ibm.com/rs6000/firmware/'
> > +        bios_url = ('http://ftpmirror.your.org/pub/misc/'
> > +                    'ftp.software.ibm.com/rs6000/firmware/'
> >                       '7020-40p/P12H0456.IMG')
>
> I checked the bios file can be downloaded from the mirror. I cannot
> judge the mirror is more reliable than ftp.boulder.ibm.com or not, though.
>

During the tests I did to debug Avocado I constantly got a timeout on
FTP. The HTTP URL seems to be more reliable. So far,

Tested-by: Willian Rampazzo <willianr@gmail.com>
Reviewed-by: Willian Rampazzo <willianr@gmail.com>

> Code-wise, looks good. So:
>
> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> >           bios_hash = '1775face4e6dc27f3a6ed955ef6eb331bf817f03'
> >           bios_path = self.fetch_asset(bios_url, asset_hash=bios_hash)
>

Re: [PATCH-for-5.0 5/8] .travis.yml: Cache acceptance-test assets

[Wainer dos Santos Moschetta]

On 4/3/20 2:29 PM, Philippe Mathieu-Daudé wrote:
> Keep all acceptance-test assets in the same cache bucket.
>
> As of v5.0.0-rc1, the cache is 2610.11MB:
> https://travis-ci.org/github/philmd/qemu/jobs/670558103
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   .travis.yml | 2 ++
>   1 file changed, 2 insertions(+)

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

>
> diff --git a/.travis.yml b/.travis.yml
> index e0c72210b7..2fd63eceaa 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -318,8 +318,10 @@ jobs:
>         env:
>           - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>           - TEST_CMD="make check-acceptance"
> +        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
>         after_script:
>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> +        - du -chs $HOME/avocado/data/cache
>         addons:
>           apt:
>             packages:

Re: [PATCH-for-5.0 5/8] .travis.yml: Cache acceptance-test assets

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> Keep all acceptance-test assets in the same cache bucket.
>
> As of v5.0.0-rc1, the cache is 2610.11MB:
> https://travis-ci.org/github/philmd/qemu/jobs/670558103
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Acked-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .travis.yml | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/.travis.yml b/.travis.yml
> index e0c72210b7..2fd63eceaa 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -318,8 +318,10 @@ jobs:
>        env:
>          - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>          - TEST_CMD="make check-acceptance"
> +        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
>        after_script:
>          - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> +        - du -chs $HOME/avocado/data/cache
>        addons:
>          apt:
>            packages:


-- 
Alex Bennée

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .travis.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 2fd63eceaa..c6b32da447 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -317,7 +317,7 @@ jobs:
>        dist: bionic
>        env:
>          - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
> -        - TEST_CMD="make check-acceptance"
> +        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets
> check-acceptance DEBUG=1"

You could use TEST_BUILD_CMD for the fetching of acceptance tests - can
that be done in parallel?

Also no point in -j1 here - it's implied.

>          - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
>        after_script:
>          - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat


-- 
Alex Bennée

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

[Philippe Mathieu-Daudé]

On 4/6/20 5:31 PM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> 
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   .travis.yml | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 2fd63eceaa..c6b32da447 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -317,7 +317,7 @@ jobs:
>>         dist: bionic
>>         env:
>>           - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>> -        - TEST_CMD="make check-acceptance"
>> +        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets
>> check-acceptance DEBUG=1"
> 
> You could use TEST_BUILD_CMD for the fetching of acceptance tests - can
> that be done in parallel?

I'd say calling fetch-acceptance-assets parallelized is what seems to 
break this Travis job (which is why I enforced -j1), but I'll refer to 
Willian here.

> 
> Also no point in -j1 here - it's implied.

You are right, I wanted to be sure this still work even if the globlal 
script running the tests is updated. But the correct fix is probably to 
use .NOTPARALLEL in the fetch-acceptance-assets rule in the previous patch.

> 
>>           - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
>>         after_script:
>>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> 
> 

Re: [PATCH-for-5.1 6/8] tests/Makefile: Add fetch-acceptance-assets rule

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/Makefile.include | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 51de676298..90f457593c 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -906,6 +906,13 @@ get-vm-image-fedora-31-%: check-venv
>  # download all vm images, according to defined targets
>  get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>  
> +fetch-acceptance-assets: check-venv
> +	$(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m avocado \
> +            --show=$(if $(DEBUG),avocado.test,$(AVOCADO_SHOW)) assets fetch \
> +            tests/acceptance/*py, \
> +            "AVOCADO", "tests/acceptance")
> +

I'm wondering if we could expand this to a rule per-test group and
therefor allow parallel fetching of groups of assets?

>  check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>  	$(call quiet-command, \
>              $(TESTS_VENV_DIR)/bin/python -m avocado \


-- 
Alex Bennée

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

[Willian Rampazzo]

On Mon, Apr 6, 2020 at 12:39 PM Philippe Mathieu-Daudé
<philmd@redhat.com> wrote:
>
> On 4/6/20 5:31 PM, Alex Bennée wrote:
> >
> > Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> >
> >> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> >> ---
> >>   .travis.yml | 2 +-
> >>   1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/.travis.yml b/.travis.yml
> >> index 2fd63eceaa..c6b32da447 100644
> >> --- a/.travis.yml
> >> +++ b/.travis.yml
> >> @@ -317,7 +317,7 @@ jobs:
> >>         dist: bionic
> >>         env:
> >>           - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
> >> -        - TEST_CMD="make check-acceptance"
> >> +        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets
> >> check-acceptance DEBUG=1"
> >
> > You could use TEST_BUILD_CMD for the fetching of acceptance tests - can
> > that be done in parallel?

If by `parallel` you mean running it with other targets, the
fetch-acceptance-assets target can run at any time before the
check-acceptance runs, concurrently with different targets. Now, if by
`parallel` you mean fetch more than one asset at a time, right now it
is not supported by Avocado, assets are fetched one by one, but it is
an excellent idea. I have added it to my list.

>
> I'd say calling fetch-acceptance-assets parallelized is what seems to
> break this Travis job (which is why I enforced -j1), but I'll refer to
> Willian here.
>

As I mentioned above, no problem with running it in parallel to other
targets. The errors we saw when you were testing are not related to
the parallel run. It should be fixed in the next release of Avocado.

> >
> > Also no point in -j1 here - it's implied.
>
> You are right, I wanted to be sure this still work even if the globlal
> script running the tests is updated. But the correct fix is probably to
> use .NOTPARALLEL in the fetch-acceptance-assets rule in the previous patch.
>
> >
> >>           - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
> >>         after_script:
> >>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> >
> >
>

Re: [PATCH-for-5.1 6/8] tests/Makefile: Add fetch-acceptance-assets rule

[Willian Rampazzo]

On Mon, Apr 6, 2020 at 12:41 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>
> > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> > ---
> >  tests/Makefile.include | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index 51de676298..90f457593c 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -906,6 +906,13 @@ get-vm-image-fedora-31-%: check-venv
> >  # download all vm images, according to defined targets
> >  get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
> >
> > +fetch-acceptance-assets: check-venv
> > +     $(call quiet-command, \
> > +            $(TESTS_VENV_DIR)/bin/python -m avocado \
> > +            --show=$(if $(DEBUG),avocado.test,$(AVOCADO_SHOW)) assets fetch \
> > +            tests/acceptance/*py, \
> > +            "AVOCADO", "tests/acceptance")
> > +
>
> I'm wondering if we could expand this to a rule per-test group and
> therefor allow parallel fetching of groups of assets?

This is a valid idea! Additionally, my suggestion is to organize the
tests by subdirectories under `tests/acceptance/`. Doing so makes it
easy to handle the target rule, adding the subdirectory to the assets
fetch command instead of listing specific tests for each group. Making
a list of tests for each group requires extra editing to the makefile
target when a new test is added. This can lead to new tests being
missed in their groups by mistake.

>
> >  check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> >       $(call quiet-command, \
> >              $(TESTS_VENV_DIR)/bin/python -m avocado \
>
>
> --
> Alex Bennée
>

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

[Alex Bennée]

Willian Rampazzo <wrampazz@redhat.com> writes:

> On Mon, Apr 6, 2020 at 12:39 PM Philippe Mathieu-Daudé
> <philmd@redhat.com> wrote:
>>
>> On 4/6/20 5:31 PM, Alex Bennée wrote:
>> >
>> > Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>> >
>> >> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> >> ---
>> >>   .travis.yml | 2 +-
>> >>   1 file changed, 1 insertion(+), 1 deletion(-)
>> >>
>> >> diff --git a/.travis.yml b/.travis.yml
>> >> index 2fd63eceaa..c6b32da447 100644
>> >> --- a/.travis.yml
>> >> +++ b/.travis.yml
>> >> @@ -317,7 +317,7 @@ jobs:
>> >>         dist: bionic
>> >>         env:
>> >>           - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>> >> -        - TEST_CMD="make check-acceptance"
>> >> +        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets
>> >> check-acceptance DEBUG=1"
>> >
>> > You could use TEST_BUILD_CMD for the fetching of acceptance tests - can
>> > that be done in parallel?
>
> If by `parallel` you mean running it with other targets, the
> fetch-acceptance-assets target can run at any time before the
> check-acceptance runs, concurrently with different targets. Now, if by
> `parallel` you mean fetch more than one asset at a time, right now it
> is not supported by Avocado, assets are fetched one by one, but it is
> an excellent idea. I have added it to my list.

See the comment I made about the make file. We could expand the
fetch-acceptance-assets target to have one dependency per python file so
you could invoke avocado in parallel to fetch the assets for each test
group in parallel. Of course this works best if the assets mentioned in
each file are unique otherwise avocado might race with itself.

>> I'd say calling fetch-acceptance-assets parallelized is what seems to
>> break this Travis job (which is why I enforced -j1), but I'll refer to
>> Willian here.
>>
>
> As I mentioned above, no problem with running it in parallel to other
> targets. The errors we saw when you were testing are not related to
> the parallel run. It should be fixed in the next release of Avocado.
>
>> >
>> > Also no point in -j1 here - it's implied.
>>
>> You are right, I wanted to be sure this still work even if the globlal
>> script running the tests is updated. But the correct fix is probably to
>> use .NOTPARALLEL in the fetch-acceptance-assets rule in the previous patch.
>>
>> >
>> >>           - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
>> >>         after_script:
>> >>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
>> >
>> >
>>


-- 
Alex Bennée

Re: [PATCH-for-5.1 6/8] tests/Makefile: Add fetch-acceptance-assets rule

[Alex Bennée]

Willian Rampazzo <wrampazz@redhat.com> writes:

> On Mon, Apr 6, 2020 at 12:41 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>
>> > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> > ---
>> >  tests/Makefile.include | 7 +++++++
>> >  1 file changed, 7 insertions(+)
>> >
>> > diff --git a/tests/Makefile.include b/tests/Makefile.include
>> > index 51de676298..90f457593c 100644
>> > --- a/tests/Makefile.include
>> > +++ b/tests/Makefile.include
>> > @@ -906,6 +906,13 @@ get-vm-image-fedora-31-%: check-venv
>> >  # download all vm images, according to defined targets
>> >  get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>> >
>> > +fetch-acceptance-assets: check-venv
>> > +     $(call quiet-command, \
>> > +            $(TESTS_VENV_DIR)/bin/python -m avocado \
>> > +            --show=$(if $(DEBUG),avocado.test,$(AVOCADO_SHOW)) assets fetch \
>> > +            tests/acceptance/*py, \
>> > +            "AVOCADO", "tests/acceptance")
>> > +
>>
>> I'm wondering if we could expand this to a rule per-test group and
>> therefor allow parallel fetching of groups of assets?
>
> This is a valid idea! Additionally, my suggestion is to organize the
> tests by subdirectories under `tests/acceptance/`. Doing so makes it
> easy to handle the target rule, adding the subdirectory to the assets
> fetch command instead of listing specific tests for each group. Making
> a list of tests for each group requires extra editing to the makefile
> target when a new test is added. This can lead to new tests being
> missed in their groups by mistake.

I'd use wildcards to build up the groups so the Makefile doesn't need to
be messed with again.

>
>>
>> >  check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> >       $(call quiet-command, \
>> >              $(TESTS_VENV_DIR)/bin/python -m avocado \
>>
>>
>> --
>> Alex Bennée
>>


-- 
Alex Bennée

Re: [PATCH-for-5.1 7/8] .travis.yml: Run fetch-acceptance-assets before check-acceptance

[Willian Rampazzo]

On Mon, Apr 6, 2020 at 3:21 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Willian Rampazzo <wrampazz@redhat.com> writes:
>
> > On Mon, Apr 6, 2020 at 12:39 PM Philippe Mathieu-Daudé
> > <philmd@redhat.com> wrote:
> >>
> >> On 4/6/20 5:31 PM, Alex Bennée wrote:
> >> >
> >> > Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> >> >
> >> >> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> >> >> ---
> >> >>   .travis.yml | 2 +-
> >> >>   1 file changed, 1 insertion(+), 1 deletion(-)
> >> >>
> >> >> diff --git a/.travis.yml b/.travis.yml
> >> >> index 2fd63eceaa..c6b32da447 100644
> >> >> --- a/.travis.yml
> >> >> +++ b/.travis.yml
> >> >> @@ -317,7 +317,7 @@ jobs:
> >> >>         dist: bionic
> >> >>         env:
> >> >>           - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
> >> >> -        - TEST_CMD="make check-acceptance"
> >> >> +        - TEST_CMD="travis_retry make -j1 fetch-acceptance-assets
> >> >> check-acceptance DEBUG=1"
> >> >
> >> > You could use TEST_BUILD_CMD for the fetching of acceptance tests - can
> >> > that be done in parallel?
> >
> > If by `parallel` you mean running it with other targets, the
> > fetch-acceptance-assets target can run at any time before the
> > check-acceptance runs, concurrently with different targets. Now, if by
> > `parallel` you mean fetch more than one asset at a time, right now it
> > is not supported by Avocado, assets are fetched one by one, but it is
> > an excellent idea. I have added it to my list.
>
> See the comment I made about the make file. We could expand the
> fetch-acceptance-assets target to have one dependency per python file so
> you could invoke avocado in parallel to fetch the assets for each test
> group in parallel. Of course this works best if the assets mentioned in
> each file are unique otherwise avocado might race with itself.

I see what you mean. About the unique asset definitions, this should
not be a big problem as Avocado downloads the asset on a temporary
file first and then copy it to the target file. In the worst case,
multiple instances of Avocado will download the same asset at the same
time, but just one will make it the target asset. The other instances
will fail, but as one already succeeded, the downside is the time and
bandwidth spent by the other instances.

>
> >> I'd say calling fetch-acceptance-assets parallelized is what seems to
> >> break this Travis job (which is why I enforced -j1), but I'll refer to
> >> Willian here.
> >>
> >
> > As I mentioned above, no problem with running it in parallel to other
> > targets. The errors we saw when you were testing are not related to
> > the parallel run. It should be fixed in the next release of Avocado.
> >
> >> >
> >> > Also no point in -j1 here - it's implied.
> >>
> >> You are right, I wanted to be sure this still work even if the globlal
> >> script running the tests is updated. But the correct fix is probably to
> >> use .NOTPARALLEL in the fetch-acceptance-assets rule in the previous patch.
> >>
> >> >
> >> >>           - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
> >> >>         after_script:
> >> >>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> >> >
> >> >
> >>
>
>
> --
> Alex Bennée
>