[PATCH] avocado: use sha1 for fc31 imgs to avoid first time re-download

* [PATCH] avocado: use sha1 for fc31 imgs to avoid first time re-download
@ 2022-10-22 17:03 Daniel Henrique Barboza
  2022-10-24  9:02 ` Daniel P. Berrangé
  0 siblings, 1 reply; 10+ messages in thread
From: Daniel Henrique Barboza @ 2022-10-22 17:03 UTC (permalink / raw)
  To: qemu-devel
  Cc: Daniel Henrique Barboza, Alex Bennée, Thomas Huth,
	John Snow, Cleber Rosa, Wainer dos Santos Moschetta

'make check-avocado' will download any images that aren't present in the
cache via 'get-vm-images' in tests/Makefile.include. The target that
downloads fedora 31 images, get-vm-image-fedora-31, will use 'avocado
vmimage get  --distro=fedora --distro-version=31 --arch=(...)' to
download the image for each arch. Note that this command does not
support any argument to set the hash algorithm used and, based on the
avocado source code [1], DEFAULT_HASH_ALGORITHM is set to "sha1". The
sha1 hash is stored in a Fedora-Cloud-Base-31-1.9.{ARCH}.qcow2-CHECKSUM
in the cache.

Back in QEMU, the LinuxDistro class defines the images it is going to
use in the KNOWN_DISTROS attribute. All images have the 'checksum'
attribute, defined using sha256, that will make avocado check if it has
the right asset in the cache. So this means that we're downloading a
sha1 image during check-avocado, then trying to fetch it using a sha256
hash.

This doesn't work. Avocado will match the provided checksum, a sha256
hash, with the sha1 hash from the existing CHECKSUM file for the asset,
and will re-download the image again. Avocado doesn't take into account
that we're using a different hash algorithm than the one used to fetch
the image via get-vm-image-fedora31 (in avocado: utils/asset.py,
self.find_asset_file() and fetch(self, timeout=None). The new download will
take into account the chosen hash algorithm, and subsequent avocado runs
will not re-download the image again because now it's comparing sha256 with
sha256.

All of this, of course, assuming that the system was able to re-download
the image before the avocado test timeouts. A more "fun" scenario
happens if the network is not fast enough to download the image during
the avocado timeout, the test will fail and the sha1 image will remain
in the cache. And the test will keep trying to re-fetch it, and will
keep failing with a timeout.

There's something to be said about avocado matching different sha
algorithms instead of re-calculating the hash with the chosen algorithm
during fetch(), especially considering that the CHECKSUM file contains
the name of the algorithm used. An avocado feature request [2] was
opened to see if this is a feasible change to do.

For now, in QEMU, let's use sha1 for all Fedora 31 images. This will
immediately spares us at least one extra download for each Fedora 31
image that we're doing in all our CI runs.

[1] https://github.com/avocado-framework/avocado.git @ 942a5d6972906
[2] https://github.com/avocado-framework/avocado/issues/5496

Cc: Alex Bennée <alex.bennee@linaro.org>
Cc: Thomas Huth <thuth@redhat.com>
Cc: John Snow <jsnow@redhat.com>
Cc: Cleber Rosa <crosa@redhat.com>
Cc: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com>
---
 tests/avocado/avocado_qemu/__init__.py | 30 ++++++++++++++++++--------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/tests/avocado/avocado_qemu/__init__.py b/tests/avocado/avocado_qemu/__init__.py
index 910f3ba1ea..68c42948a3 100644
--- a/tests/avocado/avocado_qemu/__init__.py
+++ b/tests/avocado/avocado_qemu/__init__.py
@@ -423,12 +423,24 @@ class LinuxDistro:
     Holds information of known distros.
     """
     #: A collection of known distros and their respective image checksum
+    #
+    # 'get-vm-image-fedora-31' target from Makefile.include will
+    # download the images using avocado default 'sha1' algorithm.
+    # This happens because the command line 'avocado vmimage get'
+    # from Makefile.include is not able to handle an extra --algorithm
+    # argument.
+    #
+    # To avoid a re-download of a recent image download due to a sha
+    # mismatch in the first test run, all Fedora 31 images are using
+    # 'sha1' instead of 'sha256'.
+    #
+    # FIXME: revisit this change if/when avocado learns to do CHECKSUM
+    # matches using different algorithms.
     KNOWN_DISTROS = {
         'fedora': {
             '31': {
                 'x86_64':
-                {'checksum': ('e3c1b309d9203604922d6e255c2c5d09'
-                              '8a309c2d46215d8fc026954f3c5c27a0'),
+                {'checksum': ('f62f8eabbf3687ea610c495bd59551a0025f99b7'),
                  'pxeboot_url': ('https://archives.fedoraproject.org/'
                                  'pub/archive/fedora/linux/releases/31/'
                                  'Everything/x86_64/os/images/pxeboot/'),
@@ -438,8 +450,7 @@ class LinuxDistro:
                                    'console=ttyS0,115200n8'),
                 },
                 'aarch64':
-                {'checksum': ('1e18d9c0cf734940c4b5d5ec592facae'
-                              'd2af0ad0329383d5639c997fdf16fe49'),
+                {'checksum': ('39d1fe099cdecacae894480d421be57d18b4e854'),
                 'pxeboot_url': 'https://archives.fedoraproject.org/'
                                'pub/archive/fedora/linux/releases/31/'
                                'Everything/aarch64/os/images/pxeboot/',
@@ -450,11 +461,9 @@ class LinuxDistro:
                                   ' console=ttyAMA0'),
                 },
                 'ppc64':
-                {'checksum': ('7c3528b85a3df4b2306e892199a9e1e4'
-                              '3f991c506f2cc390dc4efa2026ad2f58')},
+                {'checksum': ('9993dc28e7c49ceb52125f9513130dfe2ace026c')},
                 's390x':
-                {'checksum': ('4caaab5a434fd4d1079149a072fdc789'
-                              '1e354f834d355069ca982fdcaf5a122d')},
+                {'checksum': ('455f017b82decf32f366e06e7a7d0f6da86f96a7')},
             },
             '32': {
                 'aarch64':
@@ -595,15 +604,18 @@ def download_boot(self):
         self.log.info('Downloading/preparing boot image')
         # Fedora 31 only provides ppc64le images
         image_arch = self.arch
+        hash_algorithm = 'sha256'
         if self.distro.name == 'fedora':
             if image_arch == 'ppc64':
                 image_arch = 'ppc64le'
+            if self.distro.version == '31':
+                hash_algorithm = 'sha1'
 
         try:
             boot = vmimage.get(
                 self.distro.name, arch=image_arch, version=self.distro.version,
                 checksum=self.distro.checksum,
-                algorithm='sha256',
+                algorithm=hash_algorithm,
                 cache_dir=self.cache_dirs[0],
                 snapshot_dir=self.workdir)
         except:
-- 
2.37.3



^ permalink raw reply related	[flat|nested] 10+ messages in thread