Re: "make check-acceptance" takes way too long

[Cleber Rosa <crosa@redhat.com>]

On Tue, Feb 1, 2022 at 6:07 AM Kashyap Chamarthy <kchamart@redhat.com> wrote:
>
> On Tue, Jan 25, 2022 at 10:20:11AM +0100, Gerd Hoffmann wrote:
> >   Hi,
> >
> > > IMHO the ideal scenario would be for us to have a kernel, initrd
> > > containing just busybox tools for the key arch targets we care
> > > about. Those could be used with direct kernel boot or stuffed
> > > into a disk iamge. Either way, they would boot in ~1 second,
> > > even with TCG, and would be able to execute simple shell scripts
> > > to test a decent amount of QEMU functionality.
> >
> > I have some test images based on buildroot which are essentially that.
> > https://gitlab.com/kraxel/br-kraxel/
> >
> > Still a significant download, but much smaller than a full fedora or
> > ubuntu cloud image and it boots much faster too.  Not down to only one
> > second though.
>
> Any objection to using CirrOS[1] images for boot-testing?   FWIW,
> OpenStack upstream CI boots thousands of guests each day with these for
> many years now.  It boots quick, and also satisfies one of Peter's
> other requirements: AArch64 images.
>

Even though I strongly support CirrOS (see my reply to Dan), I strongly object
using it as the only OS on "boot tests" (that is, testing that QEMU can fully
boot a system).  The reason is because actual functional coverage is reduced
and detached from most real world scenarios (I'm not aware of CirrOS, Alpine
and similar distros being used significantly on real world workloads).

This is the reasoning behind tests such as
"tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm" which takes ~12
seconds to run on my 4 years old laptop.

Depending on what one considers a system to be booted, the existing approach
on "tests/avocado/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc" of
booting only a kernel / initrd is also valid.  That takes around 0.4
seconds with KVM
and ~2 seconds to run with TCG on my system.

> A downside of CirrOS is it doesn't have a package manager, so installing
> custom packages is a PITA.  The main use-case of CirrOS images
> is any kind of boot-testing only.
>
> To make the booting even quicker with CirrOS, do disable the "metadata
> service lookup" (this is queried 20 times) at boot time.  It can be
> trivially done by making this change in this file
> /etc/cirros-init/config (in the disk image):
>
>     - DATASOURCE_LIST="nocloud configdrive ec2"
>     + DATASOURCE_LIST="nocloud"
>

That's a good tip!

If CirrOS had better support for "nocloud"[1], the existing boot tests could
transparently use it.  For instance, you can currently do this:

$ ./tests/venv/bin/avocado vmimage get --distro=ubuntu --distro-version=20.04
The image was downloaded:
Provider Version Architecture File
ubuntu   20.04   amd64
/home/cleber/avocado/data/cache/by_location/ca6ab0fdb5d175bbf3dfc3d070511559f6eab449/ubuntu-20.04-server-cloudimg-amd64.img

$ ./tests/venv/bin/avocado run -p distro=ubuntu -p
distro_version=20.04
tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm

The "-p distro=cirros" works, but only up to the downloading/preparing
the image.
The lack of proper support for cloud-init/nocloud then breaks it. I
would be a bit
reluctant of adding another family of tests or a third way of dealing
with guests
because they implement a custom behavior for something that is supposed
to be so standard at this point (cloud-init / nocloud).

Regards,
- Cleber.

[1] https://github.com/cirros-dev/cirros/issues/67