Re: "make check-acceptance" takes way too long

[Cleber Rosa <crosa@redhat.com>]

On Tue, Feb 1, 2022 at 12:01 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Feb 01, 2022 at 12:29:56AM -0500, Cleber Rosa wrote:
> >
> > Assuming this is about "Testing that QEMU can boot a full distro", I wouldn't
> > try to solve the problem by making the distro too slim to get to the
> > point of becoming
> > an unrealistic system.
>
> At a high level our with acceptance (integration) testing is of
> course to make sure that QEMU is correctly emulating a full virtual
> machine, such that we have confidence that it can run real world
> operating systems.
>
> There are a number of approaches to achieve that with varying
> tradeoffs.
>
>   - Testing with very specific tailored environments, running
>     very specific userspace tools and minimal kernel setup.
>
>     This can give us a pretty decent amount of coverage of
>     the core features of the emulated environment in a tightly
>     controlled amount of wallclock time. When it fails it ought
>     to be relatively easy to understand and debug.
>
>     The downside is that it is the QEMU code paths it hits are
>     going to be fairly static.
>
>
>   - Testing with arbitrary execution of real world OS images.
>
>     I think of this as a bit of scattergun approach. We're not
>     trying to tightly control what runs, we actually want it
>     to run alot of arbitrarily complex and unusual stuff.
>
>     This is going to be time consuming and is likely to have
>     higher false positive failure rates. It is worthwhile
>     because it is going to find the edge cases that you simply
>     won't detect any other way, because you can't even imagine
>     the problems that you're trying to uncover until you uncover
>     them by accident with a real OS workload.
>
>     It is kinda like fuzzing QEMU with an entire OS :-)
>
>
> Both of these approaches are valid/complementary and we should
> want to have both.
>

Agreed.

> Any test suite is only going to find bugs though if it is
> actually executed.
>
> As a contributor though the former is stuff I'm likely to be
> willing to run myself before sending patches, while the latter
> is stuff I'm just always going to punt to merge testing infra.
>
> We want to be wary of leaving too much to be caught at time
> of merge tests, because that puts a significant burden on the
> person responsible for merging code in QEMU.  We need our
> contributors to be motivated to run as much testing as possible
> ahead of submitting patches.
>
> > IMO the deal breaker with regards to test time can be solved more cheaply by
> > having and using KVM where these tests will run, and not running them by
> > default otherwise.  With the tagging mechanism we should be able to set a
> > condition such as: "If using TCG, exclude tests that boot a full blown distro.
> > If using KVM, do not criticize what gets booted".  Resulting in something
> > like:
>
> > Does that sound like something appropriate?
>
> Depends whether you only care about KVM or not. From a POV of QEMU
> community CI, I think it is valid to want to test TCG functionality
>
>

Maybe I wasn't clear enough.  I am suggesting that tests using TCG do
not run by default (on a "make check-avocado") if, and only if, they
are booting a complete OS.  That would  bring the time to run "make
check-avocado" to a fifth of its current time.

And to be clear, there are a *lot* of tests running TCG, but they
happen to boot kernel+initrd by default, so we're not necessarily
abandoning TCG at all.

Also, we can have another target, or option as suggested by others in
this thread, where those lengthy TCG based full fistro boot tests get
to run.

> > BTW, on the topic of "Using something as a base OS for scripts (tests) to run
> > on it", another possibility for using full blown OS would be to save
> > their initialized
> > state, and load it to memory for each test, saving the guest boot time.  This
> > should of course be done at the framework level and transparent to tests.
>
> There is *massive* virtue in simplicity & predictability for testing.
>
> Building more complex infrastructure to pre-initialize caches with
> clever techniques like saving running OS state is clever, but is
> certainly not simple or predictable. When that kind of stuff goes
> wrong, whoever gets to debug it is going to have a really bad day.
>
> This can be worth doing if there's no other viable approach to achieve
> the desired end goal. I don't think that's the case for our integration
> testing needs in QEMU though. There's masses of scope for us to explore
> testing with minimal tailored guest images/environments, before we need
> to resort to building more complex optimization strategies.
>

I'm aware and second that. Avocado-VT tests transitioned from a model
where VMs would, by default, be reused across tests, to a "start every
VM from scratch".  But, users can still opt-in to the "reuse VM model"
if they feel the tradeoff is valid.

Best regards!
- Cleber