Re: "make check-acceptance" takes way too long

From: "Daniel P. Berrangé" <berrange@redhat.com>
To: "Alex Bennée" <alex.bennee@linaro.org>
Cc: "Peter Maydell" <peter.maydell@linaro.org>,
	"Thomas Huth" <thuth@redhat.com>,
	"Beraldo Leal" <bleal@redhat.com>,
	"QEMU Developers" <qemu-devel@nongnu.org>,
	"Markus Armbruster" <armbru@redhat.com>,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
	"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
	"Cleber Rosa" <crosa@redhat.com>
Subject: Re: "make check-acceptance" takes way too long
Date: Fri, 21 Jan 2022 15:21:55 +0000	[thread overview]
Message-ID: <YerPk2ABMHQf/3QP@redhat.com> (raw)
In-Reply-To: <875yqd6zib.fsf@linaro.org>

On Fri, Jan 21, 2022 at 12:23:23PM +0000, Alex Bennée wrote:
> 
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
> > On Fri, 21 Jan 2022 at 10:50, Markus Armbruster <armbru@redhat.com> wrote:
> >> No objection, but it's no replacement for looking into why these tests
> >> are so slow.
> >>
> >> The #1 reason for things being slow is not giving a damn :)
> >
> > See previous messages in the thread -- the test starts a
> > full-fat guest OS including UEFI boot, and it takes forever to
> > get to the login prompt because systemd is starting everything
> > including the kitchen sink.
> 
> There has to be a half-way house between booting a kernel until it fails
> to find a rootfs and running a full Ubuntu distro. Maybe just asking
> systemd to reach "rescue.target" would be enough to show the disks are
> up and userspace works.

Booting up full OS distros is useful, but at the same time I feel it
is too much as something to expect developers to do on any kind of
regular basis.

Ideally some decent amount of acceptance testing could be a standard
part of the 'make check', but that's impossible as long as we're
downloading large disk images or booting things that are very slow,
especially so with TCG.

IMHO the ideal scenario would be for us to have a kernel, initrd
containing just busybox tools for the key arch targets we care
about. Those could be used with direct kernel boot or stuffed
into a disk iamge. Either way, they would boot in ~1 second,
even with TCG, and would be able to execute simple shell scripts
to test a decent amount of QEMU functionality.

It wouldn't eliminate the need to test with full OS, but it
would let us have some acceptance testing run as standard with
'make check' in a decently fast time.  It would then be less
critical if the more thorough full OS tests were somewhat
slower than we'd like. We could just leave those as a scheduled
job to run overnight post-merge. If they do detect any problems
post-merge, then write a dedicated test scenario to replicate it
under the minimal kernel/initrd acceptance test so it'll be
caught pre-merge in future.

> Running the EFI firmware is probably useful coverage but I'm not sure
> how one passes command line args to the guest in that approach? Do we
> need to set a magic EFI variable?

Same as with SeaBIOS - if you're booting off the guest then its the
grub.conf that controls this, if you're booting with direct kernel
on the host then QEMU cli.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|