Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210

From: Cleber Rosa <crosa@redhat.com>
To: Peter Maydell <peter.maydell@linaro.org>
Cc: "Frédéric Basse" <contact@fredericb.info>,
	"Eduardo Habkost" <ehabkost@redhat.com>,
	"Evgeny Voevodin" <e.voevodin@samsung.com>,
	"Bartlomiej Zolnierkiewicz" <b.zolnierkie@samsung.com>,
	"QEMU Developers" <qemu-devel@nongnu.org>,
	"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
	"Krzysztof Kozlowski" <krzk@kernel.org>,
	"Jean-Christophe Dubois" <jcd@tribudubois.net>,
	"Igor Mitsyanko" <i.mitsyanko@gmail.com>,
	qemu-arm <qemu-arm@nongnu.org>,
	"Dmitry Solodkiy" <d.solodkiy@samsung.com>,
	"Maksim Kozlov" <m.kozlov@samsung.com>,
	"Philippe Mathieu-Daudé" <philmd@redhat.com>,
	"Guenter Roeck" <linux@roeck-us.net>
Subject: Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210
Date: Tue, 8 Oct 2019 17:49:07 -0400	[thread overview]
Message-ID: <20191008214907.GA28137@localhost.localdomain> (raw)
In-Reply-To: <CAFEAcA8ME5U5=rNLRSvNx7LmanqHhn_KWj6qtgym_=W1M9WDoA@mail.gmail.com>

On Mon, Oct 07, 2019 at 05:28:49PM +0100, Peter Maydell wrote:
> On Sat, 5 Oct 2019 at 16:47, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> >
> > This test boots a Linux kernel on a smdkc210 board and verify
> > the serial output is working.
> >
> > The cpio image used comes from the linux-build-test project:
> > https://github.com/groeck/linux-build-test
> 
> Thanks for putting this test case together, very helpful.
> 
> > +        initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> > +                      '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
> > +                      'arm/rootfs-armv5.cpio.gz')
> 
> Do our other acceptance tests download random third-party
> (ie "not a well-known distro") binaries for the tests ?
> It seems a bit hazardous for reproducability and trustability
> reasons...
> 
> thanks
> -- PMM

A quick and dirty grep shows (excluding comments and docs):

   boot_linux_console.py:        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
   boot_linux_console.py:        deb_url = ('http://snapshot.debian.org/archive/debian/'
   boot_linux_console.py:        deb_url = ('http://snapshot.debian.org/archive/debian/'
   boot_linux_console.py:        deb_url = ('http://snapshot.debian.org/archive/debian/'
   boot_linux_console.py:        initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
   boot_linux_console.py:        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
   boot_linux_console.py:        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
   boot_linux_console.py:        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
   boot_linux_console.py:        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
   boot_linux_console.py:        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
   boot_linux_console.py:        uboot_url = ('https://raw.githubusercontent.com/'
   boot_linux_console.py:        spi_url = ('https://raw.githubusercontent.com/'
   boot_linux_console.py:        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
   boot_linux_console.py:        kernel_url = ('http://archive.debian.org/debian/dists/lenny/main/'
   boot_linux_console.py:        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
   linux_initrd.py:        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora/li'
   linux_initrd.py:        kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
   linux_ssh_mips_malta.py:        'be': {'image_url': ('https://people.debian.org/~aurel32/qemu/mips/'
   linux_ssh_mips_malta.py:        'le': {'image_url': ('https://people.debian.org/~aurel32/qemu/mipsel/'
   linux_ssh_mips_malta.py:        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
   linux_ssh_mips_malta.py:        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
   linux_ssh_mips_malta.py:        kernel_url = ('https://people.debian.org/~aurel32/qemu/mips/'
   linux_ssh_mips_malta.py:        kernel_url = ('https://people.debian.org/~aurel32/qemu/mipsel/'
   machine_m68k_nextcube.py:        rom_url = ('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/'

I find it hard to judge precisely how much of a third-party some of
these are.  I remember Philippe mentioning that one of them, I guess
the images used on linux_ssh_mips_malta.py, were "as official as it
gets" (my words, from my often misleading memory).

Reproducibility is definitely an issue, in the sense given that some
of these can indeed go away, but as long as they're available the hash
recorded in the test should guarantee that we're running the same
images.

Do you think we should do something different here?

Thanks!
- Cleber.