Hi Alex, Cleber. On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote: > On 06/28/2018 03:36 PM, Alex Bennée wrote: >> Philippe Mathieu-Daudé writes: >>> On 06/28/2018 01:23 PM, Alex Bennée wrote: >>>> Philippe Mathieu-Daudé writes: >>>> >>>>> Similar to the BootLinuxConsoleX86_64 test: >>>>> boot a Linux kernel on a Malta board and verify the serial is working. >>>>> >>>>> This test can be run using: >>>>> >>>>> $ avocado run -t endian:big tests/acceptance >>>>> >>>>> Signed-off-by: Philippe Mathieu-Daudé >>>>> --- >>>>> tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++ >>>>> 1 file changed, 38 insertions(+) >>>>> >>>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py >>>>> index 17dc8d58c1..72cf5e943c 100644 >>>>> --- a/tests/acceptance/boot_linux_console.py >>>>> +++ b/tests/acceptance/boot_linux_console.py >>>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test): >>>>> break >>>>> if 'Kernel panic - not syncing' in msg: >>>>> self.fail("Kernel panic reached") >>>>> + >>>>> + >>>>> +class BootLinuxConsoleMips(Test): >>>>> + """ >>>>> + Boots a mips Linux kernel and checks that the console is operational >>>>> + and the kernel command line is properly passed from QEMU to the kernel >>>>> + >>>>> + :avocado: enable >>>>> + :avocado: tags=endian:big >>>>> + :avocado: tags=arch:mips >>>>> + :avocado: tags=board:malta >>>>> + """ >>>>> + >>>>> + arch = "mips" >>>>> + timeout = 60 >>>>> + >>>>> + def test(self): >>>>> + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' >>>>> + 'vmlinux-3.2.0-4-4kc-malta') >>>>> + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' >>>>> + kernel_path = self.fetch_asset(kernel_url, >>>>> asset_hash=kernel_hash) >>>> >>>> I'm uncomfortable using "random" binaries of websites as the source of >>>> our test kernels. I can see the justification for distro kernels as they >>>> at least have the infrastructure to rebuild from source if you really >>>> want to, but even then the distros don't cover a lot of the >>>> architectures. Alex: I could find all the Linux kernel I'm interested to console-test with Avocado on the http://snapshot.debian.org/ archive website. For example Aurelien's one (more up-to-date) is available here: http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48 I also added a SH-4 test for the SM501 series of Zoltan BALATON using the kernel extracted from this distrib built kernel: http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30 The Debian distribution also provide the source package and the kernels can be simply rebuilt using make-kpkg or (make bindeb-pkg with more recent kernels). Would it be enough to satisfy the GPL requirements to provided that info in the header and use these handy pre-compiled kernels? Cleber: As you added tarballs and zip support, I'd also need a "dpkg" support in your fetch_asset() archive management... Cleber: And another future request is to be able to extract files from ISO images with the guestfish tool :P To be able to add a test for the Apple Machintosh Quadra 800 machine: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg08139.html >>> And now I notice I made an mistake here :) I guess remember the Avocado >>> team started using the SHA-1 hash as default and I suggested them to be >>> able to use other hashes for this particular case, since Aurelien >>> provided the MD5 hashes signed by his GPG key, which is signed/trusted >>> by Peter and used to merge mips32 pulls. >>> >>> That would verify the QEMU community circle of trust right? >> >> It's not so much the chain of trust and more repeatability of building >> the test cases. I trust Aurel32's binaries are good test cases for MIPS >> but at the moment I have no idea how to rebuild them which is a bit of >> an issue given it is covered by the GPL. > > OK! I didn't even think about that since this is a "Linux" kernel and a > "Debian" rootfs provided by a Debian developer, hosted on a Debian > server, so I'm pretty sure this is GPL compliant, but it makes sens. > > I'll find a way to have reproducible test binaries for acceptance testing. > >>> I don't think Avocado should parse the FTP/HTTP signed indexes, but a >>> manual verification when merging this series should suffice. >>> >>>> >>>> I had experimented with using docker based builds for building test >>>> fixtures (see tests/docker/dockerbuilds): >>>> >>>> https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2 >>>> >>>> As these tests are fairly simple boot tests that just need kernels maybe >>>> we could look at tooling up the generation of these images in a >>>> repeatable way - similar to the way vmtest builds VMs. >>> >>> Yes, I have another acceptance branch where I cross-build an old mipssim >>> kernel to test the board, using the following: >>> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html >>> >>> But preparing a Docker cross image, fetching the Linux kernel source, >>> building it, takes a lot of time/storage I'd rather avoid; at least with >>> Aurelien kernels, since they are known to work since years. >> >> Well you can throw away the docker image as long as we have a mechanism >> to dump out the final binary. I have no interest in forcing people to >> keep checked out code using up space, just that they can re-build if >> they need to. > > The Avocado guys said they have plenty of storage for assets. > A cool feature would be to have some config built by some CI builder > that then sign and push reproducible config + generated binary to the > assets storage automatically. > >>>>> + >>>>> + self.vm.set_machine('malta') >>>>> + self.vm.set_console() >>>>> + kernel_command_line = 'console=ttyS0 printk.time=0' >>>>> + self.vm.add_args('-m', "64", >>>>> + '-kernel', kernel_path, >>>>> + '-append', kernel_command_line) >>>>> + self.vm.launch() >>>>> + console = self.vm.console_socket.makefile() >>>>> + console_logger = logging.getLogger('console') >>>>> + while True: >>>>> + msg = console.readline() >>>>> + console_logger.debug(msg.strip()) >>>>> + if 'Kernel command line: %s' % kernel_command_line in msg: >>>>> + break >>>>> + if 'Kernel panic - not syncing' in msg: >>>>> + self.fail("Kernel panic reached") >>>> >>>> Of course for bonus points a simple rootfs with hackbench or some such >>>> in it would be nice. But I appreciate this makes the building job a lot >>>> more complex than just a kernel. >>> >>> My idea is to use the rootfs for larger tests, and tag the "Kernel >>> panic" tests as "quick", so we can have a "make acceptance-speed" or >>> similar. >>> >>> We can already test than many devices were initialized correctly quickly >>> looking at this console output. >> >> Yeah the kernel boot up is a pretty good smoke test (it's all most of >> kernelci manages as well). However it certainly doesn't fully exercise >> the translator. I've lost track of the number of bugs we found after >> successfully booting an kernel because we were doing exhaustive >> instruction testing. > > This is not indeed a translator test, but simple enough to test than > devices are instantiated, mapped at the correct address, detected by > Linux, probe the device enough to initialize it. > > I actually wrote this test after breaking a I/O device why a previous > Super I/O refactor, this test would have failed. > > You could say a simple "info mtree" parser would notice it too... but > acceptance testing is closer to real-world imho, and would be also > useful to catch changes in the guest code (as in "QEMU expects things > that way, but the Linux kernel updated to something new"). > > Regards, > > Phil. >