Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

["Alex Bennée" <alex.bennee@linaro.org>]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> On 06/28/2018 01:23 PM, Alex Bennée wrote:
>>
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Similar to the BootLinuxConsoleX86_64 test:
>>> boot a Linux kernel on a Malta board and verify the serial is working.
>>>
>>> This test can be run using:
>>>
>>>     $ avocado run -t endian:big tests/acceptance
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>>  1 file changed, 38 insertions(+)
>>>
>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>> index 17dc8d58c1..72cf5e943c 100644
>>> --- a/tests/acceptance/boot_linux_console.py
>>> +++ b/tests/acceptance/boot_linux_console.py
>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>>                  break
>>>              if 'Kernel panic - not syncing' in msg:
>>>                  self.fail("Kernel panic reached")
>>> +
>>> +
>>> +class BootLinuxConsoleMips(Test):
>>> +    """
>>> +    Boots a mips Linux kernel and checks that the console is operational
>>> +    and the kernel command line is properly passed from QEMU to the kernel
>>> +
>>> +    :avocado: enable
>>> +    :avocado: tags=endian:big
>>> +    :avocado: tags=arch:mips
>>> +    :avocado: tags=board:malta
>>> +    """
>>> +
>>> +    arch = "mips"
>>> +    timeout = 60
>>> +
>>> +    def test(self):
>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>> +        kernel_path = self.fetch_asset(kernel_url,
>>> asset_hash=kernel_hash)
>>
>> I'm uncomfortable using "random" binaries of websites as the source of
>> our test kernels. I can see the justification for distro kernels as they
>> at least have the infrastructure to rebuild from source if you really
>> want to, but even then the distros don't cover a lot of the
>> architectures.
>
> And now I notice I made an mistake here :) I guess remember the Avocado
> team started using the SHA-1 hash as default and I suggested them to be
> able to use other hashes for this particular case, since Aurelien
> provided the MD5 hashes signed by his GPG key, which is signed/trusted
> by Peter and used to merge mips32 pulls.
>
> That would verify the QEMU community circle of trust right?

It's not so much the chain of trust and more repeatability of building
the test cases. I trust Aurel32's binaries are good test cases for MIPS
but at the moment I have no idea how to rebuild them which is a bit of
an issue given it is covered by the GPL.

>
> I don't think Avocado should parse the FTP/HTTP signed indexes, but a
> manual verification when merging this series should suffice.
>
>>
>> I had experimented with using docker based builds for building test
>> fixtures (see tests/docker/dockerbuilds):
>>
>>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
>>
>> As these tests are fairly simple boot tests that just need kernels maybe
>> we could look at tooling up the generation of these images in a
>> repeatable way - similar to the way vmtest builds VMs.
>
> Yes, I have another acceptance branch where I cross-build an old mipssim
> kernel to test the board, using the following:
> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html
>
> But preparing a Docker cross image, fetching the Linux kernel source,
> building it, takes a lot of time/storage I'd rather avoid; at least with
> Aurelien kernels, since they are known to work since years.

Well you can throw away the docker image as long as we have a mechanism
to dump out the final binary. I have no interest in forcing people to
keep checked out code using up space, just that they can re-build if
they need to.

>
>>> +
>>> +        self.vm.set_machine('malta')
>>> +        self.vm.set_console()
>>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>>> +        self.vm.add_args('-m', "64",
>>> +                         '-kernel', kernel_path,
>>> +                         '-append', kernel_command_line)
>>> +        self.vm.launch()
>>> +        console = self.vm.console_socket.makefile()
>>> +        console_logger = logging.getLogger('console')
>>> +        while True:
>>> +            msg = console.readline()
>>> +            console_logger.debug(msg.strip())
>>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>>> +                break
>>> +            if 'Kernel panic - not syncing' in msg:
>>> +                self.fail("Kernel panic reached")
>>
>> Of course for bonus points a simple rootfs with hackbench or some such
>> in it would be nice. But I appreciate this makes the building job a lot
>> more complex than just a kernel.
>
> My idea is to use the rootfs for larger tests, and tag the "Kernel
> panic" tests as "quick", so we can have a "make acceptance-speed" or
> similar.
>
> We can already test than many devices were initialized correctly quickly
> looking at this console output.

Yeah the kernel boot up is a pretty good smoke test (it's all most of
kernelci manages as well). However it certainly doesn't fully exercise
the translator. I've lost track of the number of bugs we found after
successfully booting an kernel because we were doing exhaustive
instruction testing.

>
> Regards,
>
> Phil.


--
Alex Bennée