venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth <thuth@redhat.com>]

On 28/06/2022 08.57, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
>>> On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>
>>>> On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
>>>>> On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>>>
>>>>>> On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
>>>>>>> On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>>>>>
>>>>>>>> On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
>>>>>>>>> The README file is added describing the directory structure and the purpose
>>>>>>>>> of every file it contains. It also describes how to add new tests, make changes
>>>>>>>>> to existing tests or bits config files or regenerate the bits software.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Ani Sinha <ani@anisinha.ca>
>>>>>>>>> ---
>>>>>>>>>   tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
>>>>>>>>>   1 file changed, 168 insertions(+)
>>>>>>>>>   create mode 100644 tests/qtest/acpi-bits/README
>>>>>>>>>
>>>>>>>>> diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
>>>>>>>>> new file mode 100644
>>>>>>>>> index 0000000000..97b15f1665
>>>>>>>>> --- /dev/null
>>>>>>>>> +++ b/tests/qtest/acpi-bits/README
>>>>>>>>> @@ -0,0 +1,168 @@
>>>>>>>>> +=============================================================================
>>>>>>>>> +ACPI/SMBIOS QTESTS USING BIOSBITS
>>>>>>>>> +=============================================================================
>>>>>>>>> +
>>>>>>>>> +Biosbits is a software written by Josh Triplett that can be downloaded by
>>>>>>>>> +visiting https://biosbits.org/. The github codebase can be found here:
>>>>>>>>> +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
>>>>>>>>> +the bios components such as acpi and smbios tables directly through acpica
>>>>>>>>> +bios interpreter (a freely available C based library written by Intel,
>>>>>>>>> +downloadable from https://acpica.org/ and is included with biosbits) without an
>>>>>>>>> +operating system getting involved in between.
>>>>>>>>> +There are several advantages to directly testing the bios in a real physical
>>>>>>>>> +machine or VM as opposed to indirectly discovering bios issues through the
>>>>>>>>> +operating system. For one thing, the OSes tend to hide bios problems from the
>>>>>>>>> +end user. The other is that we have more control of what we wanted to test
>>>>>>>>> +and how by directly using acpica interpreter on top of the bios on a running
>>>>>>>>> +system. More details on the inspiration for developing biosbits and its real
>>>>>>>>> +life uses can be found in (a) and (b).
>>>>>>>>> +This directory contains QEMU qtests written in python that exercizes the QEMU
>>>>>>>>> +bios components using biosbits and reports test failures.
>>>>>>>>> +
>>>>>>>>> +These tests use python virtual environment. In debian/ubuntu system, the tests
>>>>>>>>> +would require python3.8-venv and python3-pip packages to be installed.
>>>>>>>>
>>>>>>>> Why do we mess with venv and pip? Certainly possible but
>>>>>>>> what's wrong with using distro provided packages?
>>>>>>>
>>>>>>> There are two things:
>>>>>>> (a) We are already using pip and venv for our avocado based
>>>>>>> integration tests. Look for TESTS_VENV_DIR in Makefile.include under
>>>>>>> tests.
>>>>>>> (b) the venv is primarily needed because I wanted to take advantage of
>>>>>>> our rich python library that handles QEMU based machines. There are
>>>>>>> python qtest libraries as well. These are well tested and used with
>>>>>>> integration tests and I wanted to keep the test part of the code
>>>>>>> simple by simply reusing them. however, in order to use them, we need
>>>>>>> a venv environment within which these qemu python libraries are
>>>>>>> installed. Integration tests does the same thing.
>>>>>>>
>>>>>>> A note about my language of choice - python. I gave a lot of thoughts
>>>>>>> on this. We do not do a lot of stuff here. All we do is:
>>>>>>> (a) generate bits iso.
>>>>>>> (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
>>>>>>> (c) collect and analyze logs.
>>>>>>>
>>>>>>> We are not inspecting guest memory or manipulating devices or pci
>>>>>>> buses. We do not need the power of C here. We need something that is
>>>>>>> simple to write, easy to maintain and understand and can deal with
>>>>>>> things like manipulating text files and configs easily. Python seems a
>>>>>>> better fit for the role.
>>>>>>
>>>>>> No problem with that. So that's venv. But do we need pip and pulling
>>>>>> packages from the net during testing?
>>>>>
>>>>> We do that too. See requirements.txt in tests/
>>>>> Following two are downloaded:
>>>>> avocado-framework==88.1
>>>>> pycdlib==1.11.0
>>>>>
>>>>> Also see this line in Makefie.include:
>>>>>
>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>>>>
>>>> Right but that's avocado since it pulls lots of stuff from
>>>> the net anyway.
>>>> Are the libraries in question not packaged on major distros?
>>>
>>> Currently I only need this:
>>> https://github.com/python-tap/tappy
>>> which is the basic TAP processing library for python.
>>>
>>> It seems its only installed through pip:
>>> https://tappy.readthedocs.io/en/latest/
>>>
>>> I do not think this is packaged by default. It's such a basic library
>>> for parsing test output that maybe we can keep this somewhere within
>>> the python src tree? Not sure ...
>>
>> It's pretty small for sure. Another submodule?
> 
> Unlike BITS, this one is likely going to be maintained for a while and
> will receive new releases through
> https://pypi.org/project/tap.py/
> so forking is OK but someone has to keep this updated.
> 
> I am open to anything. Whatever feels right is fine to me.

John Snow is currently working on the "Pythonification" of various QEMU 
bits, I think you should loop him into this discussion, too.

  Thomas