[PATCH 00/12] Introduce new acpi/smbios qtests using biosbits

[PATCH 00/12] Introduce new acpi/smbios qtests using biosbits

[Ani Sinha]

Biosbits is a software written by Josh Triplett that can be downloaded by
visiting https://biosbits.org/. The github codebase can be found here:
https://github.com/biosbits/bits/tree/master. It is a software that exercizes
the bios components such as acpi and smbios tables directly through acpica
bios interpreter (a freely available C based library written by Intel,
downloadable from https://acpica.org/ and is included with biosbits) without an
operating system getting involved in between.
There are several advantages to directly testing the bios in a real physical
machine or VM as opposed to indirectly discovering bios issues through the
operating system. For one thing, the OSes tend to hide bios problems from the
end user. The other is that we have more control of what we wanted to test
and how by directly using acpica interpreter on top of the bios on a running
system. More details on the inspiration for developing biosbits and its real
life uses can be found in (a) and (b).
This patchset contains QEMU qtests written in python that exercizes the QEMU
bios components using biosbits and reports test failures.

Details of each of the files added by this patchset are provided in the README
file which is part of Patch 11. Every effort to contact Josh, through various
means including email, twitter, linkedIn etc has failed. Hence, the changes to
build biosbits with the newer compiler, upgrade acpica and other changes are
currently maintained in a forked project in my personal github. We may want to
maintain bits in a separate fork in a stable repository that is accessible by
QEMU developers.

The newly introduced qtest currently only run for x86_64 platform. They pass
both when running make check on a baremetal box as well as from inside a vm.

Thanks to Igor M for pointing me to this work.

(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
(b) https://www.youtube.com/watch?v=36QIepyUuhg

Ani Sinha (12):
  qtest: meson.build changes required to integrate python based qtests
  acpi/tests/bits: add prebuilt bios bits zip archive
  acpi/tests/bits: add prebuilt bits generated grub modules and scripts
  acpi/tests/bits: initial commit of test scripts that are run by
    biosbits
  acpi/tests/bits: disable acpi PSS tests that are failing in biosbits
  acpi/tests/bits: add smilatency test suite from bits in order to
    disable it
  acpi/tests/bits: disable smilatency test since it does not pass
    everytime
  acpi/tests/bits: add biosbits config file for running bios tests
  acpi/tests/bits: add acpi and smbios python tests that uses biosbits
  acpi/tests/bits: add acpi bits qtest directory in meson for running
    tests
  acpi/tests/bits: add README file for bits qtests
  MAINTAINERS: add myself as the maintainer for acpi biosbits qtests

 MAINTAINERS                                   |    5 +
 tests/qtest/acpi-bits/README                  |  168 ++
 tests/qtest/acpi-bits/acpi-bits-test-venv.sh  |   59 +
 tests/qtest/acpi-bits/acpi-bits-test.py       |  327 +++
 .../qtest/acpi-bits/bits-config/bits-cfg.txt  |   18 +
 tests/qtest/acpi-bits/bits-config/meson.build |   11 +
 tests/qtest/acpi-bits/bits-tests/meson.build  |   11 +
 tests/qtest/acpi-bits/bits-tests/smbios.py    | 2430 +++++++++++++++++
 .../qtest/acpi-bits/bits-tests/smilatency.py  |  103 +
 tests/qtest/acpi-bits/bits-tests/testacpi.py  |  283 ++
 tests/qtest/acpi-bits/bits-tests/testcpuid.py |   83 +
 tests/qtest/acpi-bits/meson.build             |   39 +
 .../acpi-bits/prebuilt/bits-2095-grub.tar.gz  |  Bin 0 -> 41416278 bytes
 tests/qtest/acpi-bits/prebuilt/bits-2095.zip  |  Bin 0 -> 31922898 bytes
 tests/qtest/acpi-bits/prebuilt/meson.build    |   11 +
 tests/qtest/acpi-bits/requirements.txt        |    1 +
 tests/qtest/meson.build                       |    7 +-
 17 files changed, 3555 insertions(+), 1 deletion(-)
 create mode 100644 tests/qtest/acpi-bits/README
 create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh
 create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py
 create mode 100644 tests/qtest/acpi-bits/bits-config/bits-cfg.txt
 create mode 100644 tests/qtest/acpi-bits/bits-config/meson.build
 create mode 100644 tests/qtest/acpi-bits/bits-tests/meson.build
 create mode 100644 tests/qtest/acpi-bits/bits-tests/smbios.py
 create mode 100644 tests/qtest/acpi-bits/bits-tests/smilatency.py
 create mode 100644 tests/qtest/acpi-bits/bits-tests/testacpi.py
 create mode 100644 tests/qtest/acpi-bits/bits-tests/testcpuid.py
 create mode 100644 tests/qtest/acpi-bits/meson.build
 create mode 100644 tests/qtest/acpi-bits/prebuilt/bits-2095-grub.tar.gz
 create mode 100644 tests/qtest/acpi-bits/prebuilt/bits-2095.zip
 create mode 100644 tests/qtest/acpi-bits/prebuilt/meson.build
 create mode 100644 tests/qtest/acpi-bits/requirements.txt

-- 
2.25.1

[PATCH 01/12] qtest: meson.build changes required to integrate python based qtests

[Ani Sinha]

These are some basic changes required in meson.build file in order to
incorporate python based qtests later on. No new qtests have been added in this
change.

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/meson.build | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 31287a9173..ad52f1c81b 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -310,6 +310,8 @@ qtests += {'dbus-display-test': [dbus_display1, gio]}
 endif
 
 qtest_executables = {}
+other_deps = []
+
 foreach dir : target_dirs
   if not dir.endswith('-softmmu')
     continue
@@ -327,6 +329,7 @@ foreach dir : target_dirs
   endif
   qtest_env.set('G_TEST_DBUS_DAEMON', meson.project_source_root() / 'tests/dbus-vmstate-daemon.sh')
   qtest_env.set('QTEST_QEMU_BINARY', './qemu-system-' + target_base)
+  qtest_env.set('QTEST_SOURCE_ROOT', meson.project_source_root())
   if have_tools and have_vhost_user_blk_server
     qtest_env.set('QTEST_QEMU_STORAGE_DAEMON_BINARY', './storage-daemon/qemu-storage-daemon')
     test_deps += [qsd]
@@ -351,7 +354,7 @@ foreach dir : target_dirs
     endif
     test('qtest-@0@/@1@'.format(target_base, test),
          qtest_executables[test],
-         depends: [test_deps, qtest_emulator, emulator_modules],
+         depends: [test_deps, qtest_emulator, emulator_modules, other_deps],
          env: qtest_env,
          args: ['--tap', '-k'],
          protocol: 'tap',
-- 
2.25.1

[PATCH 04/12] acpi/tests/bits: initial commit of test scripts that are run by biosbits

[Ani Sinha]

This is initial commit of cpuid, acpi and smbios python test scripts for
biosbits to execute. No change has been made to them from the original code
written by the biosbits author Josh Triplett. They are required to be installed
into the bits iso file and then run from within the virtual machine booted off
with biosbits iso.

The original location of these tests are here:
https://github.com/biosbits/bits/blob/master/python/testacpi.py
https://github.com/biosbits/bits/blob/master/python/smbios.py
https://github.com/biosbits/bits/blob/master/python/testcpuid.py

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/acpi-bits/bits-tests/meson.build  |   11 +
 tests/qtest/acpi-bits/bits-tests/smbios.py    | 2430 +++++++++++++++++
 tests/qtest/acpi-bits/bits-tests/testacpi.py  |  283 ++
 tests/qtest/acpi-bits/bits-tests/testcpuid.py |   83 +
 4 files changed, 2807 insertions(+)
 create mode 100644 tests/qtest/acpi-bits/bits-tests/meson.build
 create mode 100644 tests/qtest/acpi-bits/bits-tests/smbios.py
 create mode 100644 tests/qtest/acpi-bits/bits-tests/testacpi.py
 create mode 100644 tests/qtest/acpi-bits/bits-tests/testcpuid.py

diff --git a/tests/qtest/acpi-bits/bits-tests/meson.build b/tests/qtest/acpi-bits/bits-tests/meson.build
new file mode 100644
index 0000000000..3056731a53
--- /dev/null
+++ b/tests/qtest/acpi-bits/bits-tests/meson.build
@@ -0,0 +1,11 @@
+test_files = ['smbios.py', 'testacpi.py', 'testcpuid.py']
+
+copytestfiles = custom_target('copy test files',
+  input : test_files,
+  output :  test_files,
+  command : ['cp', '@INPUT@', '@OUTDIR@'],
+  install : true,
+  install_dir : 'bits-tests',
+  build_by_default : true)
+
+other_deps += copytestfiles
diff --git a/tests/qtest/acpi-bits/bits-tests/smbios.py b/tests/qtest/acpi-bits/bits-tests/smbios.py
new file mode 100644
index 0000000000..9667d0542c
--- /dev/null
+++ b/tests/qtest/acpi-bits/bits-tests/smbios.py
@@ -0,0 +1,2430 @@
+# Copyright (c) 2015, Intel Corporation
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+#     * Redistributions of source code must retain the above copyright notice,
+#       this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright notice,
+#       this list of conditions and the following disclaimer in the documentation
+#       and/or other materials provided with the distribution.
+#     * Neither the name of Intel Corporation nor the names of its contributors
+#       may be used to endorse or promote products derived from this software
+#       without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""SMBIOS/DMI module."""
+
+import bits
+import bitfields
+import ctypes
+import redirect
+import struct
+import uuid
+import unpack
+import ttypager
+import sys
+
+class SMBIOS(unpack.Struct):
+    def __new__(cls):
+        if sys.platform == "BITS-EFI":
+            import efi
+            sm_ptr = efi.system_table.ConfigurationTableDict.get(efi.SMBIOS_TABLE_GUID)
+        else:
+            address = 0xF0000
+            mem = bits.memory(0xF0000, 0x10000)
+            for offset in range(0, len(mem), 16):
+                signature = (ctypes.c_char * 4).from_address(address + offset).value
+                if signature == "_SM_":
+                    entry_point_length = ctypes.c_ubyte.from_address(address + offset + 5).value
+                    csum = sum(map(ord, mem[offset:offset + entry_point_length])) & 0xff
+                    if csum == 0:
+                        sm_ptr = address + offset
+                        break
+            else:
+                return None
+
+        if not sm_ptr:
+            return None
+
+        sm = super(SMBIOS, cls).__new__(cls)
+        sm._header_memory = bits.memory(sm_ptr, 0x1f)
+        return sm
+
+    def __init__(self):
+        super(SMBIOS, self).__init__()
+        u = unpack.Unpackable(self._header_memory)
+        self.add_field('header', Header(u))
+        self._structure_memory = bits.memory(self.header.structure_table_address, self.header.structure_table_length)
+        u = unpack.Unpackable(self._structure_memory)
+        self.add_field('structures', unpack.unpack_all(u, _smbios_structures, self), unpack.format_each("\n\n{!r}"))
+
+    def structure_type(self, num):
+        '''Dumps structure of given Type if present'''
+        try:
+            types_present = [self.structures[x].smbios_structure_type for x in range(len(self.structures))]
+            matrix = dict()
+            for index in range(len(types_present)):
+                if types_present.count(types_present[index]) == 1:
+                    matrix[types_present[index]] = self.structures[index]
+                else: # if multiple structures of the same type, return a list of structures for the type number
+                    if matrix.has_key(types_present[index]):
+                        matrix[types_present[index]].append(self.structures[index])
+                    else:
+                        matrix[types_present[index]] = [self.structures[index]]
+            return matrix[num]
+        except:
+            print "Failure: Type {} - not found".format(num)
+
+class Header(unpack.Struct):
+    def __new__(cls, u):
+        return super(Header, cls).__new__(cls)
+
+    def __init__(self, u):
+        super(Header, self).__init__()
+        self.raw_data = u.unpack_rest()
+        u = unpack.Unpackable(self.raw_data)
+        self.add_field('anchor_string', u.unpack_one("4s"))
+        self.add_field('checksum', u.unpack_one("B"))
+        self.add_field('length', u.unpack_one("B"))
+        self.add_field('major_version', u.unpack_one("B"))
+        self.add_field('minor_version', u.unpack_one("B"))
+        self.add_field('max_structure_size', u.unpack_one("<H"))
+        self.add_field('entry_point_revision', u.unpack_one("B"))
+        self.add_field('formatted_area', u.unpack_one("5s"))
+        self.add_field('intermediate_anchor_string', u.unpack_one("5s"))
+        self.add_field('intermediate_checksum', u.unpack_one("B"))
+        self.add_field('structure_table_length', u.unpack_one("<H"))
+        self.add_field('structure_table_address', u.unpack_one("<I"))
+        self.add_field('number_structures', u.unpack_one("<H"))
+        self.add_field('bcd_revision', u.unpack_one("B"))
+        if not u.at_end():
+            self.add_field('data', u.unpack_rest())
+
+class SmbiosBaseStructure(unpack.Struct):
+    def __new__(cls, u, sm):
+        t = u.unpack_peek_one("B")
+        if cls.smbios_structure_type is not None and t != cls.smbios_structure_type:
+            return None
+        return super(SmbiosBaseStructure, cls).__new__(cls)
+
+    def __init__(self, u, sm):
+        super(SmbiosBaseStructure, self).__init__()
+        self.start_offset = u.offset
+        length = u.unpack_peek_one("<xB")
+        self.raw_data = u.unpack_raw(length)
+        self.u = unpack.Unpackable(self.raw_data)
+
+        self.strings_offset = u.offset
+        def unpack_string():
+            return "".join(iter(lambda: u.unpack_one("c"), "\x00"))
+        strings = list(iter(unpack_string, ""))
+        if not strings:
+            u.skip(1)
+
+        self.strings_length = u.offset - self.strings_offset
+        self.raw_strings = str(bits.memory(sm.header.structure_table_address + self.strings_offset, self.strings_length))
+
+        if len(strings):
+            self.strings = strings
+
+        self.add_field('type', self.u.unpack_one("B"))
+        self.add_field('length', self.u.unpack_one("B"))
+        self.add_field('handle', self.u.unpack_one("<H"))
+
+    def fini(self):
+        if not self.u.at_end():
+            self.add_field('data', self.u.unpack_rest())
+        del self.u
+
+    def fmtstr(self, i):
+        """Format the specified index and the associated string"""
+        return "{} '{}'".format(i, self.getstr(i))
+
+    def getstr(self, i):
+        """Get the string associated with the given index"""
+        if i == 0:
+            return "(none)"
+        if not hasattr(self, "strings"):
+            return "(error: structure has no strings)"
+        if i > len(self.strings):
+            return "(error: string index out of range)"
+        return self.strings[i - 1]
+
+class BIOSInformation(SmbiosBaseStructure):
+    smbios_structure_type = 0
+
+    def __init__(self, u, sm):
+        super(BIOSInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('vendor', u.unpack_one("B"), self.fmtstr)
+            self.add_field('version', u.unpack_one("B"), self.fmtstr)
+            self.add_field('starting_address_segment', u.unpack_one("<H"))
+            self.add_field('release_date', u.unpack_one("B"), self.fmtstr)
+            self.add_field('rom_size', u.unpack_one("B"))
+            self.add_field('characteristics', u.unpack_one("<Q"))
+            minor_version_str = str(sm.header.minor_version) # 34 is .34, 4 is .4, 41 is .41; compare ASCIIbetically to compare initial digits rather than numeric value
+            if (sm.header.major_version, minor_version_str) >= (2,"4"):
+                characteristic_bytes = 2
+            else:
+                characteristic_bytes = self.length - 0x12
+            self.add_field('characteristics_extensions', [u.unpack_one("B") for b in range(characteristic_bytes)])
+            if (sm.header.major_version, minor_version_str) >= (2,"4"):
+                self.add_field('major_release', u.unpack_one("B"))
+                self.add_field('minor_release', u.unpack_one("B"))
+                self.add_field('ec_major_release', u.unpack_one("B"))
+                self.add_field('ec_minor_release', u.unpack_one("B"))
+        except:
+            self.decode_failure = True
+            print "Error parsing BIOSInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class SystemInformation(SmbiosBaseStructure):
+    smbios_structure_type = 1
+
+    def __init__(self, u, sm):
+        super(SystemInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
+            self.add_field('product_name', u.unpack_one("B"), self.fmtstr)
+            self.add_field('version', u.unpack_one("B"), self.fmtstr)
+            self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x8:
+                self.add_field('uuid', uuid.UUID(bytes_le=u.unpack_one("16s")))
+                wakeup_types = {
+                    0: 'Reserved',
+                    1: 'Other',
+                    2: 'Unknown',
+                    3: 'APM Timer',
+                    4: 'Modem Ring',
+                    5: 'LAN Remote',
+                    6: 'Power Switch',
+                    7: 'PCI PME#',
+                    8: 'AC Power Restored'
+                }
+                self.add_field('wakeup_type', u.unpack_one("B"), unpack.format_table("{}", wakeup_types))
+            if self.length > 0x19:
+                self.add_field('sku_number', u.unpack_one("B"), self.fmtstr)
+                self.add_field('family', u.unpack_one("B"), self.fmtstr)
+        except:
+            self.decode_failure = True
+            print "Error parsing SystemInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+_board_types = {
+    1: 'Unknown',
+    2: 'Other',
+    3: 'Server Blade',
+    4: 'Connectivity Switch',
+    5: 'System Management Module',
+    6: 'Processor Module',
+    7: 'I/O Module',
+    8: 'Memory Module',
+    9: 'Daughter Board',
+    0xA: 'Motherboard',
+    0xB: 'Processor/Memory Module',
+    0xC: 'Processor/IO Module',
+    0xD: 'Interconnect Board'
+}
+
+class BaseboardInformation(SmbiosBaseStructure):
+    smbios_structure_type = 2
+
+    def __init__(self, u, sm):
+        super(BaseboardInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
+            self.add_field('product', u.unpack_one("B"), self.fmtstr)
+            self.add_field('version', u.unpack_one("B"), self.fmtstr)
+            self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
+
+            if self.length > 0x8:
+                self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
+
+            if self.length > 0x9:
+                self.add_field('feature_flags', u.unpack_one("B"))
+                self.add_field('hosting_board', bool(bitfields.getbits(self.feature_flags, 0)), "feature_flags[0]={}")
+                self.add_field('requires_daughter_card', bool(bitfields.getbits(self.feature_flags, 1)), "feature_flags[1]={}")
+                self.add_field('removable', bool(bitfields.getbits(self.feature_flags, 2)), "feature_flags[2]={}")
+                self.add_field('replaceable', bool(bitfields.getbits(self.feature_flags, 3)), "feature_flags[3]={}")
+                self.add_field('hot_swappable', bool(bitfields.getbits(self.feature_flags, 4)), "feature_flags[4]={}")
+
+            if self.length > 0xA:
+                self.add_field('location', u.unpack_one("B"), self.fmtstr)
+
+            if self.length > 0xB:
+                self.add_field('chassis_handle', u.unpack_one("<H"))
+
+            if self.length > 0xD:
+                self.add_field('board_type', u.unpack_one("B"), unpack.format_table("{}", _board_types))
+
+            if self.length > 0xE:
+                self.add_field('handle_count', u.unpack_one("B"))
+                if self.handle_count > 0:
+                    self.add_field('contained_object_handles', tuple(u.unpack_one("<H") for i in range(self.handle_count)))
+        except:
+            self.decode_failure = True
+            print "Error parsing BaseboardInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class SystemEnclosure(SmbiosBaseStructure):
+    smbios_structure_type = 3
+
+    def __init__(self, u, sm):
+        super(SystemEnclosure, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
+            self.add_field('enumerated_type', u.unpack_one("B"))
+            self.add_field('chassis_lock_present', bool(bitfields.getbits(self.enumerated_type, 7)), "enumerated_type[7]={}")
+            board_types = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'Desktop',
+                0x04: 'Low Profile Desktop',
+                0x05: 'Pizza Box',
+                0x06: 'Mini Tower',
+                0x07: 'Tower',
+                0x08: 'Portable',
+                0x09: 'Laptop',
+                0x0A: 'Notebook',
+                0x0B: 'Hand Held',
+                0x0C: 'Docking Station',
+                0x0D: 'All in One',
+                0x0E: 'Sub Notebook',
+                0x0F: 'Space-saving',
+                0x10: 'Lunch Box',
+                0x11: 'Main Server Chassis',
+                0x12: 'Expansion Chassis',
+                0x13: 'SubChassis',
+                0x14: 'Bus Expansion Chassis',
+                0x15: 'Peripheral Chassis',
+                0x16: 'RAID Chassis',
+                0x17: 'Rack Mount Chassis',
+                0x18: 'Sealed-case PC',
+                0x19: 'Multi-system chassis W',
+                0x1A: 'Compact PCI',
+                0x1B: 'Advanced TCA',
+                0x1C: 'Blade',
+                0x1D: 'Blade Enclosure',
+            }
+            self.add_field('system_enclosure_type', bitfields.getbits(self.enumerated_type, 6, 0), unpack.format_table("enumerated_type[6:0]={}", board_types))
+            self.add_field('version', u.unpack_one("B"), self.fmtstr)
+            self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
+            self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
+            minor_version_str = str(sm.header.minor_version) # 34 is .34, 4 is .4, 41 is .41; compare ASCIIbetically to compare initial digits rather than numeric value
+            if self.length > 9:
+                chassis_states = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Safe',
+                    0x04: 'Warning',
+                    0x05: 'Critical',
+                    0x06: 'Non-recoverable',
+                }
+                self.add_field('bootup_state', u.unpack_one("B"), unpack.format_table("{}", chassis_states))
+                self.add_field('power_supply_state', u.unpack_one("B"), unpack.format_table("{}", chassis_states))
+                self.add_field('thermal_state', u.unpack_one("B"), unpack.format_table("{}", chassis_states))
+                security_states = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'None',
+                    0x04: 'External interface locked out',
+                    0x05: 'External interface enabled',
+                }
+                self.add_field('security_status', u.unpack_one("B"), unpack.format_table("{}", security_states))
+            if self.length > 0xd:
+                self.add_field('oem_defined', u.unpack_one("<I"))
+            if self.length > 0x11:
+                self.add_field('height', u.unpack_one("B"))
+                self.add_field('num_power_cords', u.unpack_one("B"))
+                self.add_field('contained_element_count', u.unpack_one("B"))
+                self.add_field('contained_element_length', u.unpack_one("B"))
+            if getattr(self, 'contained_element_count', 0):
+                self.add_field('contained_elements', tuple(SystemEnclosureContainedElement(u, self.contained_element_length) for i in range(self.contained_element_count)))
+            if self.length > (0x15 + (getattr(self, 'contained_element_count', 0) * getattr(self, 'contained_element_length', 0))):
+                self.add_field('sku_number', u.unpack_one("B"), self.fmtstr)
+        except:
+            self.decode_failure = True
+            print "Error parsing SystemEnclosure"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class SystemEnclosureContainedElement(unpack.Struct):
+    def __init__(self, u, length):
+        super(SystemEnclosureContainedElement, self).__init__()
+        self.start_offset = u.offset
+        self.raw_data = u.unpack_raw(length)
+        self.u = unpack.Unpackable(self.raw_data)
+        u = self.u
+        self.add_field('contained_element_type', u.unpack_one("B"))
+        type_selections = {
+            0: 'SMBIOS baseboard type enumeration',
+            1: 'SMBIOS structure type enumeration',
+        }
+        self.add_field('type_select', bitfields.getbits(self.contained_element_type, 7), unpack.format_table("contained_element_type[7]={}", type_selections))
+        self.add_field('type', bitfields.getbits(self.contained_element_type, 6, 0))
+        if self.type_select == 0:
+            self.add_field('smbios_board_type', self.type, unpack.format_table("{}", _board_types))
+        else:
+            self.add_field('smbios_structure_type', self.type)
+        self.add_field('minimum', u.unpack_one("B"))
+        self.add_field('maximum', u.unpack_one("B"))
+        if not u.at_end():
+            self.add_field('data', u.unpack_rest())
+        del self.u
+
+class ProcessorInformation(SmbiosBaseStructure):
+    smbios_structure_type = 4
+
+    def __init__(self, u, sm):
+        super(ProcessorInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('socket_designation', u.unpack_one("B"), self.fmtstr)
+            processor_types = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'Central Processor',
+                0x04: 'Math Processor',
+                0x05: 'DSP Processor',
+                0x06: 'Video Processor',
+            }
+            self.add_field('processor_type', u.unpack_one("B"), unpack.format_table("{}", processor_types))
+            self.add_field('processor_family', u.unpack_one("B"))
+            self.add_field('processor_manufacturer', u.unpack_one("B"), self.fmtstr)
+            self.add_field('processor_id', u.unpack_one("<Q"))
+            self.add_field('processor_version', u.unpack_one("B"), self.fmtstr)
+            self.add_field('voltage', u.unpack_one("B"))
+            self.add_field('external_clock', u.unpack_one("<H"))
+            self.add_field('max_speed', u.unpack_one("<H"))
+            self.add_field('current_speed', u.unpack_one("<H"))
+            self.add_field('status', u.unpack_one("B"))
+            processor_upgrades = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'Daughter Board',
+                0x04: 'ZIF Socket',
+                0x05: 'Replaceable Piggy Back',
+                0x06: 'None',
+                0x07: 'LIF Socket',
+                0x08: 'Slot 1',
+                0x09: 'Slot 2',
+                0x0A: '370-pin socket',
+                0x0B: 'Slot A',
+                0x0C: 'Slot M',
+                0x0D: 'Socket 423',
+                0x0E: 'Socket A (Socket 462)',
+                0x0F: 'Socket 478',
+                0x10: 'Socket 754',
+                0x11: 'Socket 940',
+                0x12: 'Socket 939',
+                0x13: 'Socket mPGA604',
+                0x14: 'Socket LGA771',
+                0x15: 'Socket LGA775',
+                0x16: 'Socket S1',
+                0x17: 'Socket AM2',
+                0x18: 'Socket F (1207)',
+                0x19: 'Socket LGA1366',
+                0x1A: 'Socket G34',
+                0x1B: 'Socket AM3',
+                0x1C: 'Socket C32',
+                0x1D: 'Socket LGA1156',
+                0x1E: 'Socket LGA1567',
+                0x1F: 'Socket PGA988A',
+                0x20: 'Socket BGA1288',
+                0x21: 'Socket rPGA988B',
+                0x22: 'Socket BGA1023',
+                0x23: 'Socket BGA1224',
+                0x24: 'Socket BGA1155',
+                0x25: 'Socket LGA1356',
+                0x26: 'Socket LGA2011',
+                0x27: 'Socket FS1',
+                0x28: 'Socket FS2',
+                0x29: 'Socket FM1',
+                0x2A: 'Socket FM2',
+            }
+            self.add_field('processor_upgrade', u.unpack_one("B"), unpack.format_table("{}", processor_upgrades))
+            if self.length > 0x1A:
+                self.add_field('l1_cache_handle', u.unpack_one("<H"))
+                self.add_field('l2_cache_handle', u.unpack_one("<H"))
+                self.add_field('l3_cache_handle', u.unpack_one("<H"))
+            if self.length > 0x20:
+                self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
+                self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
+                self.add_field('part_number', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x24:
+                self.add_field('core_count', u.unpack_one("B"))
+                self.add_field('core_enabled', u.unpack_one("B"))
+                self.add_field('thread_count', u.unpack_one("B"))
+                self.add_field('processor_characteristics', u.unpack_one("<H"))
+            if self.length > 0x28:
+                self.add_field('processor_family_2', u.unpack_one("<H"))
+            if self.length > 0x2A:
+                self.add_field('core_count2', u.unpack_one("<H"))
+                self.add_field('core_enabled2', u.unpack_one("<H"))
+                self.add_field('thread_count2', u.unpack_one("<H"))
+        except:
+            self.decode_failure = True
+            print "Error parsing Processor Information"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class MemoryControllerInformation(SmbiosBaseStructure): #obsolete starting with v2.1
+    smbios_structure_type = 5
+
+    def __init__(self, u, sm):
+        super(MemoryControllerInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            _error_detecting_method = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'None',
+                0x04: '8-bit Parity',
+                0x05: '32-bit ECC',
+                0x06: '64-bit ECC',
+                0x07: '128-bit ECC',
+                0x08: 'CRC'
+                }
+            self.add_field('error_detecting_method', u.unpack_one("B"), unpack.format_table("{}", _error_detecting_method))
+            self.add_field('error_correcting_capability', u.unpack_one("B"))
+            _interleaves = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'One-Way Interleave',
+                0x04: 'Two-Way Interleave',
+                0x05: 'Four-Way Interleave',
+                0x06: 'Eight-Way Interleave',
+                0x07: 'Sixteen-Way Interleave'
+                }
+            self.add_field('supported_interleave', u.unpack_one("B"), unpack.format_table("{}", _interleaves))
+            self.add_field('current_interleave', u.unpack_one("B"), unpack.format_table("{}", _interleaves))
+            self.add_field('max_memory_module_size', u.unpack_one("B"), self.fmtstr)
+            self.add_field('supported_speeds', u.unpack_one("<H"))
+            self.add_field('supported_memory_types', u.unpack_one("<H"))
+            self.add_field('memory_module_voltage', u.unpack_one("B"))
+            self.add_field('req_voltage_b2', bitfields.getbits(self.memory_module_voltage, 2), "memory_module_voltage[2]={}")
+            self.add_field('req_voltage_b1', bitfields.getbits(self.memory_module_voltage, 1), "memory_module_voltage[1]={}")
+            self.add_field('req_voltage_b0', bitfields.getbits(self.memory_module_voltage, 0), "memory_module_voltage[0]={}")
+            self.add_field('num_associated_memory_slots', u.unpack_one("B"))
+            self.add_field('memory_module_configuration_handles', u.unpack_one("<(self.num_associated_memory_slots)H"))
+            self.add_field('enabled_error_correcting_capabilities', u.unpack_one("B"))
+        except:
+            self.decode_failure = True
+            print "Error parsing MemoryControllerInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class MemoryModuleInformation(SmbiosBaseStructure): #obsolete starting with v2.1
+    smbios_structure_type = 6
+
+    def __init__(self, u, sm):
+        super(MemoryModuleInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('socket_designation', u.unpack_one("B"), self.fmtstr)
+            self.add_field('bank_connections', u.unpack_one("B"))
+            self.add_field('current_speed', u.unpack_one("B"))
+            self.add_field('current_memory_type', u.unpack_one("<H"))
+            _mem_connection = {
+                0: 'single',
+                1: 'double-bank'
+                }
+            self.add_field('installed_mem', u.unpack_one("B"))
+            self.add_field('installed_size', bitfields.getbits(self.installed_mem, 6, 0), "installed_mem[6:0]={}")
+            self.add_field('installed_memory_module_connection', bitfields.getbits(self.installed_mem, 7), unpack.format_table("installed_mem[7]={}", _mem_connection))
+            self.add_field('enabled_mem', u.unpack_one("B"))
+            self.add_field('enabled_size', bitfields.getbits(self.installed_mem, 6, 0), "enabled_mem[6:0]={}")
+            self.add_field('enabled_memory_module_connection', bitfields.getbits(self.installed_mem, 7), unpack.format_table("enabled_mem[7]={}", _mem_connection))
+            self.add_field('error_status', u.unpack_one("B"))
+            self.add_field('error_status_info_obstained_from_event_log', bool(bitfields.getbits(self.error_status, 2)), unpack.format_table("error_status[2]={}", _mem_connection))
+            self.add_field('correctable_errors_received', bool(bitfields.getbits(self.error_status, 1)), unpack.format_table("error_status[1]={}", _mem_connection))
+            self.add_field('uncorrectable_errors_received', bool(bitfields.getbits(self.error_status, 0)), unpack.format_table("error_status[0]={}", _mem_connection))
+        except:
+            self.decode_failure = True
+            print "Error parsing MemoryModuleInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class CacheInformation(SmbiosBaseStructure):
+    smbios_structure_type = 7
+
+    def __init__(self, u, sm):
+        super(CacheInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('socket_designation', u.unpack_one("B"), self.fmtstr)
+            processor_types = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'Central Processor',
+                0x04: 'Math Processor',
+                0x05: 'DSP Processor',
+                0x06: 'Video Processor',
+            }
+            self.add_field('cache_configuration', u.unpack_one("<H"))
+            _operational_mode = {
+                0b00: 'Write Through',
+                0b01: 'Write Back',
+                0b10: 'Varies with Memory Address',
+                0b11: 'Unknown'
+                }
+            self.add_field('operational_mode', bitfields.getbits(self.cache_configuration, 9, 8), unpack.format_table("cache_configuration[9:8]={}", _operational_mode))
+            self.add_field('enabled_at_boot_time', bool(bitfields.getbits(self.cache_configuration, 7)), "cache_configuration[7]={}")
+            _location = {
+                0b00: 'Internal',
+                0b01: 'External',
+                0b10: 'Reserved',
+                0b11: 'Unknown'
+                }
+            self.add_field('location_relative_to_cpu_module', bitfields.getbits(self.cache_configuration, 6, 5), unpack.format_table("cache_configuration[6:5]={}", _location))
+            self.add_field('cache_socketed', bool(bitfields.getbits(self.cache_configuration, 3)), "cache_configuration[3]={}")
+            self.add_field('cache_level', bitfields.getbits(self.cache_configuration, 2, 0), "cache_configuration[2:0]={}")
+            self.add_field('max_cache_size', u.unpack_one("<H"))
+            _granularity = {
+                0: '1K granularity',
+                1: '64K granularity'
+                }
+            self.add_field('max_granularity', bitfields.getbits(self.cache_configuration, 15), unpack.format_table("max_cache_size[15]={}", _granularity))
+            self.add_field('max_size_in_granularity', bitfields.getbits(self.cache_configuration, 14, 0), "max_cache_size[14, 0]={}")
+            self.add_field('installed_size', u.unpack_one("<H"))
+            if self.installed_size != 0:
+                self.add_field('installed_granularity', bitfields.getbits(self.cache_configuration, 15), unpack.format_table("installed_size[15]={}", _granularity))
+                self.add_field('installed_size_in_granularity', bitfields.getbits(self.cache_configuration, 14, 0), "installed_size[14, 0]={}")
+            self.add_field('supported_sram_type', u.unpack_one("<H"))
+            self.add_field('current_sram_type', u.unpack_one("<H"))
+            if self.length > 0x0F:
+                self.add_field('cache_speed', u.unpack_one("B"))
+            if self.length > 0x10:
+                _error_correction = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'None',
+                    0x04: 'Parity',
+                    0x05: 'Single-bit ECC',
+                    0x06: 'Multi-bit ECC'
+                    }
+                self.add_field('error_correction', u.unpack_one("B"), unpack.format_table("{}", _error_correction))
+            if self.length > 0x10:
+                _system_cache_type = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Instruction',
+                    0x04: 'Data',
+                    0x05: 'Unified'
+                    }
+                self.add_field('system_cache_type', u.unpack_one("B"), unpack.format_table("{}", _system_cache_type))
+            if self.length > 0x12:
+                _associativity = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Direct Mapped',
+                    0x04: '2-way Set-Associative',
+                    0x05: '4-way Set-Associative',
+                    0x06: 'Fully Associative',
+                    0x07: '8-way Set-Associative',
+                    0x08: '16-way Set-Associative',
+                    0x09: '12-way Set-Associative',
+                    0x0A: '24-way Set-Associative',
+                    0x0B: '32-way Set-Associative',
+                    0x0C: '48-way Set-Associative',
+                    0x0D: '64-way Set-Associative',
+                    0x0E: '20-way Set-Associative'
+                    }
+                self.add_field('associativity', u.unpack_one("B"), unpack.format_table("{}", _associativity))
+
+        except:
+            self.decode_failure = True
+            print "Error parsing CacheInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class PortConnectorInfo(SmbiosBaseStructure):
+    smbios_structure_type = 8
+
+    def __init__(self, u, sm):
+        super(PortConnectorInfo, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('internal_reference_designator', u.unpack_one("B"), self.fmtstr)
+            connector_types = {
+                0x00: 'None',
+                0x01: 'Centronics',
+                0x02: 'Mini Centronics',
+                0x03: 'Proprietary',
+                0x04: 'DB-25 pin male',
+                0x05: 'DB-25 pin female',
+                0x06: 'DB-15 pin male',
+                0x07: 'DB-15 pin female',
+                0x08: 'DB-9 pin male',
+                0x09: 'DB-9 pin female',
+                0x0A: 'RJ-11',
+                0x0B: 'RJ-45',
+                0x0C: '50-pin MiniSCSI',
+                0x0D: 'Mini-DIN',
+                0x0E: 'Micro-DIN',
+                0x0F: 'PS/2',
+                0x10: 'Infrared',
+                0x11: 'HP-HIL',
+                0x12: 'Access Bus (USB)',
+                0x13: 'SSA SCSI',
+                0x14: 'Circular DIN-8 male',
+                0x15: 'Circular DIN-8 female',
+                0x16: 'On Board IDE',
+                0x17: 'On Board Floppy',
+                0x18: '9-pin Dual Inline (pin 10 cut)',
+                0x19: '25-pin Dual Inline (pin 26 cut)',
+                0x1A: '50-pin Dual Inline',
+                0x1B: '68-pin Dual Inline',
+                0x1C: 'On Board Sound Input from CD-ROM',
+                0x1D: 'Mini-Centronics Type-14',
+                0x1E: 'Mini-Centronics Type-26',
+                0x1F: 'Mini-jack (headphones)',
+                0x20: 'BNC',
+                0x21: '1394',
+                0x22: 'SAS/SATA Plug Receptacle',
+                0xA0: 'PC-98',
+                0xA1: 'PC-98Hireso',
+                0xA2: 'PC-H98',
+                0xA3: 'PC-98Note',
+                0xA4: 'PC-98Full',
+                0xFF: 'Other',
+            }
+            self.add_field('internal_connector_type', u.unpack_one("B"), unpack.format_table("{}", connector_types))
+            self.add_field('external_reference_designator', u.unpack_one("B"), self.fmtstr)
+            self.add_field('external_connector_type', u.unpack_one("B"), unpack.format_table("{}", connector_types))
+            port_types = {
+                0x00: 'None',
+                0x01: 'Parallel Port XT/AT Compatible',
+                0x02: 'Parallel Port PS/2',
+                0x03: 'Parallel Port ECP',
+                0x04: 'Parallel Port EPP',
+                0x05: 'Parallel Port ECP/EPP',
+                0x06: 'Serial Port XT/AT Compatible',
+                0x07: 'Serial Port 16450 Compatible',
+                0x08: 'Serial Port 16550 Compatible',
+                0x09: 'Serial Port 16550A Compatible',
+                0x0A: 'SCSI Port',
+                0x0B: 'MIDI Port',
+                0x0C: 'Joy Stick Port',
+                0x0D: 'Keyboard Port',
+                0x0E: 'Mouse Port',
+                0x0F: 'SSA SCSI',
+                0x10: 'USB',
+                0x11: 'FireWire (IEEE P1394)',
+                0x12: 'PCMCIA Type I2',
+                0x13: 'PCMCIA Type II',
+                0x14: 'PCMCIA Type III',
+                0x15: 'Cardbus',
+                0x16: 'Access Bus Port',
+                0x17: 'SCSI II',
+                0x18: 'SCSI Wide',
+                0x19: 'PC-98',
+                0x1A: 'PC-98-Hireso',
+                0x1B: 'PC-H98',
+                0x1C: 'Video Port',
+                0x1D: 'Audio Port',
+                0x1E: 'Modem Port',
+                0x1F: 'Network Port',
+                0x20: 'SATA',
+                0x21: 'SAS',
+                0xA0: '8251 Compatible',
+                0xA1: '8251 FIFO Compatible',
+                0xFF: 'Other',
+            }
+            self.add_field('port_type', u.unpack_one("B"), unpack.format_table("{}", port_types))
+        except:
+            self.decodeFailure = True
+            print "Error parsing PortConnectorInfo"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class SystemSlots(SmbiosBaseStructure):
+    smbios_structure_type = 9
+
+    def __init__(self, u, sm):
+        super(SystemSlots, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('designation', u.unpack_one("B"), self.fmtstr)
+            _slot_types = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'ISA',
+                0x04: 'MCA',
+                0x05: 'EISA',
+                0x06: 'PCI',
+                0x07: 'PC Card (PCMCIA)',
+                0x08: 'VL-VESA',
+                0x09: 'Proprietary',
+                0x0A: 'Processor Card Slot',
+                0x0B: 'Proprietary Memory Card Slot',
+                0x0C: 'I/O Riser Card Slot',
+                0x0D: 'NuBus',
+                0x0E: 'PCI 66MHz Capable',
+                0x0F: 'AGP',
+                0x10: 'AGP 2X',
+                0x11: 'AGP 4X',
+                0x12: 'PCI-X',
+                0x13: 'AGP 8X',
+                0xA0: 'PC-98/C20',
+                0xA1: 'PC-98/C24',
+                0xA2: 'PC-98/E',
+                0xA3: 'PC-98/Local Bus',
+                0xA4: 'PC-98/Card',
+                0xA5: 'PCI Express',
+                0xA6: 'PCI Express x1',
+                0xA7: 'PCI Express x2',
+                0xA8: 'PCI Express x4',
+                0xA9: 'PCI Express x8',
+                0xAA: 'PCI Express x16',
+                0xAB: 'PCI Express Gen 2',
+                0xAC: 'PCI Express Gen 2 x1',
+                0xAD: 'PCI Express Gen 2 x2',
+                0xAE: 'PCI Express Gen 2 x4',
+                0xAF: 'PCI Express Gen 2 x8',
+                0xB0: 'PCI Express Gen 2 x16',
+                0xB1: 'PCI Express Gen 3',
+                0xB2: 'PCI Express Gen 3 x1',
+                0xB3: 'PCI Express Gen 3 x2',
+                0xB4: 'PCI Express Gen 3 x4',
+                0xB5: 'PCI Express Gen 3 x8',
+                0xB6: 'PCI Express Gen 3 x16',
+            }
+            self.add_field('slot_type', u.unpack_one("B"), unpack.format_table("{}", _slot_types))
+            _slot_data_bus_widths = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: '8 bit',
+                0x04: '16 bit',
+                0x05: '32 bit',
+                0x06: '64 bit',
+                0x07: '128 bit',
+                0x08: '1x or x1',
+                0x09: '2x or x2',
+                0x0A: '4x or x4',
+                0x0B: '8x or x8',
+                0x0C: '12x or x12',
+                0x0D: '16x or x16',
+                0x0E: '32x or x32',
+            }
+            self.add_field('slot_data_bus_width', u.unpack_one('B'), unpack.format_table("{}", _slot_data_bus_widths))
+            _current_usages = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'Available',
+                0x04: 'In use',
+            }
+            self.add_field('current_usage', u.unpack_one('B'), unpack.format_table("{}", _current_usages))
+            _slot_lengths = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'Short Length',
+                0x04: 'Long Length',
+            }
+            self.add_field('slot_length', u.unpack_one('B'), unpack.format_table("{}", _slot_lengths))
+            self.add_field('slot_id', u.unpack_one('<H'))
+            self.add_field('characteristics1', u.unpack_one('B'))
+            self.add_field('characteristics_unknown', bool(bitfields.getbits(self.characteristics1, 0)), "characteristics1[0]={}")
+            self.add_field('provides_5_0_volts', bool(bitfields.getbits(self.characteristics1, 1)), "characteristics1[1]={}")
+            self.add_field('provides_3_3_volts', bool(bitfields.getbits(self.characteristics1, 2)), "characteristics1[2]={}")
+            self.add_field('shared_slot', bool(bitfields.getbits(self.characteristics1, 3)), "characteristics1[3]={}")
+            self.add_field('supports_pc_card_16', bool(bitfields.getbits(self.characteristics1, 4)), "characteristics1[4]={}")
+            self.add_field('supports_cardbus', bool(bitfields.getbits(self.characteristics1, 5)), "characteristics1[5]={}")
+            self.add_field('supports_zoom_video', bool(bitfields.getbits(self.characteristics1, 6)), "characteristics1[6]={}")
+            self.add_field('supports_modem_ring_resume', bool(bitfields.getbits(self.characteristics1, 7)), "characteristics1[7]={}")
+            if self.length > 0x0C:
+                self.add_field('characteristics2', u.unpack_one('B'))
+                self.add_field('supports_PME', bool(bitfields.getbits(self.characteristics2, 0)), "characteristics2[0]={}")
+                self.add_field('supports_hot_plug', bool(bitfields.getbits(self.characteristics2, 1)), "characteristics2[1]={}")
+                self.add_field('supports_smbus', bool(bitfields.getbits(self.characteristics2, 2)), "characteristics2[2]={}")
+            if self.length > 0x0D:
+                self.add_field('segment_group_number', u.unpack_one('<H'))
+                self.add_field('bus_number', u.unpack_one('B'))
+                self.add_field('device_function_number', u.unpack_one('B'))
+                self.add_field('device_number', bitfields.getbits(self.device_function_number, 7, 3), "device_function_number[7:3]={}")
+                self.add_field('function_number', bitfields.getbits(self.device_function_number, 2, 0), "device_function_number[2:0]={}")
+        except:
+            self.decodeFailure = True
+            print "Error parsing SystemSlots"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class OnBoardDevicesInformation(SmbiosBaseStructure):
+    smbios_structure_type = 10
+
+    def __init__(self, u, sm):
+        super(OnBoardDevicesInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('device_type', u.unpack_one("B"))
+            self.add_field('device_enabled', bool(bitfields.getbits(self.device_type, 7)), "device_type[7]={}")
+            _device_types = {
+                0x01: 'Other',
+                0x02: 'Unknown',
+                0x03: 'Video',
+                0x04: 'SCSI Controller',
+                0x05: 'Ethernet',
+                0x06: 'Token Ring',
+                0x07: 'Sound',
+                0x08: 'PATA Controller',
+                0x09: 'SATA Controller',
+                0x0A: 'SAS Controller'
+            }
+            self.add_field('type_of_device', bitfields.getbits(self.device_type, 6, 0), unpack.format_table("device_type[6:0]={}", _device_types))
+            self.add_field('description_string', u.unpack_one("B"), self.fmtstr)
+        except:
+            self.decodeFailure = True
+            print "Error parsing OnBoardDevicesInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class OEMStrings(SmbiosBaseStructure):
+    smbios_structure_type = 11
+
+    def __init__(self, u, sm):
+        super(OEMStrings, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('count', u.unpack_one("B"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing OEMStrings"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class SystemConfigOptions(SmbiosBaseStructure):
+    smbios_structure_type = 12
+
+    def __init__(self, u, sm):
+        super(SystemConfigOptions, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('count', u.unpack_one("B"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing SystemConfigOptions"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class BIOSLanguageInformation(SmbiosBaseStructure):
+    smbios_structure_type = 13
+
+    def __init__(self, u, sm):
+        super(BIOSLanguageInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('installable_languages', u.unpack_one("B"))
+            if self.length > 0x05:
+                self.add_field('flags', u.unpack_one('B'))
+                self.add_field('abbreviated_format', bool(bitfields.getbits(self.flags, 0)), "flags[0]={}")
+            if self.length > 0x6:
+                u.skip(15)
+                self.add_field('current_language', u.unpack_one('B'), self.fmtstr)
+        except:
+            self.decodeFailure = True
+            print "Error parsing BIOSLanguageInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class GroupAssociations(SmbiosBaseStructure):
+    smbios_structure_type = 14
+
+    def __init__(self, u, sm):
+        super(GroupAssociations, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('group_name', u.unpack_one("B"), self.fmtstr)
+            self.add_field('item_type', u.unpack_one('B'))
+            self.add_field('item_handle', u.unpack_one('<H'))
+        except:
+            self.decodeFailure = True
+            print "Error parsing GroupAssociations"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class SystemEventLog(SmbiosBaseStructure):
+    smbios_structure_type = 15
+
+    def __init__(self, u, sm):
+        super(SystemEventLog, self).__init__(u, sm)
+        u = self.u
+        try:
+            self.add_field('log_area_length', u.unpack_one("<H"))
+            self.add_field('log_header_start_offset', u.unpack_one('<H'))
+            self.add_field('log_data_start_offset', u.unpack_one('<H'))
+            _access_method = {
+                0x00: 'Indexed I/O: 1 8-bit index port, 1 8-bit data port',
+                0x01: 'Indexed I/O: 2 8-bit index ports, 1 8-bit data port',
+                0x02: 'Indexed I/O: 1 16-bit index port, 1 8-bit data port',
+                0x03: 'Memory-mapped physical 32-bit address',
+                0x04: 'Available through General-Purpose NonVolatile Data functions',
+                xrange(0x05, 0x07F): 'Available for future assignment',
+                xrange(0x80, 0xFF): 'BIOS Vendor/OEM-specific'
+                }
+            self.add_field('access_method', u.unpack_one('B'), unpack.format_table("{}", _access_method))
+            self.add_field('log_status', u.unpack_one('B'))
+            self.add_field('log_area_full', bool(bitfields.getbits(self.log_status, 1)), "log_status[1]={}")
+            self.add_field('log_area_valid', bool(bitfields.getbits(self.log_status, 0)), "log_status[0]={}")
+            self.add_field('log_change_token', u.unpack_one('<I'))
+            self.add_field('access_method_address', u.unpack_one('<I'))
+            if self.length > 0x14:
+                _log_header_formats = {
+                    0: 'No header',
+                    1: 'Type 1 log header',
+                    xrange(2, 0x7f): 'Available for future assignment',
+                    xrange(0x80, 0xff): 'BIOS vendor or OEM-specific format'
+                    }
+                self.add_field('log_header_format', u.unpack_one("B"), unpack.format_table("{}", _log_header_formats))
+            if self.length > 0x15:
+                self.add_field('num_supported_log_type_descriptors', u.unpack_one('B'))
+            if self.length > 0x16:
+                self.add_field('length_log_type_descriptor', u.unpack_one('B'))
+            if self.length != (0x17 + (self.num_supported_log_type_descriptors * self.length_log_type_descriptor)):
+                print "Error: structure length ({}) != 0x17 + (num_supported_log_type_descriptors ({}) * length_log_type_descriptor({}))".format(self.length, self.num_supported_log_type_descriptors, self.length_log_type_descriptor)
+                print "structure length = {}".format(self.length)
+                print "num_supported_log_type_descriptors = {}".format(self.num_supported_log_type_descriptors)
+                print "length_log_type_descriptor = {}".format(self.length_log_type_descriptor)
+                self.decodeFailure = True
+            self.add_field('descriptors', tuple(EventLogDescriptor.unpack(u) for i in range(self.num_supported_log_type_descriptors)), unpack.format_each("\n{!r}"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing SystemEventLog"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class EventLogDescriptor(unpack.Struct):
+    @staticmethod
+    def _unpack(u):
+        _event_log_type_descriptors = {
+            0x00: 'Reserved',
+            0x01: 'Single-bit ECC memory error',
+            0x02: 'Multi-bit ECC memory error',
+            0x03: 'Parity memory error',
+            0x04: 'Bus time-out',
+            0x05: 'I/O Channel Check',
+            0x06: 'Software NMI',
+            0x07: 'POST Memory Resize',
+            0x08: 'POST Error',
+            0x09: 'PCI Parity Error',
+            0x0A: 'PCI System Error',
+            0x0B: 'CPU Failure',
+            0x0C: 'EISA FailSafe Timer time-out',
+            0x0D: 'Correctable memory log disabled',
+            0x0E: 'Logging disabled for a specific Event Type - too many errors of the same type received in a short amount of time',
+            0x0F: 'Reserved',
+            0x10: 'System Limit Exceeded',
+            0x11: 'Asynchronous hardware timer expired and issued a system reset',
+            0x12: 'System configuration information',
+            0x13: 'Hard-disk information',
+            0x14: 'System reconfigured',
+            0x15: 'Uncorrectable CPU-complex error',
+            0x16: 'Log Area Reset/Cleared',
+            0x17: 'System boot',
+            xrange(0x18, 0x7F): 'Unused, available for assignment',
+            xrange(0x80, 0xFE): 'Availalbe for system- and OEM-specific assignments',
+            0xFF: 'End of log'
+        }
+        yield 'log_type', u.unpack_one('B'), unpack.format_table("{}", _event_log_type_descriptors)
+        _event_log_format = {
+            0x00: 'None',
+            0x01: 'Handle',
+            0x02: 'Multiple-Event',
+            0x03: 'Multiple-Event Handle',
+            0x04: 'POST Results Bitmap',
+            0x05: 'System Management Type',
+            0x06: 'Multiple-Event System Management Type',
+            xrange(0x80, 0xFF): 'OEM assigned'
+        }
+        yield 'variable_data_format_type', u.unpack_one('B'), unpack.format_table("{}", _event_log_format)
+
+class PhysicalMemoryArray(SmbiosBaseStructure):
+    smbios_structure_type = 16
+
+    def __init__(self, u, sm):
+        super(PhysicalMemoryArray, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                _location_field = {
+                    0x01: "Other",
+                    0x02: "Unknown",
+                    0x03: "System board or motherboard",
+                    0x04: "ISA add-on card",
+                    0x05: "EISA add-on card",
+                    0x06: "PCI add-on card",
+                    0x07: "MCA add-on card",
+                    0x08: "PCMCIA add-on card",
+                    0x09: "Proprietary add-on card",
+                    0x0A: "NuBus",
+                    0xA0: "PC-98/C20 add-on card",
+                    0xA1: "PC-98/C24 add-on card",
+                    0xA2: "PC-98/E add-on card",
+                    0xA3: "PC-98/Local bus add-on card"
+                    }
+                self.add_field('location', u.unpack_one("B"), unpack.format_table("{}", _location_field))
+            if self.length > 0x05:
+                _use = {
+                    0x01: "Other",
+                    0x02: "Unknown",
+                    0x03: "System memory",
+                    0x04: "Video memory",
+                    0x05: "Flash memory",
+                    0x06: "Non-volatile RAM",
+                    0x07: "Cache memory"
+                    }
+                self.add_field('use', u.unpack_one('B'), unpack.format_table("{}", _use))
+            if self.length > 0x06:
+                _error_correction = {
+                    0x01: "Other",
+                    0x02: "Unknown",
+                    0x03: "None",
+                    0x04: "Parity",
+                    0x05: "Single-bit ECC",
+                    0x06: "Multi-bit ECC",
+                    0x07: "CRC"
+                    }
+                self.add_field('memory_error_correction', u.unpack_one('B'), unpack.format_table("{}", _error_correction))
+            if self.length > 0x07:
+                self.add_field('maximum_capacity', u.unpack_one('<I'))
+            if self.length > 0x0B:
+                self.add_field('memory_error_information_handle', u.unpack_one('<H'))
+            if self.length > 0x0D:
+                self.add_field('num_memory_devices', u.unpack_one('<H'))
+            if self.length > 0x0F:
+                self.add_field('extended_maximum_capacity', u.unpack_one('<Q'))
+        except:
+            self.decodeFailure = True
+            print "Error parsing PhysicalMemoryArray"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class MemoryDevice(SmbiosBaseStructure):
+    smbios_structure_type = 17
+
+    def __init__(self, u, sm):
+        super(MemoryDevice, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('physical_memory_array_handle', u.unpack_one("<H"))
+            if self.length > 0x6:
+                self.add_field('memory_error_information_handle', u.unpack_one("<H"))
+            if self.length > 0x8:
+                self.add_field('total_width', u.unpack_one("<H"))
+            if self.length > 0xA:
+                self.add_field('data_width', u.unpack_one("<H"))
+            if self.length > 0xC:
+                self.add_field('size', u.unpack_one("<H"))
+            if self.length > 0xE:
+                _form_factors = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'SIMM',
+                    0x04: 'SIP',
+                    0x05: 'Chip',
+                    0x06: 'DIP',
+                    0x07: 'ZIP',
+                    0x08: 'Proprietary Card',
+                    0x09: 'DIMM',
+                    0x0A: 'TSOP',
+                    0x0B: 'Row of chips',
+                    0x0C: 'RIMM',
+                    0x0D: 'SODIMM',
+                    0x0E: 'SRIMM',
+                    0x0F: 'FB-DIMM'
+                    }
+                self.add_field('form_factor', u.unpack_one("B"), unpack.format_table("{}", _form_factors))
+            if self.length > 0xF:
+                self.add_field('device_set', u.unpack_one("B"))
+            if self.length > 0x10:
+                self.add_field('device_locator', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x11:
+                self.add_field('bank_locator', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x12:
+                _memory_types = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'DRAM',
+                    0x04: 'EDRAM',
+                    0x05: 'VRAM',
+                    0x06: 'SRAM',
+                    0x07: 'RAM',
+                    0x08: 'ROM',
+                    0x09: 'FLASH',
+                    0x0A: 'EEPROM',
+                    0x0B: 'FEPROM',
+                    0x0C: 'EPROM',
+                    0x0D: 'CDRAM',
+                    0x0E: '3DRAM',
+                    0x0F: 'SDRAM',
+                    0x10: 'SGRAM',
+                    0x11: 'RDRAM',
+                    0x12: 'DDR',
+                    0x13: 'DDR2',
+                    0x14: 'DDR2 FB-DIMM',
+                    xrange(0x15, 0x17): 'Reserved',
+                    0x18: 'DDR3',
+                    0x19: 'FBD2'
+                    }
+                self.add_field('memory_type', u.unpack_one("B"), unpack.format_table("{}", _memory_types))
+            if self.length > 0x13:
+                self.add_field('type_detail', u.unpack_one('<H'))
+            if self.length > 0x15:
+                self.add_field('speed', u.unpack_one("<H"))
+            if self.length > 0x17:
+                self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x18:
+                self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x19:
+                self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x1A:
+                self.add_field('part_number', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x1B:
+                self.add_field('attributes', u.unpack_one("B"))
+                self.add_field('rank', bitfields.getbits(self.attributes, 3, 0), "attributes[3:0]={}")
+            if self.length > 0x1C:
+                if self.size == 0x7FFF:
+                    self.add_field('extended_size', u.unpack_one('<I'))
+                    self.add_field('mem_size', bitfields.getbits(self.type_detail, 30, 0), "type_detail[30:0]={}")
+                else:
+                    u.skip(4)
+            if self.length > 0x20:
+                self.add_field('configured_memory_clock_speed', u.unpack_one("<H"))
+            if self.length > 0x22:
+                self.add_field('minimum_voltage', u.unpack_one("<H"))
+            if self.length > 0x24:
+                self.add_field('maximum_voltage', u.unpack_one("<H"))
+            if self.length > 0x26:
+                self.add_field('configured_voltage', u.unpack_one("<H"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing MemoryDevice"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class MemoryErrorInfo32Bit(SmbiosBaseStructure):
+    smbios_structure_type = 18
+
+    def __init__(self, u, sm):
+        super(MemoryErrorInfo32Bit, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                _error_types = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'OK',
+                    0x04: 'Bad read',
+                    0x05: 'Parity error',
+                    0x06: 'Single-bit error',
+                    0x07: 'Double-bit error',
+                    0x08: 'Multi-bit error',
+                    0x09: 'Nibble error',
+                    0x0A: 'Checksum error',
+                    0x0B: 'CRC error',
+                    0x0C: 'Corrected single-bit error',
+                    0x0D: 'Corrected error',
+                    0x0E: 'Uncorrectable error'
+                    }
+                self.add_field('error_type', u.unpack_one("B"), unpack.format_table("{}", _error_types))
+            if self.length > 0x5:
+                 _error_granularity_field = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Device level',
+                    0x04: 'Memory partition level'
+                    }
+                 self.add_field('error_granularity', u.unpack_one("B"), unpack.format_table("{}", _error_granularity_field))
+            if self.length > 0x6:
+                _error_operation_field = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Read',
+                    0x04: 'Write',
+                    0x05: 'Partial write'
+                    }
+                self.add_field('error_operation', u.unpack_one("B"), unpack.format_table("{}", _error_operation_field))
+            if self.length > 0x7:
+                self.add_field('vendor_syndrome', u.unpack_one("<I"))
+            if self.length > 0xB:
+                self.add_field('memory_array_error_address', u.unpack_one("<I"))
+            if self.length > 0xF:
+                self.add_field('device_error_address', u.unpack_one("<I"))
+            if self.length > 0x13:
+                self.add_field('error_resolution', u.unpack_one("<I"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing MemoryErrorInfo32Bit"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class MemoryArrayMappedAddress(SmbiosBaseStructure):
+    smbios_structure_type = 19
+
+    def __init__(self, u, sm):
+        super(MemoryArrayMappedAddress, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('starting_address', u.unpack_one("<I"))
+                # if FFFF FFFF: address stored in Extended Starting Address
+            if self.length > 0x8:
+                self.add_field('ending_address', u.unpack_one("<I"))
+            if self.length > 0xC:
+                self.add_field('memory_array_handle', u.unpack_one("<H"))
+            if self.length > 0xE:
+                self.add_field('partition_width', u.unpack_one("B"))
+            if self.length > 0xF:
+                # valid if starting_address = FFFF FFFF
+                if self.starting_address == 0xFFFFFFFF:
+                    self.add_field('extended_starting_address', u.unpack_one("<Q"))
+                    if self.length > 0x17:
+                        self.add_field('extended_ending_address', u.unpack_one("<Q"))
+                else:
+                    u.skip(16)
+
+        except:
+            self.decodeFailure = True
+            print "Error parsing MemoryArrayMappedAddress"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class MemoryDeviceMappedAddress(SmbiosBaseStructure):
+    smbios_structure_type = 20
+
+    def __init__(self, u, sm):
+        super(MemoryDeviceMappedAddress, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('starting_address', u.unpack_one("<I"))
+                # if FFFF FFFF: address stored in Extended Starting Address
+            if self.length > 0x8:
+                self.add_field('ending_address', u.unpack_one("<I"))
+            if self.length > 0xC:
+                self.add_field('memory_device_handle', u.unpack_one("<H"))
+            if self.length > 0xE:
+                self.add_field('memory_array_mapped_address_handle', u.unpack_one("<H"))
+            if self.length > 0x10:
+                self.add_field('partition_row_position', u.unpack_one("B"))
+            if self.length > 0x11:
+                self.add_field('interleave_position', u.unpack_one("B"))
+            if self.length > 0x12:
+                self.add_field('interleave_data_depth', u.unpack_one("B"))
+            if self.length > 0x13:
+                # valid if starting_address = FFFF FFFF
+                if self.starting_address == 0xFFFFFFFF:
+                    self.add_field('extended_starting_address', u.unpack_one("<Q"))
+                    if self.length > 0x1B:
+                        self.add_field('extended_ending_address', u.unpack_one("<Q"))
+                else:
+                    u.skip(16)
+        except:
+            self.decodeFailure = True
+            print "Error parsing MemoryDeviceMappedAddress"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class BuiltInPointingDevice(SmbiosBaseStructure):
+    smbios_structure_type = 21
+
+    def __init__(self, u, sm):
+        super(BuiltInPointingDevice, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                _pointing_device_types = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Mouse',
+                    0x04: 'Track Ball',
+                    0x05: 'Track Point',
+                    0x06: 'Glide Point',
+                    0x07: 'Touch Pad',
+                    0x08: 'Touch Screen',
+                    0x09: 'Optical Sensor'
+                    }
+                self.add_field('pointing_device_type', u.unpack_one("B"), unpack.format_table("{}", _pointing_device_types))
+            if self.length > 0x5:
+                _interfaces = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Serial',
+                    0x04: 'PS/2',
+                    0x05: 'Infared',
+                    0x06: 'HP-HIL',
+                    0x07: 'Bus mouse',
+                    0x08: 'ADB (Apple Desktop Bus)',
+                    0x09: 'Bus mouse DB-9',
+                    0x0A: 'Bus mouse micro-DIN',
+                    0x0B: 'USB'
+                    }
+                self.add_field('interface', u.unpack_one("B"), unpack.format_table("{}", _interfaces))
+            if self.length > 0x6:
+                self.add_field('num_buttons', u.unpack_one("B"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing BuiltInPointingDevice"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class PortableBattery(SmbiosBaseStructure):
+    smbios_structure_type = 22
+
+    def __init__(self, u, sm):
+        super(PortableBattery, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('location', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x5:
+                self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x6:
+                self.add_field('manufacturer_date', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x7:
+                self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x8:
+                self.add_field('device_name', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x9:
+                _device_chemistry = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Lead Acid',
+                    0x04: 'Nickel Cadmium',
+                    0x05: 'Nickel metal hydride',
+                    0x06: 'Lithium-ion',
+                    0x07: 'Zinc air',
+                    0x08: 'Lithium Polymer'
+                    }
+                self.add_field('device_chemistry', u.unpack_one("B"), unpack.format_table("{}", _device_chemistry))
+            if self.length > 0xA:
+                self.add_field('design_capacity', u.unpack_one("<H"))
+            if self.length > 0xC:
+                self.add_field('design_voltage', u.unpack_one("<H"))
+            if self.length > 0xE:
+                self.add_field('sbds_version_number', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0xF:
+                self.add_field('max_error_battery_data', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x10:
+                if self.serial_number == 0:
+                    self.add_field('sbds_serial_number', u.unpack_one("<H"))
+                else:
+                    u.skip(2)
+            if self.length > 0x12:
+                if self.manufacturer_date == 0:
+                    self.add_field('sbds_manufacture_date', u.unpack_one("<H"))
+                    self.add_field('year_biased_by_1980', bitfields.getbits(self.sbds_manufacture_date, 15, 9), "sbds_manufacture_date[15:9]={}")
+                    self.add_field('month', bitfields.getbits(self.sbds_manufacture_date, 8, 5), "sbds_manufacture_date[8:5]={}")
+                    self.add_field('date', bitfields.getbits(self.sbds_manufacture_date, 4, 0), "sbds_manufacture_date[4:0]={}")
+                else:
+                    u.skip(2)
+            if self.length > 0x14:
+                if self.device_chemistry == 0x02:
+                    self.add_field('sbds_device_chemistry', u.unpack_one("B"), self.fmtstr)
+                else:
+                    u.skip(1)
+            if self.length > 0x15:
+                self.add_field('design_capacity_multiplier', u.unpack_one("B"))
+            if self.length > 0x16:
+                self.add_field('oem_specific', u.unpack_one("<I"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing PortableBattery"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class SystemReset(SmbiosBaseStructure):
+    smbios_structure_type = 23
+
+    def __init__(self, u, sm):
+        super(SystemReset, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('capabilities', u.unpack_one("B"))
+                self.add_field('contains_watchdog_timer', bool(bitfields.getbits(self.capabilities, 5)), "capabilities[5]={}")
+                _boot_option = {
+                    0b00: 'Reserved, do not use',
+                    0b01: 'Operating System',
+                    0b10: 'System utilities',
+                    0b11: 'Do not reboot'
+                    }
+                self.add_field('boot_option_on_limit', bitfields.getbits(self.capabilities, 4, 3), unpack.format_table("capabilities[4:3]={}", _boot_option))
+                self.add_field('boot_option_after_watchdog_reset', bitfields.getbits(self.capabilities, 2, 1), unpack.format_table("capabilities[2:1]={}", _boot_option))
+                self.add_field('system_reset_enabled_by_user', bool(bitfields.getbits(self.capabilities, 0)), "capabilities[0]={}")
+            if self.length > 0x5:
+                self.add_field('reset_count', u.unpack_one("<H"))
+            if self.length > 0x5:
+                self.add_field('reset_limit', u.unpack_one("<H"))
+            if self.length > 0x9:
+                self.add_field('timer_interval', u.unpack_one("<H"))
+            if self.length > 0xB:
+                self.add_field('timeout', u.unpack_one("<H"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing SystemReset"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class HardwareSecurity(SmbiosBaseStructure):
+    smbios_structure_type = 24
+
+    def __init__(self, u, sm):
+        super(HardwareSecurity, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('hardware_security_settings', u.unpack_one("B"))
+                _status = {
+                    0x00: 'Disabled',
+                    0x01: 'Enabled',
+                    0x02: 'Not Implemented',
+                    0x03: 'Unknown'
+                    }
+                self.add_field('power_on_password_status', bitfields.getbits(self.hardware_security_settings, 7, 6), unpack.format_table("hardware_security_settings[7:6]={}", _status))
+                self.add_field('keyboard_password_status', bitfields.getbits(self.hardware_security_settings, 5, 4), unpack.format_table("hardware_security_settings[5:4]={}", _status))
+                self.add_field('admin_password_status', bitfields.getbits(self.hardware_security_settings, 3, 2), unpack.format_table("hardware_security_settings0[3:2]={}", _status))
+                self.add_field('front_panel_reset_status', bitfields.getbits(self.hardware_security_settings, 1, 0), unpack.format_table("hardware_security_settings[1:0]={}", _status))
+        except:
+            self.decodeFailure = True
+            print "Error parsing HardwareSecurity"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class SystemPowerControls(SmbiosBaseStructure):
+    smbios_structure_type = 25
+
+    def __init__(self, u, sm):
+        super(SystemPowerControls, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('next_scheduled_poweron_month', u.unpack_one("B"))
+                self.add_field('next_scheduled_poweron_day_of_month', u.unpack_one("B"))
+                self.add_field('next_scheduled_poweron_hour', u.unpack_one("B"))
+                self.add_field('next_scheduled_poweron_minute', u.unpack_one("B"))
+                self.add_field('next_scheduled_poweron_second', u.unpack_one("B"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing SystemPowerControls"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class VoltageProbe(SmbiosBaseStructure):
+    smbios_structure_type = 26
+
+    def __init__(self, u, sm):
+        super(VoltageProbe, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('description', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x5:
+                self.add_field('location_and_status', u.unpack_one("B"))
+                _status = {
+                    0b001: 'Other',
+                    0b010: 'Unknown',
+                    0b011: 'OK',
+                    0b100: 'Non-critical',
+                    0b101: 'Critical',
+                    0b110: 'Non-recoverable'
+                    }
+                _location = {
+                    0b00001: 'Other',
+                    0b00010: 'Unknown',
+                    0b00011: 'Processor',
+                    0b00100: 'Disk',
+                    0b00101: 'Peripheral Bay',
+                    0b00110: 'System Management Module',
+                    0b00111: 'Motherboard',
+                    0b01000: 'Memory Module',
+                    0b01001: 'Processor Module',
+                    0b01010: 'Power Unit',
+                    0b01011: 'Add-in Card'
+                    }
+                self.add_field('status', bitfields.getbits(self.location_and_status, 7, 5), unpack.format_table("location_and_status[7:5]={}", _status))
+                self.add_field('location', bitfields.getbits(self.location_and_status, 4, 0), unpack.format_table("location_and_status[4:0]={}", _location))
+            if self.length > 0x6:
+                self.add_field('max_value', u.unpack_one("<H"))
+            if self.length > 0x8:
+                self.add_field('min_value', u.unpack_one("<H"))
+            if self.length > 0xA:
+                self.add_field('resolution', u.unpack_one("<H"))
+            if self.length > 0xC:
+                self.add_field('tolerance', u.unpack_one("<H"))
+            if self.length > 0xE:
+                self.add_field('accuracy', u.unpack_one("<H"))
+            if self.length > 0x10:
+                self.add_field('oem_defined', u.unpack_one("<I"))
+            if self.length > 0x14:
+                self.add_field('nominal_value', u.unpack_one("<H"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing VoltageProbe"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class CoolingDevice(SmbiosBaseStructure):
+    smbios_structure_type = 27
+
+    def __init__(self, u, sm):
+        super(CoolingDevice, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('temperature_probe_handle', u.unpack_one("<H"))
+            if self.length > 0x6:
+                self.add_field('device_type_and_status', u.unpack_one("B"))
+                _status = {
+                    0b001: 'Other',
+                    0b010: 'Unknown',
+                    0b011: 'OK',
+                    0b100: 'Non-critical',
+                    0b101: 'Critical',
+                    0b110: 'Non-recoverable'
+                    }
+                _type = {
+                    0b00001: 'Other',
+                    0b00010: 'Unknown',
+                    0b00011: 'Fan',
+                    0b00100: 'Centrifugal Blower',
+                    0b00101: 'Chip Fan',
+                    0b00110: 'Cabinet Fan',
+                    0b00111: 'Power Supply Fan',
+                    0b01000: 'Heat Pipe',
+                    0b01001: 'Integrated Refrigeration',
+                    0b10000: 'Active Cooling',
+                    0b10001: 'Passive Cooling'
+                    }
+                self.add_field('status', bitfields.getbits(self.device_type_and_status, 7, 5), unpack.format_table("device_type_and_status[7:5]={}", _status))
+                self.add_field('device_type', bitfields.getbits(self.device_type_and_status, 4, 0), unpack.format_table("device_type_and_status[4:0]={}", _type))
+            if self.length > 0x7:
+                self.add_field('cooling_unit_group', u.unpack_one("B"))
+            if self.length > 0x8:
+                self.add_field('OEM_defined', u.unpack_one("<I"))
+            if self.length > 0xC:
+                self.add_field('nominal_speed', u.unpack_one("<H"))
+            if self.length > 0xE:
+               self.add_field('description', u.unpack_one("B"), self.fmtstr)
+        except:
+            self.decodeFailure = True
+            print "Error parsing CoolingDevice"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class TemperatureProbe(SmbiosBaseStructure):
+    smbios_structure_type = 28
+
+    def __init__(self, u, sm):
+        super(TemperatureProbe, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('description', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x5:
+                self.add_field('location_and_status', u.unpack_one("B"))
+                _status = {
+                    0b001: 'Other',
+                    0b010: 'Unknown',
+                    0b011: 'OK',
+                    0b100: 'Non-critical',
+                    0b101: 'Critical',
+                    0b110: 'Non-recoverable'
+                    }
+                _location = {
+                    0b00001: 'Other',
+                    0b00010: 'Unknown',
+                    0b00011: 'Processor',
+                    0b00100: 'Disk',
+                    0b00101: 'Peripheral Bay',
+                    0b00110: 'System Management Module',
+                    0b00111: 'Motherboard',
+                    0b01000: 'Memory Module',
+                    0b01001: 'Processor Module',
+                    0b01010: 'Power Unit',
+                    0b01011: 'Add-in Card',
+                    0b01100: 'Front Panel Board',
+                    0b01101: 'Back Panel Board',
+                    0b01110: 'Power System Board',
+                    0b01111: 'Drive Back Plane'
+                    }
+                self.add_field('status', bitfields.getbits(self.location_and_status, 7, 5), unpack.format_table("location_and_status[7:5]={}", _status))
+                self.add_field('location', bitfields.getbits(self.location_and_status, 4, 0), unpack.format_table("location_and_status[4:0]={}", _location))
+            if self.length > 0x6:
+                self.add_field('maximum_value', u.unpack_one("<H"))
+            if self.length > 0x8:
+                self.add_field('minimum_value', u.unpack_one("<H"))
+            if self.length > 0xA:
+                self.add_field('resolution', u.unpack_one("<H"))
+            if self.length > 0xC:
+                self.add_field('tolerance', u.unpack_one("<H"))
+            if self.length > 0xE:
+                self.add_field('accuracy', u.unpack_one("<H"))
+            if self.length > 0x10:
+                self.add_field('OEM_defined', u.unpack_one("<I"))
+            if self.length > 0x14:
+                self.add_field('nominal_value', u.unpack_one("<H"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing TemperatureProbe"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class ElectricalCurrentProbe(SmbiosBaseStructure):
+    smbios_structure_type = 29
+
+    def __init__(self, u, sm):
+        super(ElectricalCurrentProbe, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('description', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x5:
+                self.add_field('location_and_status', u.unpack_one("B"))
+                _status = {
+                    0b001: 'Other',
+                    0b010: 'Unknown',
+                    0b011: 'OK',
+                    0b100: 'Non-critical',
+                    0b101: 'Critical',
+                    0b110: 'Non-recoverable'
+                    }
+                _location = {
+                    0b00001: 'Other',
+                    0b00010: 'Unknown',
+                    0b00011: 'Processor',
+                    0b00100: 'Disk',
+                    0b00101: 'Peripheral Bay',
+                    0b00110: 'System Management Module',
+                    0b00111: 'Motherboard',
+                    0b01000: 'Memory Module',
+                    0b01001: 'Processor Module',
+                    0b01010: 'Power Unit',
+                    0b01011: 'Add-in Card',
+                    0b01100: 'Front Panel Board',
+                    0b01101: 'Back Panel Board',
+                    0b01110: 'Power System Board',
+                    0b01111: 'Drive Back Plane'
+                    }
+                self.add_field('status', bitfields.getbits(self.location_and_status, 7, 5), unpack.format_table("location_and_status[7:5]={}", _status))
+                self.add_field('location', bitfields.getbits(self.location_and_status, 4, 0), unpack.format_table("location_and_status[4:0]={}", _location))
+            if self.length > 0x6:
+                self.add_field('maximum_value', u.unpack_one("<H"))
+            if self.length > 0x8:
+                self.add_field('minimum_value', u.unpack_one("<H"))
+            if self.length > 0xA:
+                self.add_field('resolution', u.unpack_one("<H"))
+            if self.length > 0xC:
+                self.add_field('tolerance', u.unpack_one("<H"))
+            if self.length > 0xE:
+                self.add_field('accuracy', u.unpack_one("<H"))
+            if self.length > 0x10:
+                self.add_field('OEM_defined', u.unpack_one("<I"))
+            if self.length > 0x14:
+                self.add_field('nominal_value', u.unpack_one("<H"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing ElectricalCurrentProbe"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class OutOfBandRemoteAccess(SmbiosBaseStructure):
+    smbios_structure_type = 30
+
+    def __init__(self, u, sm):
+        super(OutOfBandRemoteAccess, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('manufacturer_name', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x5:
+                self.add_field('connections', u.unpack_one("B"))
+                self.add_field('outbound_connection_enabled', bool(bitfields.getbits(self.connections, 1)), "connections[1]={}")
+                self.add_field('inbound_connection_enabled', bool(bitfields.getbits(self.connections, 0)), "connections[0]={}")
+        except:
+            self.decodeFailure = True
+            print "Error parsing OutOfBandRemoteAccess"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class BootIntegrityServicesEntryPoint(SmbiosBaseStructure):
+    smbios_structure_type = 31
+
+class SystemBootInformation(SmbiosBaseStructure):
+    smbios_structure_type = 32
+
+    def __init__(self, u, sm):
+        super(SystemBootInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0xA:
+                u.skip(6)
+                _boot_status = {
+                    0: 'No errors detected',
+                    1: 'No bootable media',
+                    2: '"normal" operating system failed to load',
+                    3: 'Firmware-detected hardware failure, including "unknown" failure types',
+                    4: 'Operating system-detected hardware failure',
+                    5: 'User-requested boot, usually through a keystroke',
+                    6: 'System security violation',
+                    7: 'Previously-requested image',
+                    8: 'System watchdog timer expired, causing the system to reboot',
+                    xrange(9,127): 'Reserved for future assignment',
+                    xrange(128, 191): 'Vendor/OEM-specific implementations',
+                    xrange(192, 255): 'Product-specific implementations'
+                    }
+                self.add_field('boot_status', u.unpack_one("B"), unpack.format_table("{}", _boot_status))
+        except:
+            self.decodeFailure = True
+            print "Error parsing SystemBootInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class MemoryErrorInfo64Bit(SmbiosBaseStructure):
+    smbios_structure_type = 33
+
+    def __init__(self, u, sm):
+        super(MemoryErrorInfo64Bit, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                _error_types = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'OK',
+                    0x04: 'Bad read',
+                    0x05: 'Parity error',
+                    0x06: 'Single-bit error',
+                    0x07: 'Double-bit error',
+                    0x08: 'Multi-bit error',
+                    0x09: 'Nibble error',
+                    0x0A: 'Checksum error',
+                    0x0B: 'CRC error',
+                    0x0C: 'Corrected single-bit error',
+                    0x0D: 'Corrected error',
+                    0x0E: 'Uncorrectable error'
+                    }
+                self.add_field('error_type', u.unpack_one("B"), unpack.format_table("{}", _error_types))
+            if self.length > 0x5:
+                 _error_granularity_field = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Device level',
+                    0x04: 'Memory partition level'
+                    }
+                 self.add_field('error_granularity', u.unpack_one("B"), unpack.format_table("{}", _error_granularity_field))
+            if self.length > 0x6:
+                _error_operation_field = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Read',
+                    0x04: 'Write',
+                    0x05: 'Partial write'
+                    }
+                self.add_field('error_operation', u.unpack_one("B"), unpack.format_table("{}", _error_operation_field))
+            if self.length > 0x7:
+                self.add_field('vendor_syndrome', u.unpack_one("<I"))
+            if self.length > 0xB:
+                self.add_field('memory_array_error_address', u.unpack_one("<Q"))
+            if self.length > 0xF:
+                self.add_field('device_error_address', u.unpack_one("<Q"))
+            if self.length > 0x13:
+                self.add_field('error_resolution', u.unpack_one("<Q"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing MemoryErrorInfo64Bit"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class ManagementDevice(SmbiosBaseStructure):
+    smbios_structure_type = 34
+
+    def __init__(self, u, sm):
+        super(ManagementDevice, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('description', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x5:
+                _type = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'National Semiconductor LM75',
+                    0x04: 'National Semiconductor LM78',
+                    0x05: 'National Semiconductor LM79',
+                    0x06: 'National Semiconductor LM80',
+                    0x07: 'National Semiconductor LM81',
+                    0x08: 'Analog Devices ADM9240',
+                    0x09: 'Dallas Semiconductor DS1780',
+                    0x0A: 'Maxim 1617',
+                    0x0B: 'Genesys GL518SM',
+                    0x0C: 'Winbond W83781D',
+                    0x0D: 'Holtek HT82H791'
+                    }
+                self.add_field('device_type', u.unpack_one("B"), unpack.format_table("{}", _type))
+            if self.length > 0x6:
+                self.add_field('address', u.unpack_one("<I"))
+            if self.length > 0xA:
+                 _address_type = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'I/O Port',
+                    0x04: 'Memory',
+                    0x05: 'SM Bus'
+                    }
+                 self.add_field('address_type', u.unpack_one("B"), unpack.format_table("{}", _address_type))
+        except:
+            self.decodeFailure = True
+            print "Error parsing ManagementDevice"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class ManagementDeviceComponent(SmbiosBaseStructure):
+    smbios_structure_type = 35
+
+    def __init__(self, u, sm):
+        super(ManagementDeviceComponent, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('description', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x5:
+                self.add_field('management_device_handle', u.unpack_one("<H"))
+            if self.length > 0x7:
+                self.add_field('component_handle', u.unpack_one("<H"))
+            if self.length > 0x9:
+                self.add_field('threshold_handle', u.unpack_one("<H"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing ManagementDeviceComponent"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class ManagementDeviceThresholdData(SmbiosBaseStructure):
+    smbios_structure_type = 36
+
+    def __init__(self, u, sm):
+        super(ManagementDeviceThresholdData, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('lower_threshold_noncritical', u.unpack_one("<H"))
+            if self.length > 0x6:
+                self.add_field('upper_threshold_noncritical', u.unpack_one("<H"))
+            if self.length > 0x8:
+                self.add_field('lower_threshold_critical', u.unpack_one("<H"))
+            if self.length > 0xA:
+                self.add_field('upper_threshold_critical', u.unpack_one("<H"))
+            if self.length > 0xC:
+                self.add_field('lower_threshold_nonrecoverable', u.unpack_one("<H"))
+            if self.length > 0xE:
+                self.add_field('upper_threshold_nonrecoverable', u.unpack_one("<H"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing ManagementDeviceThresholdData"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class MemoryChannel(SmbiosBaseStructure):
+    smbios_structure_type = 37
+
+    def __init__(self, u, sm):
+        super(MemoryChannel, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                _channel_type = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'RamBus',
+                    0x04: 'SyncLink'
+                    }
+                self.add_field('channel_type', u.unpack_one("B"), unpack.format_table("{}", _channel_type))
+            if self.length > 0x6:
+                self.add_field('max_channel_load', u.unpack_one("B"))
+            if self.length > 0x8:
+                self.add_field('memory_device_count', u.unpack_one("B"))
+            if self.length > 0xA:
+                self.add_field('memory_device_load', u.unpack_one("B"))
+            if self.length > 0xC:
+                self.add_field('memory_device_handle', u.unpack_one("<H"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing MemoryChannel"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class IPMIDeviceInformation(SmbiosBaseStructure):
+    smbios_structure_type = 38
+
+    def __init__(self, u, sm):
+        super(IPMIDeviceInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            _interface_type = {
+                0x00: 'Unknown',
+                0x01: 'KCS: Keyboard Controller Style',
+                0x02: 'SMIC: Server Management Interface Chip',
+                0x03: 'BT: Block Transfer',
+                xrange(0x04, 0xFF): 'Reserved'
+                }
+            self.add_field('interface_type', u.unpack_one("B"), unpack.format_table("{}", _interface_type))
+            self.add_field('ipmi_specification_revision', u.unpack_one("B"))
+            self.add_field('msd_revision', bitfields.getbits(self.ipmi_specification_revision, 7, 4), "ipmi_specification_revision[7:4]={}")
+            self.add_field('lsd_revision', bitfields.getbits(self.ipmi_specification_revision, 3, 0), "ipmi_specification_revision[3:0]={}")
+
+            self.add_field('i2c_slave_address', u.unpack_one("B"))
+            self.add_field('nv_storage_device_address', u.unpack_one("B"))
+            self.add_field('base_address', u.unpack_one("<Q"))
+            # if lsb is 1, address is in IO space. otherwise, memory-mapped
+            self.add_field('base_address_modifier_interrupt_info', u.unpack_one("B"))
+            _reg_spacing = {
+                0b00: 'Interface registers are on successive byte boundaries',
+                0b01: 'Interface registers are on 32-bit boundaries',
+                0b10: 'Interface registers are on 16-byte boundaries',
+                0b11: 'Reserved'
+                }
+            self.add_field('register_spacing', bitfields.getbits(self.base_address_modifier_interrupt_info, 7, 6), unpack.format_table("base_address_modifier_interrupt_info[7:6]={}", _reg_spacing))
+            self.add_field('ls_bit_for_addresses', bitfields.getbits(self.base_address_modifier_interrupt_info, 4), "base_address_modifier_interrupt_info[4]={}")
+            self.add_field('interrupt_info_specified', bool(bitfields.getbits(self.base_address_modifier_interrupt_info, 3)), "base_address_modifier_interrupt_info[3]={}")
+            _polarity = {
+                0: 'active low',
+                1: 'active high'
+                }
+            self.add_field('interrupt_polarity', bitfields.getbits(self.base_address_modifier_interrupt_info, 1), unpack.format_table("base_address_modifier_interrupt_info[1]={}", _polarity))
+            _interrupt_trigger = {
+                0: 'edge',
+                1: 'level'
+                }
+            self.add_field('interrupt_trigger_mode', bitfields.getbits(self.base_address_modifier_interrupt_info, 0), unpack.format_table("base_address_modifier_interrupt_info[0]={}", _interrupt_trigger))
+            self.add_field('interrupt_number', u.unpack_one("B"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing IPMIDeviceInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class SystemPowerSupply(SmbiosBaseStructure):
+    smbios_structure_type = 39
+
+    def __init__(self, u, sm):
+        super(SystemPowerSupply, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('power_unit_group', u.unpack_one("B"))
+            if self.length > 0x5:
+                self.add_field('location', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x6:
+                self.add_field('device_name', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x7:
+                self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x8:
+                self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x9:
+                self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0xA:
+                self.add_field('model_part_number', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0xB:
+                self.add_field('revision_level', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0xC:
+                self.add_field('max_power_capacity', u.unpack_one("<H"))
+            if self.length > 0xE:
+                self.add_field('power_supply_characteristics', u.unpack_one("<H"))
+                _dmtf_power_supply_type = {
+                    0b001: 'Other',
+                    0b010: 'Unknown',
+                    0b011: 'Linear',
+                    0b100: 'Switching',
+                    0b101: 'Battery',
+                    0b110: 'UPS',
+                    0b111: 'Converter',
+                    0b1000: 'Regulator',
+                    xrange(0b1001, 0b1111): 'Reserved'
+                    }
+                self.add_field('dmtf_power_supply_type', bitfields.getbits(self.power_supply_characteristics, 13, 10), unpack.format_table("power_supply_characteristics[13:10]={}", _dmtf_power_supply_type))
+                _status = {
+                    0b001: 'Other',
+                    0b010: 'Unknown',
+                    0b011: 'OK',
+                    0b100: 'Non-critical',
+                    0b101: 'Critical; power supply has failed and has been taken off-line'
+                    }
+                self.add_field('status', bitfields.getbits(self.power_supply_characteristics, 9, 7), unpack.format_table("power_supply_characteristics[9:7]={}", _status))
+                _dmtf_input_voltage_range_switching = {
+                    0b001: 'Other',
+                    0b010: 'Unknown',
+                    0b011: 'Manual',
+                    0b100: 'Auto-switch',
+                    0b101: 'Wide range',
+                    0b110: 'Not applicable',
+                    xrange(0b0111, 0b1111): 'Reserved'
+                    }
+                self.add_field('dmtf_input_voltage_range_switching', bitfields.getbits(self.power_supply_characteristics, 6, 3), unpack.format_table("power_supply_characteristics[6:3]={}", _dmtf_input_voltage_range_switching))
+                self.add_field('power_supply_unplugged', bool(bitfields.getbits(self.power_supply_characteristics, 2)), "power_supply_characteristics[2]={}")
+                self.add_field('power_supply_present', bool(bitfields.getbits(self.power_supply_characteristics, 1)), "power_supply_characteristics[1]={}")
+                self.add_field('power_supply_hot_replaceable', bool(bitfields.getbits(self.power_supply_characteristics, 0)), "power_supply_characteristics[0]={}")
+            if self.length > 0x10:
+                self.add_field('input_voltage_probe_handle', u.unpack_one("<H"))
+            if self.length > 0x12:
+                self.add_field('cooling_device_handle', u.unpack_one("<H"))
+            if self.length > 0x14:
+                self.add_field('input_current_probe_handle', u.unpack_one("<H"))
+        except:
+            self.decodeFailure = True
+            print "Error parsing SystemPowerSupply"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class AdditionalInformation(SmbiosBaseStructure):
+    smbios_structure_type = 40
+
+    def __init__(self, u, sm):
+        super(AdditionalInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('num_additional_information_entries', u.unpack_one("B"))
+            if self.length > 0x5:
+                self.add_field('additional_information_entry_length', u.unpack_one("B"))
+                self.add_field('referenced_handle', u.unpack_one("<H"))
+                self.add_field('referenced_offset', u.unpack_one("B"))
+                self.add_field('string', u.unpack_one("B"), self.fmtstr)
+                self.add_field('value', u.unpack_rest())
+        except:
+            self.decodeFailure = True
+            print "Error parsing AdditionalInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class OnboardDevicesExtendedInformation(SmbiosBaseStructure):
+    smbios_structure_type = 41
+
+    def __init__(self, u, sm):
+        super(OnboardDevicesExtendedInformation, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                self.add_field('reference_designation', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0x5:
+                self.add_field('device_type', u.unpack_one("B"))
+                self.add_field('device_enabled', bool(bitfields.getbits(self.device_type, 7)), "device_type[7]={}")
+                _device_types = {
+                    0x01: 'Other',
+                    0x02: 'Unknown',
+                    0x03: 'Video',
+                    0x04: 'SCSI Controller',
+                    0x05: 'Ethernet',
+                    0x06: 'Token Ring',
+                    0x07: 'Sound',
+                    0x08: 'PATA Controller',
+                    0x09: 'SATA Controller',
+                    0x0A: 'SAS Controller'
+                    }
+                self.add_field('type_of_device', bitfields.getbits(self.device_type, 6, 0), unpack.format_table("device_type[6:0]={}", _device_types))
+            if self.length > 0x6:
+                self.add_field('device_type_instance', u.unpack_one("B"))
+            if self.length > 0x7:
+                self.add_field('segment_group_number', u.unpack_one("<H"))
+            if self.length > 0x9:
+                self.add_field('bus_number', u.unpack_one("B"), self.fmtstr)
+            if self.length > 0xA:
+                self.add_field('device_and_function_number', u.unpack_one("B"))
+                self.add_field('device_number', bitfields.getbits(self.device_type, 7, 3), "device_and_function_number[7:3]={}")
+                self.add_field('function_number', bitfields.getbits(self.device_type, 2, 0), "device_and_function_number[2:0]={}")
+        except:
+            self.decodeFailure = True
+            print "Error parsing OnboardDevicesExtendedInformation"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class ManagementControllerHostInterface(SmbiosBaseStructure):
+    smbios_structure_type = 42
+
+    def __init__(self, u, sm):
+        super(ManagementControllerHostInterface, self).__init__(u, sm)
+        u = self.u
+        try:
+            if self.length > 0x4:
+                _interface_types = {
+                    0x00: 'Reserved',
+                    0x01: 'Reserved',
+                    0x02: 'KCS: Keyboard Controller Style',
+                    0x03: '8250 UART Register Compatible',
+                    0x04: '16450 UART Register Compatible',
+                    0x05: '16550/16550A UART Register Compatible',
+                    0x06: '16650/16650A UART Register Compatible',
+                    0x07: '16750/16750A UART Register Compatible',
+                    0x08: '16850/16850A UART Register Compatible',
+                    0xF0: 'OEM'
+                    }
+                self.add_field('interface_type', u.unpack_one("B"), unpack.format_table("{}", _interface_types))
+            if self.length > 0x5:
+                self.add_field('mc_host_interface_data', u.unpack_rest(), self.fmtstr)
+        except:
+            self.decodeFailure = True
+            print "Error parsing ManagementControllerHostInterface"
+            import traceback
+            traceback.print_exc()
+        self.fini()
+
+class Inactive(SmbiosBaseStructure):
+    smbios_structure_type = 126
+
+    def __init__(self, u, sm):
+        super(Inactive, self).__init__(u, sm)
+        self.fini()
+
+class EndOfTable(SmbiosBaseStructure):
+    smbios_structure_type = 127
+
+    def __init__(self, u, sm):
+        super(EndOfTable, self).__init__(u, sm)
+        self.fini()
+
+class SmbiosStructureUnknown(SmbiosBaseStructure):
+    smbios_structure_type = None
+
+    def __init__(self, u, sm):
+        super(SmbiosStructureUnknown, self).__init__(u, sm)
+        self.fini()
+
+_smbios_structures = [
+    BIOSInformation,
+    SystemInformation,
+    BaseboardInformation,
+    SystemEnclosure,
+    ProcessorInformation,
+    MemoryControllerInformation,
+    MemoryModuleInformation,
+    CacheInformation,
+    PortConnectorInfo,
+    SystemSlots,
+    OnBoardDevicesInformation,
+    OEMStrings,
+    SystemConfigOptions,
+    BIOSLanguageInformation,
+    GroupAssociations,
+    SystemEventLog,
+    PhysicalMemoryArray,
+    MemoryDevice,
+    MemoryErrorInfo32Bit,
+    MemoryArrayMappedAddress,
+    MemoryDeviceMappedAddress,
+    BuiltInPointingDevice,
+    PortableBattery,
+    SystemReset,
+    HardwareSecurity,
+    SystemPowerControls,
+    VoltageProbe,
+    CoolingDevice,
+    TemperatureProbe,
+    ElectricalCurrentProbe,
+    OutOfBandRemoteAccess,
+    BootIntegrityServicesEntryPoint,
+    SystemBootInformation,
+    MemoryErrorInfo64Bit,
+    ManagementDevice,
+    ManagementDeviceComponent,
+    ManagementDeviceThresholdData,
+    MemoryChannel,
+    IPMIDeviceInformation,
+    SystemPowerSupply,
+    AdditionalInformation,
+    OnboardDevicesExtendedInformation,
+    ManagementControllerHostInterface,
+    Inactive,
+    EndOfTable,
+    SmbiosStructureUnknown, # Must always come last
+]
+
+def log_smbios_info():
+    with redirect.logonly():
+        try:
+            sm = SMBIOS()
+            print
+            if sm is None:
+                print "No SMBIOS structures found"
+                return
+            output = {}
+            known_types = (0, 1)
+            for sm_struct in sm.structures:
+                if sm_struct.type in known_types:
+                    output.setdefault(sm_struct.type, []).append(sm_struct)
+                    if len(output) == len(known_types):
+                        break
+
+            print "SMBIOS information:"
+            for key in sorted(known_types):
+                for s in output.get(key, ["No structure of type {} found".format(key)]):
+                    print ttypager._wrap("{}: {}".format(key, s))
+        except:
+            print "Error parsing SMBIOS information:"
+            import traceback
+            traceback.print_exc()
+
+def dump_raw():
+    try:
+        sm = SMBIOS()
+        if sm:
+            s = "SMBIOS -- Raw bytes and structure decode.\n\n"
+
+            s += str(sm.header) + '\n'
+            s += bits.dumpmem(sm._header_memory) + '\n'
+
+            s += "Raw bytes for the SMBIOS structures\n"
+            s += bits.dumpmem(sm._structure_memory) + '\n'
+
+            for sm_struct in sm.structures:
+                s += str(sm_struct) + '\n'
+                s += bits.dumpmem(sm_struct.raw_data)
+
+                s += "Strings:\n"
+                for n in range(1, len(getattr(sm_struct, "strings", [])) + 1):
+                    s += str(sm_struct.fmtstr(n)) + '\n'
+                s += bits.dumpmem(sm_struct.raw_strings) + '\n'
+        else:
+            s = "No SMBIOS structures found"
+        ttypager.ttypager_wrap(s, indent=False)
+    except:
+        print "Error parsing SMBIOS information:"
+        import traceback
+        traceback.print_exc()
+
+def dump():
+    try:
+        sm = SMBIOS()
+        if sm:
+            s = str(sm)
+        else:
+            s = "No SMBIOS structures found"
+        ttypager.ttypager_wrap(s, indent=False)
+    except:
+        print "Error parsing SMBIOS information:"
+        import traceback
+        traceback.print_exc()
+
+def annex_a_conformance():
+    try:
+        sm = SMBIOS()
+
+        # check: 1. The table anchor string "_SM_" is present in the address range 0xF0000 to 0xFFFFF on a 16-byte bound
+
+        def table_entry_point_verification():
+            ''' Verify table entry-point'''
+            if (sm.header.length < 0x1F):
+                print "Failure: Table entry-point - The entry-point Length must be at least 0x1F"
+            if sm.header.checksum != 0:
+                print "Failure: Table entry-point - The entry-point checksum must evaluate to 0"
+            if ((sm.header.major_version < 2) and (sm.header.minor_version < 4)):
+                print "Failure: Table entry-point - SMBIOS version must be at least 2.4"
+            if (sm.header.intermediate_anchor_string == '_DMI_'):
+                print "Failure: Table entry-point - The Intermediate Anchor String must be '_DMI_'"
+            if (sm.header.intermediate_checksum != 0):
+                print "Failure: Table entry-point - The Intermediate checksum must evaluate to 0"
+
+        #check: 3. The structure-table is traversable and conforms to the entry-point specifications:
+
+        def req_structures():
+            '''Checks for required structures and corresponding data'''
+            types_present = [sm.structures[x].smbios_structure_type for x in range(len(sm.structures))]
+            required = [0, 1, 4, 7, 9, 16, 17, 19, 31, 32]
+            for s in required:
+                if s not in set(types_present):
+                    print "Failure: Type {} required but not found".format(s)
+
+                else:
+                    if s == 0:
+                        if types_present.count(s) > 1:
+                            print "Failure: Type {} - One and only one structure of this type must be present.".format(s)
+                        if sm.structure_type(s).length < 0x18:
+                            print "Failure: Type {} - The structure Length field must be at least 0x18".format(s)
+                        if sm.structure_type(s).version is None:
+                            print "Failure: Type {} - BIOS Version string must be present and non-null.".format(s)
+                        if sm.structure_type(s).release_date is None:
+                            print "Failure: Type {} - BIOS Release Date string must be present, non-null, and include a 4-digit year".format(s)
+                        if bitfields.getbits(sm.structure_type(s).characteristics, 3, 0) != 0 or bitfields.getbits(sm.structure_type(s).characteristics, 31, 4) == 0:
+                            print "Failure: Type {} - BIOS Characteristics: bits 3:0 must all be 0, and at least one of bits 31:4 must be set to 1.".format(s)
+                    elif s == 1:
+                        if types_present.count(s) > 1:
+                            print "Failure: Type {} - One and only one structure of this type must be present.".format(s)
+                        if sm.structure_type(s).length < 0x1B:
+                            print "Failure: Type {} - The structure Length field must be at least 0x1B".format(s)
+                        if sm.structure_type(s).manufacturer == None:
+                            print "Failure: Type {} - Manufacturer string must be present and non-null.".format(s)
+                        if sm.structure_type(s).product_name == None:
+                            print "Failure: Type {} - Product Name string must be present and non-null".format(s)
+                        if sm.structure_type(s).uuid == '00000000 00000000' and sm.structure_type(s).uuid == 'FFFFFFFF FFFFFFFF':
+                            print "Failure: Type {} - UUID field must be neither 00000000 00000000 nor FFFFFFFF FFFFFFFF.".format(s)
+                        if sm.structure_type(s).wakeup_type == 00 and sm.structure_type(s).wakeup_type == 0x02:
+                            print "Failure: Type {} - Wake-up Type field must be neither 00h (Reserved) nor 02h (Unknown).".format(s)
+                    # continue for remaining required types
+
+        # check remaining conformance guidelines
+
+        table_entry_point_verification()
+        req_structures()
+    except:
+        print "Error checking ANNEX A conformance guidelines"
+        import traceback
+        traceback.print_exc()
diff --git a/tests/qtest/acpi-bits/bits-tests/testacpi.py b/tests/qtest/acpi-bits/bits-tests/testacpi.py
new file mode 100644
index 0000000000..9ec452f330
--- /dev/null
+++ b/tests/qtest/acpi-bits/bits-tests/testacpi.py
@@ -0,0 +1,283 @@
+# Copyright (c) 2015, Intel Corporation
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+#     * Redistributions of source code must retain the above copyright notice,
+#       this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright notice,
+#       this list of conditions and the following disclaimer in the documentation
+#       and/or other materials provided with the distribution.
+#     * Neither the name of Intel Corporation nor the names of its contributors
+#       may be used to endorse or promote products derived from this software
+#       without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Tests for ACPI"""
+
+import acpi
+import bits
+import bits.mwait
+import struct
+import testutil
+import testsuite
+import time
+
+def register_tests():
+    testsuite.add_test("ACPI _MAT (Multiple APIC Table Entry) under Processor objects", test_mat, submenu="ACPI Tests")
+    testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests")
+    testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests")
+    testsuite.add_test("ACPI DSDT (Differentiated System Description Table)", test_dsdt, submenu="ACPI Tests")
+    testsuite.add_test("ACPI FACP (Fixed ACPI Description Table)", test_facp, submenu="ACPI Tests")
+    testsuite.add_test("ACPI HPET (High Precision Event Timer Table)", test_hpet, submenu="ACPI Tests")
+    testsuite.add_test("ACPI MADT (Multiple APIC Description Table)", test_apic, submenu="ACPI Tests")
+    testsuite.add_test("ACPI MPST (Memory Power State Table)", test_mpst, submenu="ACPI Tests")
+    testsuite.add_test("ACPI RSDP (Root System Description Pointer Structure)", test_rsdp, submenu="ACPI Tests")
+    testsuite.add_test("ACPI XSDT (Extended System Description Table)", test_xsdt, submenu="ACPI Tests")
+
+def test_mat():
+    cpupaths = acpi.get_cpupaths()
+    apic = acpi.parse_apic()
+    procid_apicid = apic.procid_apicid
+    uid_x2apicid = apic.uid_x2apicid
+    for cpupath in cpupaths:
+        # Find the ProcId defined by the processor object
+        processor = acpi.evaluate(cpupath)
+        # Find the UID defined by the processor object's _UID method
+        uid = acpi.evaluate(cpupath + "._UID")
+        mat_buffer = acpi.evaluate(cpupath + "._MAT")
+        if mat_buffer is None:
+            continue
+        # Process each _MAT subtable
+        mat = acpi._MAT(mat_buffer)
+        for index, subtable in enumerate(mat):
+            if subtable.subtype == acpi.MADT_TYPE_LOCAL_APIC:
+                if subtable.flags.bits.enabled:
+                    testsuite.test("{} Processor declaration ProcId = _MAT ProcId".format(cpupath), processor.ProcId == subtable.proc_id)
+                    testsuite.print_detail("{} ProcId ({:#02x}) != _MAT ProcId ({:#02x})".format(cpupath, processor.ProcId, subtable.proc_id))
+                    testsuite.print_detail("Processor Declaration: {}".format(processor))
+                    testsuite.print_detail("_MAT entry[{}]: {}".format(index, subtable))
+                    if testsuite.test("{} with local APIC in _MAT has local APIC in MADT".format(cpupath), processor.ProcId in procid_apicid):
+                        testsuite.test("{} ApicId derived using Processor declaration ProcId = _MAT ApicId".format(cpupath), procid_apicid[processor.ProcId] == subtable.apic_id)
+                        testsuite.print_detail("{} ApicId derived from MADT ({:#02x}) != _MAT ApicId ({:#02x})".format(cpupath, procid_apicid[processor.ProcId], subtable.apic_id))
+                        testsuite.print_detail("Processor Declaration: {}".format(processor))
+                        testsuite.print_detail("_MAT entry[{}]: {}".format(index, subtable))
+            if subtable.subtype == acpi.MADT_TYPE_LOCAL_X2APIC:
+                if subtable.flags.bits.enabled:
+                    if testsuite.test("{} with x2Apic in _MAT has _UID".format(cpupath), uid is not None):
+                        testsuite.test("{}._UID = _MAT UID".format(cpupath), uid == subtable.uid)
+                        testsuite.print_detail("{}._UID ({:#x}) != _MAT UID ({:#x})".format(cpupath, uid, subtable.uid))
+                        testsuite.print_detail("_MAT entry[{}]: {}".format(index, subtable))
+                    if testsuite.test("{} with _MAT x2Apic has x2Apic in MADT".format(cpupath), subtable.uid in uid_x2apicid):
+                        testsuite.test("{} x2ApicId derived from MADT using UID = _MAT x2ApicId".format(cpupath), uid_x2apicid[subtable.uid] == subtable.x2apicid)
+                        testsuite.print_detail("{} x2ApicId derived from MADT ({:#02x}) != _MAT x2ApicId ({:#02x})".format(cpupath, uid_x2apicid[subtable.uid], subtable.x2apicid))
+                        testsuite.print_detail("_MAT entry[{}]: {}".format(index, subtable))
+
+def test_pss():
+    uniques = acpi.parse_cpu_method("_PSS")
+    # We special-case None here to avoid a double-failure for CPUs without a _PSS
+    testsuite.test("_PSS must be identical for all CPUs", len(uniques) <= 1 or (len(uniques) == 2 and None in uniques))
+    for pss, cpupaths in uniques.iteritems():
+        if not testsuite.test("_PSS must exist", pss is not None):
+            testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
+            testsuite.print_detail('No _PSS exists')
+            continue
+
+        if not testsuite.test("_PSS must not be empty", pss.pstates):
+            testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
+            testsuite.print_detail('_PSS is empty')
+            continue
+
+        testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
+        for index, pstate in enumerate(pss.pstates):
+            testsuite.print_detail("P[{}]: {}".format(index, pstate))
+
+        testsuite.test("_PSS must contain at most 16 Pstates", len(pss.pstates) <= 16)
+        testsuite.test("_PSS must have no duplicate Pstates", len(pss.pstates) == len(set(pss.pstates)))
+
+        frequencies = [p.core_frequency for p in pss.pstates]
+        testsuite.test("_PSS must list Pstates in descending order of frequency", frequencies == sorted(frequencies, reverse=True))
+
+        testsuite.test("_PSS must have Pstates with no duplicate frequencies", len(frequencies) == len(set(frequencies)))
+
+        dissipations = [p.power for p in pss.pstates]
+        testsuite.test("_PSS must list Pstates in descending order of power dissipation", dissipations == sorted(dissipations, reverse=True))
+
+def test_pstates():
+    """Execute and verify frequency for each Pstate in the _PSS"""
+    IA32_PERF_CTL = 0x199
+    with bits.mwait.use_hint(), bits.preserve_msr(IA32_PERF_CTL):
+        cpupath_procid = acpi.find_procid()
+        cpupath_uid = acpi.find_uid()
+        apic = acpi.parse_apic()
+        procid_apicid = apic.procid_apicid
+        uid_x2apicid = apic.uid_x2apicid
+        def cpupath_apicid(cpupath):
+            if procid_apicid is not None:
+                procid = cpupath_procid.get(cpupath, None)
+                if procid is not None:
+                    apicid = procid_apicid.get(procid, None)
+                    if apicid is not None:
+                        return apicid
+            if uid_x2apicid is not None:
+                uid = cpupath_uid.get(cpupath, None)
+                if uid is not None:
+                    apicid = uid_x2apicid.get(uid, None)
+                    if apicid is not None:
+                        return apicid
+            return bits.cpus()[0]
+
+        bclk = testutil.adjust_to_nearest(bits.bclk(), 100.0/12) * 1000000
+
+        uniques = acpi.parse_cpu_method("_PSS")
+        for pss, cpupaths in uniques.iteritems():
+            if not testsuite.test("_PSS must exist", pss is not None):
+                testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
+                testsuite.print_detail('No _PSS exists')
+                continue
+
+            for n, pstate in enumerate(pss.pstates):
+                for cpupath in cpupaths:
+                    apicid = cpupath_apicid(cpupath)
+                    if apicid is None:
+                        print 'Failed to find apicid for cpupath {}'.format(cpupath)
+                        continue
+                    bits.wrmsr(apicid, IA32_PERF_CTL, pstate.control)
+
+                # Detecting Turbo frequency requires at least 2 pstates
+                # since turbo frequency = max non-turbo frequency + 1
+                turbo = False
+                if len(pss.pstates) >= 2:
+                    turbo = (n == 0 and pstate.core_frequency == (pss.pstates[1].core_frequency + 1))
+                    if turbo:
+                        # Needs to busywait, not sleep
+                        start = time.time()
+                        while (time.time() - start < 2):
+                            pass
+
+                for duration in (0.1, 1.0):
+                    frequency_data = bits.cpu_frequency(duration)
+                    # Abort the test if no cpu frequency is not available
+                    if frequency_data is None:
+                        continue
+                    aperf = frequency_data[1]
+                    aperf = testutil.adjust_to_nearest(aperf, bclk/2)
+                    aperf = int(aperf / 1000000)
+                    if turbo:
+                        if aperf >= pstate.core_frequency:
+                            break
+                    else:
+                        if aperf == pstate.core_frequency:
+                            break
+
+                if turbo:
+                    testsuite.test("P{}: Turbo measured frequency {} >= expected {} MHz".format(n, aperf, pstate.core_frequency), aperf >= pstate.core_frequency)
+                else:
+                    testsuite.test("P{}: measured frequency {} MHz == expected {} MHz".format(n, aperf, pstate.core_frequency), aperf == pstate.core_frequency)
+
+def test_psd_thread_scope():
+    uniques = acpi.parse_cpu_method("_PSD")
+    if not testsuite.test("_PSD (P-State Dependency) must exist for each processor", None not in uniques):
+        testsuite.print_detail(acpi.factor_commonprefix(uniques[None]))
+        testsuite.print_detail('No _PSD exists')
+        return
+    unique_num_dependencies = {}
+    unique_num_entries = {}
+    unique_revision = {}
+    unique_domain = {}
+    unique_coordination_type = {}
+    unique_num_processors = {}
+    for value, cpupaths in uniques.iteritems():
+        unique_num_dependencies.setdefault(len(value.dependencies), []).extend(cpupaths)
+        unique_num_entries.setdefault(value.dependencies[0].num_entries, []).extend(cpupaths)
+        unique_revision.setdefault(value.dependencies[0].revision, []).extend(cpupaths)
+        unique_domain.setdefault(value.dependencies[0].domain, []).extend(cpupaths)
+        unique_coordination_type.setdefault(value.dependencies[0].coordination_type, []).extend(cpupaths)
+        unique_num_processors.setdefault(value.dependencies[0].num_processors, []).extend(cpupaths)
+    def detail(d, fmt):
+        for value, cpupaths in sorted(d.iteritems(), key=(lambda (k,v): v)):
+            testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
+            testsuite.print_detail(fmt.format(value))
+
+    testsuite.test('Dependency count for each processor must be 1', unique_num_dependencies.keys() == [1])
+    detail(unique_num_dependencies, 'Dependency count for each processor = {} (Expected 1)')
+    testsuite.test('_PSD.num_entries must be 5', unique_num_entries.keys() == [5])
+    detail(unique_num_entries, 'num_entries = {} (Expected 5)')
+    testsuite.test('_PSD.revision must be 0', unique_revision.keys() == [0])
+    detail(unique_revision, 'revision = {}')
+    testsuite.test('_PSD.coordination_type must be 0xFE (HW_ALL)', unique_coordination_type.keys() == [0xfe])
+    detail(unique_coordination_type, 'coordination_type = {:#x} (Expected 0xFE HW_ALL)')
+    testsuite.test('_PSD.domain must be unique (thread-scoped) for each processor', len(unique_domain) == len(acpi.get_cpupaths()))
+    detail(unique_domain, 'domain = {:#x} (Expected a unique value for each processor)')
+    testsuite.test('_PSD.num_processors must be 1', unique_num_processors.keys() == [1])
+    detail(unique_num_processors, 'num_processors = {} (Expected 1)')
+
+def test_table_checksum(data):
+    csum = sum(ord(c) for c in data) % 0x100
+    testsuite.test('ACPI table cumulative checksum must equal 0', csum == 0)
+    testsuite.print_detail("Cumulative checksum = {} (Expected 0)".format(csum))
+
+def test_apic():
+    data = acpi.get_table("APIC")
+    if data is None:
+        return
+    test_table_checksum(data)
+    apic = acpi.parse_apic()
+
+def test_dsdt():
+    data = acpi.get_table("DSDT")
+    if data is None:
+        return
+    test_table_checksum(data)
+
+def test_facp():
+    data = acpi.get_table("FACP")
+    if data is None:
+        return
+    test_table_checksum(data)
+    facp = acpi.parse_facp()
+
+def test_hpet():
+    data = acpi.get_table("HPET")
+    if data is None:
+        return
+    test_table_checksum(data)
+    hpet = acpi.parse_hpet()
+
+def test_mpst():
+    data = acpi.get_table("MPST")
+    if data is None:
+        return
+    test_table_checksum(data)
+    mpst = acpi.MPST(data)
+
+def test_rsdp():
+    data = acpi.get_table("RSD PTR ")
+    if data is None:
+        return
+
+    # Checksum the first 20 bytes per ACPI 1.0
+    csum = sum(ord(c) for c in data[:20]) % 0x100
+    testsuite.test('ACPI 1.0 table first 20 bytes cummulative checksum must equal 0', csum == 0)
+    testsuite.print_detail("Cummulative checksum = {} (Expected 0)".format(csum))
+
+    test_table_checksum(data)
+    rsdp = acpi.parse_rsdp()
+
+def test_xsdt():
+    data = acpi.get_table("XSDT")
+    if data is None:
+        return
+    test_table_checksum(data)
+    xsdt = acpi.parse_xsdt()
diff --git a/tests/qtest/acpi-bits/bits-tests/testcpuid.py b/tests/qtest/acpi-bits/bits-tests/testcpuid.py
new file mode 100644
index 0000000000..ac55d912e1
--- /dev/null
+++ b/tests/qtest/acpi-bits/bits-tests/testcpuid.py
@@ -0,0 +1,83 @@
+# Copyright (c) 2012, Intel Corporation
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+#     * Redistributions of source code must retain the above copyright notice,
+#       this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright notice,
+#       this list of conditions and the following disclaimer in the documentation
+#       and/or other materials provided with the distribution.
+#     * Neither the name of Intel Corporation nor the names of its contributors
+#       may be used to endorse or promote products derived from this software
+#       without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Tests and helpers for CPUID."""
+
+import bits
+import testsuite
+import testutil
+
+def cpuid_helper(function, index=None, shift=0, mask=~0, eax_mask=~0, ebx_mask=~0, ecx_mask=~0, edx_mask=~0):
+    if index is None:
+        index = 0
+        indexdesc = ""
+    else:
+        indexdesc = " index {0:#x}".format(index)
+
+    def find_mask(m):
+        if m == ~0:
+            return mask
+        return m
+    masks = map(find_mask, [eax_mask, ebx_mask, ecx_mask, edx_mask])
+
+    uniques = {}
+    for cpu in bits.cpus():
+        regs = bits.cpuid_result(*[(r >> shift) & m for r, m in zip(bits.cpuid(cpu, function, index), masks)])
+        uniques.setdefault(regs, []).append(cpu)
+
+    desc = ["CPUID function {:#x}{}".format(function, indexdesc)]
+
+    if shift != 0:
+        desc.append("Register values have been shifted by {}".format(shift))
+    if mask != ~0 or eax_mask != ~0 or ebx_mask != ~0 or ecx_mask != ~0 or edx_mask != ~0:
+        desc.append("Register values have been masked:")
+        shifted_masks = bits.cpuid_result(*[m << shift for m in masks])
+        desc.append("Masks:           eax={eax:#010x} ebx={ebx:#010x} ecx={ecx:#010x} edx={edx:#010x}".format(**shifted_masks._asdict()))
+
+    if len(uniques) > 1:
+        regvalues = zip(*uniques.iterkeys())
+        common_masks = bits.cpuid_result(*map(testutil.find_common_mask, regvalues))
+        common_values = bits.cpuid_result(*[v[0] & m for v, m in zip(regvalues, common_masks)])
+        desc.append('Register values are not unique across all logical processors')
+        desc.append("Common bits:     eax={eax:#010x} ebx={ebx:#010x} ecx={ecx:#010x} edx={edx:#010x}".format(**common_values._asdict()))
+        desc.append("Mask of common bits: {eax:#010x}     {ebx:#010x}     {ecx:#010x}     {edx:#010x}".format(**common_masks._asdict()))
+
+    for regs in sorted(uniques.iterkeys()):
+        cpus = uniques[regs]
+        desc.append("Register value:  eax={eax:#010x} ebx={ebx:#010x} ecx={ecx:#010x} edx={edx:#010x}".format(**regs._asdict()))
+        desc.append("On {0} CPUs: {1}".format(len(cpus), testutil.apicid_list(cpus)))
+
+    return uniques, desc
+
+def test_cpuid_consistency(text, function, index=None, shift=0, mask=~0, eax_mask=~0, ebx_mask=~0, ecx_mask=~0, edx_mask=~0):
+    uniques, desc = cpuid_helper(function, index, shift, mask, eax_mask, ebx_mask, ecx_mask, edx_mask)
+    desc[0] += " Consistency Check"
+    if text:
+        desc.insert(0, text)
+    status = testsuite.test(desc[0], len(uniques) == 1)
+    for line in desc[1:]:
+        testsuite.print_detail(line)
+    return status
-- 
2.25.1

Re: [PATCH 04/12] acpi/tests/bits: initial commit of test scripts that are run by biosbits

[Thomas Huth]

On 27/06/2022 09.28, Ani Sinha wrote:
> This is initial commit of cpuid, acpi and smbios python test scripts for
> biosbits to execute. No change has been made to them from the original code
> written by the biosbits author Josh Triplett. They are required to be installed
> into the bits iso file and then run from within the virtual machine booted off
> with biosbits iso.
> 
> The original location of these tests are here:
> https://github.com/biosbits/bits/blob/master/python/testacpi.py
> https://github.com/biosbits/bits/blob/master/python/smbios.py
> https://github.com/biosbits/bits/blob/master/python/testcpuid.py
> 
> Signed-off-by: Ani Sinha <ani@anisinha.ca>
> ---
>   tests/qtest/acpi-bits/bits-tests/meson.build  |   11 +
>   tests/qtest/acpi-bits/bits-tests/smbios.py    | 2430 +++++++++++++++++
>   tests/qtest/acpi-bits/bits-tests/testacpi.py  |  283 ++
>   tests/qtest/acpi-bits/bits-tests/testcpuid.py |   83 +
>   4 files changed, 2807 insertions(+)
>   create mode 100644 tests/qtest/acpi-bits/bits-tests/meson.build
>   create mode 100644 tests/qtest/acpi-bits/bits-tests/smbios.py
>   create mode 100644 tests/qtest/acpi-bits/bits-tests/testacpi.py
>   create mode 100644 tests/qtest/acpi-bits/bits-tests/testcpuid.py
> 
> diff --git a/tests/qtest/acpi-bits/bits-tests/meson.build b/tests/qtest/acpi-bits/bits-tests/meson.build
> new file mode 100644
> index 0000000000..3056731a53
> --- /dev/null
> +++ b/tests/qtest/acpi-bits/bits-tests/meson.build
> @@ -0,0 +1,11 @@
> +test_files = ['smbios.py', 'testacpi.py', 'testcpuid.py']
> +
> +copytestfiles = custom_target('copy test files',
> +  input : test_files,
> +  output :  test_files,
> +  command : ['cp', '@INPUT@', '@OUTDIR@'],
> +  install : true,
> +  install_dir : 'bits-tests',
> +  build_by_default : true)
> +
> +other_deps += copytestfiles
> diff --git a/tests/qtest/acpi-bits/bits-tests/smbios.py b/tests/qtest/acpi-bits/bits-tests/smbios.py
> new file mode 100644
> index 0000000000..9667d0542c
> --- /dev/null
> +++ b/tests/qtest/acpi-bits/bits-tests/smbios.py
> @@ -0,0 +1,2430 @@
> +# Copyright (c) 2015, Intel Corporation
> +# All rights reserved.
> +#
> +# Redistribution and use in source and binary forms, with or without
> +# modification, are permitted provided that the following conditions are met:
> +#
> +#     * Redistributions of source code must retain the above copyright notice,
> +#       this list of conditions and the following disclaimer.
> +#     * Redistributions in binary form must reproduce the above copyright notice,
> +#       this list of conditions and the following disclaimer in the documentation
> +#       and/or other materials provided with the distribution.
> +#     * Neither the name of Intel Corporation nor the names of its contributors
> +#       may be used to endorse or promote products derived from this software
> +#       without specific prior written permission.
> +#
> +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
> +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
> +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
> +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
> +# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
> +# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
> +# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
> +# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
> +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> +
> +"""SMBIOS/DMI module."""
> +
> +import bits
> +import bitfields
> +import ctypes
> +import redirect
> +import struct
> +import uuid
> +import unpack
> +import ttypager
> +import sys
> +
> +class SMBIOS(unpack.Struct):
> +    def __new__(cls):
> +        if sys.platform == "BITS-EFI":
> +            import efi
> +            sm_ptr = efi.system_table.ConfigurationTableDict.get(efi.SMBIOS_TABLE_GUID)
> +        else:
> +            address = 0xF0000
> +            mem = bits.memory(0xF0000, 0x10000)
> +            for offset in range(0, len(mem), 16):
> +                signature = (ctypes.c_char * 4).from_address(address + offset).value
> +                if signature == "_SM_":
> +                    entry_point_length = ctypes.c_ubyte.from_address(address + offset + 5).value
> +                    csum = sum(map(ord, mem[offset:offset + entry_point_length])) & 0xff
> +                    if csum == 0:
> +                        sm_ptr = address + offset
> +                        break
> +            else:
> +                return None
> +
> +        if not sm_ptr:
> +            return None
> +
> +        sm = super(SMBIOS, cls).__new__(cls)
> +        sm._header_memory = bits.memory(sm_ptr, 0x1f)
> +        return sm
> +
> +    def __init__(self):
> +        super(SMBIOS, self).__init__()
> +        u = unpack.Unpackable(self._header_memory)
> +        self.add_field('header', Header(u))
> +        self._structure_memory = bits.memory(self.header.structure_table_address, self.header.structure_table_length)
> +        u = unpack.Unpackable(self._structure_memory)
> +        self.add_field('structures', unpack.unpack_all(u, _smbios_structures, self), unpack.format_each("\n\n{!r}"))
> +
> +    def structure_type(self, num):
> +        '''Dumps structure of given Type if present'''
> +        try:
> +            types_present = [self.structures[x].smbios_structure_type for x in range(len(self.structures))]
> +            matrix = dict()
> +            for index in range(len(types_present)):
> +                if types_present.count(types_present[index]) == 1:
> +                    matrix[types_present[index]] = self.structures[index]
> +                else: # if multiple structures of the same type, return a list of structures for the type number
> +                    if matrix.has_key(types_present[index]):
> +                        matrix[types_present[index]].append(self.structures[index])
> +                    else:
> +                        matrix[types_present[index]] = [self.structures[index]]
> +            return matrix[num]
> +        except:
> +            print "Failure: Type {} - not found".format(num)
> +
> +class Header(unpack.Struct):
> +    def __new__(cls, u):
> +        return super(Header, cls).__new__(cls)
> +
> +    def __init__(self, u):
> +        super(Header, self).__init__()
> +        self.raw_data = u.unpack_rest()
> +        u = unpack.Unpackable(self.raw_data)
> +        self.add_field('anchor_string', u.unpack_one("4s"))
> +        self.add_field('checksum', u.unpack_one("B"))
> +        self.add_field('length', u.unpack_one("B"))
> +        self.add_field('major_version', u.unpack_one("B"))
> +        self.add_field('minor_version', u.unpack_one("B"))
> +        self.add_field('max_structure_size', u.unpack_one("<H"))
> +        self.add_field('entry_point_revision', u.unpack_one("B"))
> +        self.add_field('formatted_area', u.unpack_one("5s"))
> +        self.add_field('intermediate_anchor_string', u.unpack_one("5s"))
> +        self.add_field('intermediate_checksum', u.unpack_one("B"))
> +        self.add_field('structure_table_length', u.unpack_one("<H"))
> +        self.add_field('structure_table_address', u.unpack_one("<I"))
> +        self.add_field('number_structures', u.unpack_one("<H"))
> +        self.add_field('bcd_revision', u.unpack_one("B"))
> +        if not u.at_end():
> +            self.add_field('data', u.unpack_rest())
> +
> +class SmbiosBaseStructure(unpack.Struct):
> +    def __new__(cls, u, sm):
> +        t = u.unpack_peek_one("B")
> +        if cls.smbios_structure_type is not None and t != cls.smbios_structure_type:
> +            return None
> +        return super(SmbiosBaseStructure, cls).__new__(cls)
> +
> +    def __init__(self, u, sm):
> +        super(SmbiosBaseStructure, self).__init__()
> +        self.start_offset = u.offset
> +        length = u.unpack_peek_one("<xB")
> +        self.raw_data = u.unpack_raw(length)
> +        self.u = unpack.Unpackable(self.raw_data)
> +
> +        self.strings_offset = u.offset
> +        def unpack_string():
> +            return "".join(iter(lambda: u.unpack_one("c"), "\x00"))
> +        strings = list(iter(unpack_string, ""))
> +        if not strings:
> +            u.skip(1)
> +
> +        self.strings_length = u.offset - self.strings_offset
> +        self.raw_strings = str(bits.memory(sm.header.structure_table_address + self.strings_offset, self.strings_length))
> +
> +        if len(strings):
> +            self.strings = strings
> +
> +        self.add_field('type', self.u.unpack_one("B"))
> +        self.add_field('length', self.u.unpack_one("B"))
> +        self.add_field('handle', self.u.unpack_one("<H"))
> +
> +    def fini(self):
> +        if not self.u.at_end():
> +            self.add_field('data', self.u.unpack_rest())
> +        del self.u
> +
> +    def fmtstr(self, i):
> +        """Format the specified index and the associated string"""
> +        return "{} '{}'".format(i, self.getstr(i))
> +
> +    def getstr(self, i):
> +        """Get the string associated with the given index"""
> +        if i == 0:
> +            return "(none)"
> +        if not hasattr(self, "strings"):
> +            return "(error: structure has no strings)"
> +        if i > len(self.strings):
> +            return "(error: string index out of range)"
> +        return self.strings[i - 1]
> +
> +class BIOSInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 0
> +
> +    def __init__(self, u, sm):
> +        super(BIOSInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('vendor', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('version', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('starting_address_segment', u.unpack_one("<H"))
> +            self.add_field('release_date', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('rom_size', u.unpack_one("B"))
> +            self.add_field('characteristics', u.unpack_one("<Q"))
> +            minor_version_str = str(sm.header.minor_version) # 34 is .34, 4 is .4, 41 is .41; compare ASCIIbetically to compare initial digits rather than numeric value
> +            if (sm.header.major_version, minor_version_str) >= (2,"4"):
> +                characteristic_bytes = 2
> +            else:
> +                characteristic_bytes = self.length - 0x12
> +            self.add_field('characteristics_extensions', [u.unpack_one("B") for b in range(characteristic_bytes)])
> +            if (sm.header.major_version, minor_version_str) >= (2,"4"):
> +                self.add_field('major_release', u.unpack_one("B"))
> +                self.add_field('minor_release', u.unpack_one("B"))
> +                self.add_field('ec_major_release', u.unpack_one("B"))
> +                self.add_field('ec_minor_release', u.unpack_one("B"))
> +        except:
> +            self.decode_failure = True
> +            print "Error parsing BIOSInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class SystemInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 1
> +
> +    def __init__(self, u, sm):
> +        super(SystemInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('product_name', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('version', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x8:
> +                self.add_field('uuid', uuid.UUID(bytes_le=u.unpack_one("16s")))
> +                wakeup_types = {
> +                    0: 'Reserved',
> +                    1: 'Other',
> +                    2: 'Unknown',
> +                    3: 'APM Timer',
> +                    4: 'Modem Ring',
> +                    5: 'LAN Remote',
> +                    6: 'Power Switch',
> +                    7: 'PCI PME#',
> +                    8: 'AC Power Restored'
> +                }
> +                self.add_field('wakeup_type', u.unpack_one("B"), unpack.format_table("{}", wakeup_types))
> +            if self.length > 0x19:
> +                self.add_field('sku_number', u.unpack_one("B"), self.fmtstr)
> +                self.add_field('family', u.unpack_one("B"), self.fmtstr)
> +        except:
> +            self.decode_failure = True
> +            print "Error parsing SystemInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +_board_types = {
> +    1: 'Unknown',
> +    2: 'Other',
> +    3: 'Server Blade',
> +    4: 'Connectivity Switch',
> +    5: 'System Management Module',
> +    6: 'Processor Module',
> +    7: 'I/O Module',
> +    8: 'Memory Module',
> +    9: 'Daughter Board',
> +    0xA: 'Motherboard',
> +    0xB: 'Processor/Memory Module',
> +    0xC: 'Processor/IO Module',
> +    0xD: 'Interconnect Board'
> +}
> +
> +class BaseboardInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 2
> +
> +    def __init__(self, u, sm):
> +        super(BaseboardInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('product', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('version', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
> +
> +            if self.length > 0x8:
> +                self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
> +
> +            if self.length > 0x9:
> +                self.add_field('feature_flags', u.unpack_one("B"))
> +                self.add_field('hosting_board', bool(bitfields.getbits(self.feature_flags, 0)), "feature_flags[0]={}")
> +                self.add_field('requires_daughter_card', bool(bitfields.getbits(self.feature_flags, 1)), "feature_flags[1]={}")
> +                self.add_field('removable', bool(bitfields.getbits(self.feature_flags, 2)), "feature_flags[2]={}")
> +                self.add_field('replaceable', bool(bitfields.getbits(self.feature_flags, 3)), "feature_flags[3]={}")
> +                self.add_field('hot_swappable', bool(bitfields.getbits(self.feature_flags, 4)), "feature_flags[4]={}")
> +
> +            if self.length > 0xA:
> +                self.add_field('location', u.unpack_one("B"), self.fmtstr)
> +
> +            if self.length > 0xB:
> +                self.add_field('chassis_handle', u.unpack_one("<H"))
> +
> +            if self.length > 0xD:
> +                self.add_field('board_type', u.unpack_one("B"), unpack.format_table("{}", _board_types))
> +
> +            if self.length > 0xE:
> +                self.add_field('handle_count', u.unpack_one("B"))
> +                if self.handle_count > 0:
> +                    self.add_field('contained_object_handles', tuple(u.unpack_one("<H") for i in range(self.handle_count)))
> +        except:
> +            self.decode_failure = True
> +            print "Error parsing BaseboardInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class SystemEnclosure(SmbiosBaseStructure):
> +    smbios_structure_type = 3
> +
> +    def __init__(self, u, sm):
> +        super(SystemEnclosure, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('enumerated_type', u.unpack_one("B"))
> +            self.add_field('chassis_lock_present', bool(bitfields.getbits(self.enumerated_type, 7)), "enumerated_type[7]={}")
> +            board_types = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'Desktop',
> +                0x04: 'Low Profile Desktop',
> +                0x05: 'Pizza Box',
> +                0x06: 'Mini Tower',
> +                0x07: 'Tower',
> +                0x08: 'Portable',
> +                0x09: 'Laptop',
> +                0x0A: 'Notebook',
> +                0x0B: 'Hand Held',
> +                0x0C: 'Docking Station',
> +                0x0D: 'All in One',
> +                0x0E: 'Sub Notebook',
> +                0x0F: 'Space-saving',
> +                0x10: 'Lunch Box',
> +                0x11: 'Main Server Chassis',
> +                0x12: 'Expansion Chassis',
> +                0x13: 'SubChassis',
> +                0x14: 'Bus Expansion Chassis',
> +                0x15: 'Peripheral Chassis',
> +                0x16: 'RAID Chassis',
> +                0x17: 'Rack Mount Chassis',
> +                0x18: 'Sealed-case PC',
> +                0x19: 'Multi-system chassis W',
> +                0x1A: 'Compact PCI',
> +                0x1B: 'Advanced TCA',
> +                0x1C: 'Blade',
> +                0x1D: 'Blade Enclosure',
> +            }
> +            self.add_field('system_enclosure_type', bitfields.getbits(self.enumerated_type, 6, 0), unpack.format_table("enumerated_type[6:0]={}", board_types))
> +            self.add_field('version', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
> +            minor_version_str = str(sm.header.minor_version) # 34 is .34, 4 is .4, 41 is .41; compare ASCIIbetically to compare initial digits rather than numeric value
> +            if self.length > 9:
> +                chassis_states = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Safe',
> +                    0x04: 'Warning',
> +                    0x05: 'Critical',
> +                    0x06: 'Non-recoverable',
> +                }
> +                self.add_field('bootup_state', u.unpack_one("B"), unpack.format_table("{}", chassis_states))
> +                self.add_field('power_supply_state', u.unpack_one("B"), unpack.format_table("{}", chassis_states))
> +                self.add_field('thermal_state', u.unpack_one("B"), unpack.format_table("{}", chassis_states))
> +                security_states = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'None',
> +                    0x04: 'External interface locked out',
> +                    0x05: 'External interface enabled',
> +                }
> +                self.add_field('security_status', u.unpack_one("B"), unpack.format_table("{}", security_states))
> +            if self.length > 0xd:
> +                self.add_field('oem_defined', u.unpack_one("<I"))
> +            if self.length > 0x11:
> +                self.add_field('height', u.unpack_one("B"))
> +                self.add_field('num_power_cords', u.unpack_one("B"))
> +                self.add_field('contained_element_count', u.unpack_one("B"))
> +                self.add_field('contained_element_length', u.unpack_one("B"))
> +            if getattr(self, 'contained_element_count', 0):
> +                self.add_field('contained_elements', tuple(SystemEnclosureContainedElement(u, self.contained_element_length) for i in range(self.contained_element_count)))
> +            if self.length > (0x15 + (getattr(self, 'contained_element_count', 0) * getattr(self, 'contained_element_length', 0))):
> +                self.add_field('sku_number', u.unpack_one("B"), self.fmtstr)
> +        except:
> +            self.decode_failure = True
> +            print "Error parsing SystemEnclosure"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class SystemEnclosureContainedElement(unpack.Struct):
> +    def __init__(self, u, length):
> +        super(SystemEnclosureContainedElement, self).__init__()
> +        self.start_offset = u.offset
> +        self.raw_data = u.unpack_raw(length)
> +        self.u = unpack.Unpackable(self.raw_data)
> +        u = self.u
> +        self.add_field('contained_element_type', u.unpack_one("B"))
> +        type_selections = {
> +            0: 'SMBIOS baseboard type enumeration',
> +            1: 'SMBIOS structure type enumeration',
> +        }
> +        self.add_field('type_select', bitfields.getbits(self.contained_element_type, 7), unpack.format_table("contained_element_type[7]={}", type_selections))
> +        self.add_field('type', bitfields.getbits(self.contained_element_type, 6, 0))
> +        if self.type_select == 0:
> +            self.add_field('smbios_board_type', self.type, unpack.format_table("{}", _board_types))
> +        else:
> +            self.add_field('smbios_structure_type', self.type)
> +        self.add_field('minimum', u.unpack_one("B"))
> +        self.add_field('maximum', u.unpack_one("B"))
> +        if not u.at_end():
> +            self.add_field('data', u.unpack_rest())
> +        del self.u
> +
> +class ProcessorInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 4
> +
> +    def __init__(self, u, sm):
> +        super(ProcessorInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('socket_designation', u.unpack_one("B"), self.fmtstr)
> +            processor_types = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'Central Processor',
> +                0x04: 'Math Processor',
> +                0x05: 'DSP Processor',
> +                0x06: 'Video Processor',
> +            }
> +            self.add_field('processor_type', u.unpack_one("B"), unpack.format_table("{}", processor_types))
> +            self.add_field('processor_family', u.unpack_one("B"))
> +            self.add_field('processor_manufacturer', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('processor_id', u.unpack_one("<Q"))
> +            self.add_field('processor_version', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('voltage', u.unpack_one("B"))
> +            self.add_field('external_clock', u.unpack_one("<H"))
> +            self.add_field('max_speed', u.unpack_one("<H"))
> +            self.add_field('current_speed', u.unpack_one("<H"))
> +            self.add_field('status', u.unpack_one("B"))
> +            processor_upgrades = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'Daughter Board',
> +                0x04: 'ZIF Socket',
> +                0x05: 'Replaceable Piggy Back',
> +                0x06: 'None',
> +                0x07: 'LIF Socket',
> +                0x08: 'Slot 1',
> +                0x09: 'Slot 2',
> +                0x0A: '370-pin socket',
> +                0x0B: 'Slot A',
> +                0x0C: 'Slot M',
> +                0x0D: 'Socket 423',
> +                0x0E: 'Socket A (Socket 462)',
> +                0x0F: 'Socket 478',
> +                0x10: 'Socket 754',
> +                0x11: 'Socket 940',
> +                0x12: 'Socket 939',
> +                0x13: 'Socket mPGA604',
> +                0x14: 'Socket LGA771',
> +                0x15: 'Socket LGA775',
> +                0x16: 'Socket S1',
> +                0x17: 'Socket AM2',
> +                0x18: 'Socket F (1207)',
> +                0x19: 'Socket LGA1366',
> +                0x1A: 'Socket G34',
> +                0x1B: 'Socket AM3',
> +                0x1C: 'Socket C32',
> +                0x1D: 'Socket LGA1156',
> +                0x1E: 'Socket LGA1567',
> +                0x1F: 'Socket PGA988A',
> +                0x20: 'Socket BGA1288',
> +                0x21: 'Socket rPGA988B',
> +                0x22: 'Socket BGA1023',
> +                0x23: 'Socket BGA1224',
> +                0x24: 'Socket BGA1155',
> +                0x25: 'Socket LGA1356',
> +                0x26: 'Socket LGA2011',
> +                0x27: 'Socket FS1',
> +                0x28: 'Socket FS2',
> +                0x29: 'Socket FM1',
> +                0x2A: 'Socket FM2',
> +            }
> +            self.add_field('processor_upgrade', u.unpack_one("B"), unpack.format_table("{}", processor_upgrades))
> +            if self.length > 0x1A:
> +                self.add_field('l1_cache_handle', u.unpack_one("<H"))
> +                self.add_field('l2_cache_handle', u.unpack_one("<H"))
> +                self.add_field('l3_cache_handle', u.unpack_one("<H"))
> +            if self.length > 0x20:
> +                self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
> +                self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
> +                self.add_field('part_number', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x24:
> +                self.add_field('core_count', u.unpack_one("B"))
> +                self.add_field('core_enabled', u.unpack_one("B"))
> +                self.add_field('thread_count', u.unpack_one("B"))
> +                self.add_field('processor_characteristics', u.unpack_one("<H"))
> +            if self.length > 0x28:
> +                self.add_field('processor_family_2', u.unpack_one("<H"))
> +            if self.length > 0x2A:
> +                self.add_field('core_count2', u.unpack_one("<H"))
> +                self.add_field('core_enabled2', u.unpack_one("<H"))
> +                self.add_field('thread_count2', u.unpack_one("<H"))
> +        except:
> +            self.decode_failure = True
> +            print "Error parsing Processor Information"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class MemoryControllerInformation(SmbiosBaseStructure): #obsolete starting with v2.1
> +    smbios_structure_type = 5
> +
> +    def __init__(self, u, sm):
> +        super(MemoryControllerInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            _error_detecting_method = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'None',
> +                0x04: '8-bit Parity',
> +                0x05: '32-bit ECC',
> +                0x06: '64-bit ECC',
> +                0x07: '128-bit ECC',
> +                0x08: 'CRC'
> +                }
> +            self.add_field('error_detecting_method', u.unpack_one("B"), unpack.format_table("{}", _error_detecting_method))
> +            self.add_field('error_correcting_capability', u.unpack_one("B"))
> +            _interleaves = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'One-Way Interleave',
> +                0x04: 'Two-Way Interleave',
> +                0x05: 'Four-Way Interleave',
> +                0x06: 'Eight-Way Interleave',
> +                0x07: 'Sixteen-Way Interleave'
> +                }
> +            self.add_field('supported_interleave', u.unpack_one("B"), unpack.format_table("{}", _interleaves))
> +            self.add_field('current_interleave', u.unpack_one("B"), unpack.format_table("{}", _interleaves))
> +            self.add_field('max_memory_module_size', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('supported_speeds', u.unpack_one("<H"))
> +            self.add_field('supported_memory_types', u.unpack_one("<H"))
> +            self.add_field('memory_module_voltage', u.unpack_one("B"))
> +            self.add_field('req_voltage_b2', bitfields.getbits(self.memory_module_voltage, 2), "memory_module_voltage[2]={}")
> +            self.add_field('req_voltage_b1', bitfields.getbits(self.memory_module_voltage, 1), "memory_module_voltage[1]={}")
> +            self.add_field('req_voltage_b0', bitfields.getbits(self.memory_module_voltage, 0), "memory_module_voltage[0]={}")
> +            self.add_field('num_associated_memory_slots', u.unpack_one("B"))
> +            self.add_field('memory_module_configuration_handles', u.unpack_one("<(self.num_associated_memory_slots)H"))
> +            self.add_field('enabled_error_correcting_capabilities', u.unpack_one("B"))
> +        except:
> +            self.decode_failure = True
> +            print "Error parsing MemoryControllerInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class MemoryModuleInformation(SmbiosBaseStructure): #obsolete starting with v2.1
> +    smbios_structure_type = 6
> +
> +    def __init__(self, u, sm):
> +        super(MemoryModuleInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('socket_designation', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('bank_connections', u.unpack_one("B"))
> +            self.add_field('current_speed', u.unpack_one("B"))
> +            self.add_field('current_memory_type', u.unpack_one("<H"))
> +            _mem_connection = {
> +                0: 'single',
> +                1: 'double-bank'
> +                }
> +            self.add_field('installed_mem', u.unpack_one("B"))
> +            self.add_field('installed_size', bitfields.getbits(self.installed_mem, 6, 0), "installed_mem[6:0]={}")
> +            self.add_field('installed_memory_module_connection', bitfields.getbits(self.installed_mem, 7), unpack.format_table("installed_mem[7]={}", _mem_connection))
> +            self.add_field('enabled_mem', u.unpack_one("B"))
> +            self.add_field('enabled_size', bitfields.getbits(self.installed_mem, 6, 0), "enabled_mem[6:0]={}")
> +            self.add_field('enabled_memory_module_connection', bitfields.getbits(self.installed_mem, 7), unpack.format_table("enabled_mem[7]={}", _mem_connection))
> +            self.add_field('error_status', u.unpack_one("B"))
> +            self.add_field('error_status_info_obstained_from_event_log', bool(bitfields.getbits(self.error_status, 2)), unpack.format_table("error_status[2]={}", _mem_connection))
> +            self.add_field('correctable_errors_received', bool(bitfields.getbits(self.error_status, 1)), unpack.format_table("error_status[1]={}", _mem_connection))
> +            self.add_field('uncorrectable_errors_received', bool(bitfields.getbits(self.error_status, 0)), unpack.format_table("error_status[0]={}", _mem_connection))
> +        except:
> +            self.decode_failure = True
> +            print "Error parsing MemoryModuleInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class CacheInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 7
> +
> +    def __init__(self, u, sm):
> +        super(CacheInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('socket_designation', u.unpack_one("B"), self.fmtstr)
> +            processor_types = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'Central Processor',
> +                0x04: 'Math Processor',
> +                0x05: 'DSP Processor',
> +                0x06: 'Video Processor',
> +            }
> +            self.add_field('cache_configuration', u.unpack_one("<H"))
> +            _operational_mode = {
> +                0b00: 'Write Through',
> +                0b01: 'Write Back',
> +                0b10: 'Varies with Memory Address',
> +                0b11: 'Unknown'
> +                }
> +            self.add_field('operational_mode', bitfields.getbits(self.cache_configuration, 9, 8), unpack.format_table("cache_configuration[9:8]={}", _operational_mode))
> +            self.add_field('enabled_at_boot_time', bool(bitfields.getbits(self.cache_configuration, 7)), "cache_configuration[7]={}")
> +            _location = {
> +                0b00: 'Internal',
> +                0b01: 'External',
> +                0b10: 'Reserved',
> +                0b11: 'Unknown'
> +                }
> +            self.add_field('location_relative_to_cpu_module', bitfields.getbits(self.cache_configuration, 6, 5), unpack.format_table("cache_configuration[6:5]={}", _location))
> +            self.add_field('cache_socketed', bool(bitfields.getbits(self.cache_configuration, 3)), "cache_configuration[3]={}")
> +            self.add_field('cache_level', bitfields.getbits(self.cache_configuration, 2, 0), "cache_configuration[2:0]={}")
> +            self.add_field('max_cache_size', u.unpack_one("<H"))
> +            _granularity = {
> +                0: '1K granularity',
> +                1: '64K granularity'
> +                }
> +            self.add_field('max_granularity', bitfields.getbits(self.cache_configuration, 15), unpack.format_table("max_cache_size[15]={}", _granularity))
> +            self.add_field('max_size_in_granularity', bitfields.getbits(self.cache_configuration, 14, 0), "max_cache_size[14, 0]={}")
> +            self.add_field('installed_size', u.unpack_one("<H"))
> +            if self.installed_size != 0:
> +                self.add_field('installed_granularity', bitfields.getbits(self.cache_configuration, 15), unpack.format_table("installed_size[15]={}", _granularity))
> +                self.add_field('installed_size_in_granularity', bitfields.getbits(self.cache_configuration, 14, 0), "installed_size[14, 0]={}")
> +            self.add_field('supported_sram_type', u.unpack_one("<H"))
> +            self.add_field('current_sram_type', u.unpack_one("<H"))
> +            if self.length > 0x0F:
> +                self.add_field('cache_speed', u.unpack_one("B"))
> +            if self.length > 0x10:
> +                _error_correction = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'None',
> +                    0x04: 'Parity',
> +                    0x05: 'Single-bit ECC',
> +                    0x06: 'Multi-bit ECC'
> +                    }
> +                self.add_field('error_correction', u.unpack_one("B"), unpack.format_table("{}", _error_correction))
> +            if self.length > 0x10:
> +                _system_cache_type = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Instruction',
> +                    0x04: 'Data',
> +                    0x05: 'Unified'
> +                    }
> +                self.add_field('system_cache_type', u.unpack_one("B"), unpack.format_table("{}", _system_cache_type))
> +            if self.length > 0x12:
> +                _associativity = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Direct Mapped',
> +                    0x04: '2-way Set-Associative',
> +                    0x05: '4-way Set-Associative',
> +                    0x06: 'Fully Associative',
> +                    0x07: '8-way Set-Associative',
> +                    0x08: '16-way Set-Associative',
> +                    0x09: '12-way Set-Associative',
> +                    0x0A: '24-way Set-Associative',
> +                    0x0B: '32-way Set-Associative',
> +                    0x0C: '48-way Set-Associative',
> +                    0x0D: '64-way Set-Associative',
> +                    0x0E: '20-way Set-Associative'
> +                    }
> +                self.add_field('associativity', u.unpack_one("B"), unpack.format_table("{}", _associativity))
> +
> +        except:
> +            self.decode_failure = True
> +            print "Error parsing CacheInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class PortConnectorInfo(SmbiosBaseStructure):
> +    smbios_structure_type = 8
> +
> +    def __init__(self, u, sm):
> +        super(PortConnectorInfo, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('internal_reference_designator', u.unpack_one("B"), self.fmtstr)
> +            connector_types = {
> +                0x00: 'None',
> +                0x01: 'Centronics',
> +                0x02: 'Mini Centronics',
> +                0x03: 'Proprietary',
> +                0x04: 'DB-25 pin male',
> +                0x05: 'DB-25 pin female',
> +                0x06: 'DB-15 pin male',
> +                0x07: 'DB-15 pin female',
> +                0x08: 'DB-9 pin male',
> +                0x09: 'DB-9 pin female',
> +                0x0A: 'RJ-11',
> +                0x0B: 'RJ-45',
> +                0x0C: '50-pin MiniSCSI',
> +                0x0D: 'Mini-DIN',
> +                0x0E: 'Micro-DIN',
> +                0x0F: 'PS/2',
> +                0x10: 'Infrared',
> +                0x11: 'HP-HIL',
> +                0x12: 'Access Bus (USB)',
> +                0x13: 'SSA SCSI',
> +                0x14: 'Circular DIN-8 male',
> +                0x15: 'Circular DIN-8 female',
> +                0x16: 'On Board IDE',
> +                0x17: 'On Board Floppy',
> +                0x18: '9-pin Dual Inline (pin 10 cut)',
> +                0x19: '25-pin Dual Inline (pin 26 cut)',
> +                0x1A: '50-pin Dual Inline',
> +                0x1B: '68-pin Dual Inline',
> +                0x1C: 'On Board Sound Input from CD-ROM',
> +                0x1D: 'Mini-Centronics Type-14',
> +                0x1E: 'Mini-Centronics Type-26',
> +                0x1F: 'Mini-jack (headphones)',
> +                0x20: 'BNC',
> +                0x21: '1394',
> +                0x22: 'SAS/SATA Plug Receptacle',
> +                0xA0: 'PC-98',
> +                0xA1: 'PC-98Hireso',
> +                0xA2: 'PC-H98',
> +                0xA3: 'PC-98Note',
> +                0xA4: 'PC-98Full',
> +                0xFF: 'Other',
> +            }
> +            self.add_field('internal_connector_type', u.unpack_one("B"), unpack.format_table("{}", connector_types))
> +            self.add_field('external_reference_designator', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('external_connector_type', u.unpack_one("B"), unpack.format_table("{}", connector_types))
> +            port_types = {
> +                0x00: 'None',
> +                0x01: 'Parallel Port XT/AT Compatible',
> +                0x02: 'Parallel Port PS/2',
> +                0x03: 'Parallel Port ECP',
> +                0x04: 'Parallel Port EPP',
> +                0x05: 'Parallel Port ECP/EPP',
> +                0x06: 'Serial Port XT/AT Compatible',
> +                0x07: 'Serial Port 16450 Compatible',
> +                0x08: 'Serial Port 16550 Compatible',
> +                0x09: 'Serial Port 16550A Compatible',
> +                0x0A: 'SCSI Port',
> +                0x0B: 'MIDI Port',
> +                0x0C: 'Joy Stick Port',
> +                0x0D: 'Keyboard Port',
> +                0x0E: 'Mouse Port',
> +                0x0F: 'SSA SCSI',
> +                0x10: 'USB',
> +                0x11: 'FireWire (IEEE P1394)',
> +                0x12: 'PCMCIA Type I2',
> +                0x13: 'PCMCIA Type II',
> +                0x14: 'PCMCIA Type III',
> +                0x15: 'Cardbus',
> +                0x16: 'Access Bus Port',
> +                0x17: 'SCSI II',
> +                0x18: 'SCSI Wide',
> +                0x19: 'PC-98',
> +                0x1A: 'PC-98-Hireso',
> +                0x1B: 'PC-H98',
> +                0x1C: 'Video Port',
> +                0x1D: 'Audio Port',
> +                0x1E: 'Modem Port',
> +                0x1F: 'Network Port',
> +                0x20: 'SATA',
> +                0x21: 'SAS',
> +                0xA0: '8251 Compatible',
> +                0xA1: '8251 FIFO Compatible',
> +                0xFF: 'Other',
> +            }
> +            self.add_field('port_type', u.unpack_one("B"), unpack.format_table("{}", port_types))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing PortConnectorInfo"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class SystemSlots(SmbiosBaseStructure):
> +    smbios_structure_type = 9
> +
> +    def __init__(self, u, sm):
> +        super(SystemSlots, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('designation', u.unpack_one("B"), self.fmtstr)
> +            _slot_types = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'ISA',
> +                0x04: 'MCA',
> +                0x05: 'EISA',
> +                0x06: 'PCI',
> +                0x07: 'PC Card (PCMCIA)',
> +                0x08: 'VL-VESA',
> +                0x09: 'Proprietary',
> +                0x0A: 'Processor Card Slot',
> +                0x0B: 'Proprietary Memory Card Slot',
> +                0x0C: 'I/O Riser Card Slot',
> +                0x0D: 'NuBus',
> +                0x0E: 'PCI 66MHz Capable',
> +                0x0F: 'AGP',
> +                0x10: 'AGP 2X',
> +                0x11: 'AGP 4X',
> +                0x12: 'PCI-X',
> +                0x13: 'AGP 8X',
> +                0xA0: 'PC-98/C20',
> +                0xA1: 'PC-98/C24',
> +                0xA2: 'PC-98/E',
> +                0xA3: 'PC-98/Local Bus',
> +                0xA4: 'PC-98/Card',
> +                0xA5: 'PCI Express',
> +                0xA6: 'PCI Express x1',
> +                0xA7: 'PCI Express x2',
> +                0xA8: 'PCI Express x4',
> +                0xA9: 'PCI Express x8',
> +                0xAA: 'PCI Express x16',
> +                0xAB: 'PCI Express Gen 2',
> +                0xAC: 'PCI Express Gen 2 x1',
> +                0xAD: 'PCI Express Gen 2 x2',
> +                0xAE: 'PCI Express Gen 2 x4',
> +                0xAF: 'PCI Express Gen 2 x8',
> +                0xB0: 'PCI Express Gen 2 x16',
> +                0xB1: 'PCI Express Gen 3',
> +                0xB2: 'PCI Express Gen 3 x1',
> +                0xB3: 'PCI Express Gen 3 x2',
> +                0xB4: 'PCI Express Gen 3 x4',
> +                0xB5: 'PCI Express Gen 3 x8',
> +                0xB6: 'PCI Express Gen 3 x16',
> +            }
> +            self.add_field('slot_type', u.unpack_one("B"), unpack.format_table("{}", _slot_types))
> +            _slot_data_bus_widths = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: '8 bit',
> +                0x04: '16 bit',
> +                0x05: '32 bit',
> +                0x06: '64 bit',
> +                0x07: '128 bit',
> +                0x08: '1x or x1',
> +                0x09: '2x or x2',
> +                0x0A: '4x or x4',
> +                0x0B: '8x or x8',
> +                0x0C: '12x or x12',
> +                0x0D: '16x or x16',
> +                0x0E: '32x or x32',
> +            }
> +            self.add_field('slot_data_bus_width', u.unpack_one('B'), unpack.format_table("{}", _slot_data_bus_widths))
> +            _current_usages = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'Available',
> +                0x04: 'In use',
> +            }
> +            self.add_field('current_usage', u.unpack_one('B'), unpack.format_table("{}", _current_usages))
> +            _slot_lengths = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'Short Length',
> +                0x04: 'Long Length',
> +            }
> +            self.add_field('slot_length', u.unpack_one('B'), unpack.format_table("{}", _slot_lengths))
> +            self.add_field('slot_id', u.unpack_one('<H'))
> +            self.add_field('characteristics1', u.unpack_one('B'))
> +            self.add_field('characteristics_unknown', bool(bitfields.getbits(self.characteristics1, 0)), "characteristics1[0]={}")
> +            self.add_field('provides_5_0_volts', bool(bitfields.getbits(self.characteristics1, 1)), "characteristics1[1]={}")
> +            self.add_field('provides_3_3_volts', bool(bitfields.getbits(self.characteristics1, 2)), "characteristics1[2]={}")
> +            self.add_field('shared_slot', bool(bitfields.getbits(self.characteristics1, 3)), "characteristics1[3]={}")
> +            self.add_field('supports_pc_card_16', bool(bitfields.getbits(self.characteristics1, 4)), "characteristics1[4]={}")
> +            self.add_field('supports_cardbus', bool(bitfields.getbits(self.characteristics1, 5)), "characteristics1[5]={}")
> +            self.add_field('supports_zoom_video', bool(bitfields.getbits(self.characteristics1, 6)), "characteristics1[6]={}")
> +            self.add_field('supports_modem_ring_resume', bool(bitfields.getbits(self.characteristics1, 7)), "characteristics1[7]={}")
> +            if self.length > 0x0C:
> +                self.add_field('characteristics2', u.unpack_one('B'))
> +                self.add_field('supports_PME', bool(bitfields.getbits(self.characteristics2, 0)), "characteristics2[0]={}")
> +                self.add_field('supports_hot_plug', bool(bitfields.getbits(self.characteristics2, 1)), "characteristics2[1]={}")
> +                self.add_field('supports_smbus', bool(bitfields.getbits(self.characteristics2, 2)), "characteristics2[2]={}")
> +            if self.length > 0x0D:
> +                self.add_field('segment_group_number', u.unpack_one('<H'))
> +                self.add_field('bus_number', u.unpack_one('B'))
> +                self.add_field('device_function_number', u.unpack_one('B'))
> +                self.add_field('device_number', bitfields.getbits(self.device_function_number, 7, 3), "device_function_number[7:3]={}")
> +                self.add_field('function_number', bitfields.getbits(self.device_function_number, 2, 0), "device_function_number[2:0]={}")
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing SystemSlots"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class OnBoardDevicesInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 10
> +
> +    def __init__(self, u, sm):
> +        super(OnBoardDevicesInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('device_type', u.unpack_one("B"))
> +            self.add_field('device_enabled', bool(bitfields.getbits(self.device_type, 7)), "device_type[7]={}")
> +            _device_types = {
> +                0x01: 'Other',
> +                0x02: 'Unknown',
> +                0x03: 'Video',
> +                0x04: 'SCSI Controller',
> +                0x05: 'Ethernet',
> +                0x06: 'Token Ring',
> +                0x07: 'Sound',
> +                0x08: 'PATA Controller',
> +                0x09: 'SATA Controller',
> +                0x0A: 'SAS Controller'
> +            }
> +            self.add_field('type_of_device', bitfields.getbits(self.device_type, 6, 0), unpack.format_table("device_type[6:0]={}", _device_types))
> +            self.add_field('description_string', u.unpack_one("B"), self.fmtstr)
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing OnBoardDevicesInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class OEMStrings(SmbiosBaseStructure):
> +    smbios_structure_type = 11
> +
> +    def __init__(self, u, sm):
> +        super(OEMStrings, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('count', u.unpack_one("B"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing OEMStrings"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class SystemConfigOptions(SmbiosBaseStructure):
> +    smbios_structure_type = 12
> +
> +    def __init__(self, u, sm):
> +        super(SystemConfigOptions, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('count', u.unpack_one("B"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing SystemConfigOptions"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class BIOSLanguageInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 13
> +
> +    def __init__(self, u, sm):
> +        super(BIOSLanguageInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('installable_languages', u.unpack_one("B"))
> +            if self.length > 0x05:
> +                self.add_field('flags', u.unpack_one('B'))
> +                self.add_field('abbreviated_format', bool(bitfields.getbits(self.flags, 0)), "flags[0]={}")
> +            if self.length > 0x6:
> +                u.skip(15)
> +                self.add_field('current_language', u.unpack_one('B'), self.fmtstr)
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing BIOSLanguageInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class GroupAssociations(SmbiosBaseStructure):
> +    smbios_structure_type = 14
> +
> +    def __init__(self, u, sm):
> +        super(GroupAssociations, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('group_name', u.unpack_one("B"), self.fmtstr)
> +            self.add_field('item_type', u.unpack_one('B'))
> +            self.add_field('item_handle', u.unpack_one('<H'))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing GroupAssociations"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class SystemEventLog(SmbiosBaseStructure):
> +    smbios_structure_type = 15
> +
> +    def __init__(self, u, sm):
> +        super(SystemEventLog, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            self.add_field('log_area_length', u.unpack_one("<H"))
> +            self.add_field('log_header_start_offset', u.unpack_one('<H'))
> +            self.add_field('log_data_start_offset', u.unpack_one('<H'))
> +            _access_method = {
> +                0x00: 'Indexed I/O: 1 8-bit index port, 1 8-bit data port',
> +                0x01: 'Indexed I/O: 2 8-bit index ports, 1 8-bit data port',
> +                0x02: 'Indexed I/O: 1 16-bit index port, 1 8-bit data port',
> +                0x03: 'Memory-mapped physical 32-bit address',
> +                0x04: 'Available through General-Purpose NonVolatile Data functions',
> +                xrange(0x05, 0x07F): 'Available for future assignment',
> +                xrange(0x80, 0xFF): 'BIOS Vendor/OEM-specific'
> +                }
> +            self.add_field('access_method', u.unpack_one('B'), unpack.format_table("{}", _access_method))
> +            self.add_field('log_status', u.unpack_one('B'))
> +            self.add_field('log_area_full', bool(bitfields.getbits(self.log_status, 1)), "log_status[1]={}")
> +            self.add_field('log_area_valid', bool(bitfields.getbits(self.log_status, 0)), "log_status[0]={}")
> +            self.add_field('log_change_token', u.unpack_one('<I'))
> +            self.add_field('access_method_address', u.unpack_one('<I'))
> +            if self.length > 0x14:
> +                _log_header_formats = {
> +                    0: 'No header',
> +                    1: 'Type 1 log header',
> +                    xrange(2, 0x7f): 'Available for future assignment',
> +                    xrange(0x80, 0xff): 'BIOS vendor or OEM-specific format'
> +                    }
> +                self.add_field('log_header_format', u.unpack_one("B"), unpack.format_table("{}", _log_header_formats))
> +            if self.length > 0x15:
> +                self.add_field('num_supported_log_type_descriptors', u.unpack_one('B'))
> +            if self.length > 0x16:
> +                self.add_field('length_log_type_descriptor', u.unpack_one('B'))
> +            if self.length != (0x17 + (self.num_supported_log_type_descriptors * self.length_log_type_descriptor)):
> +                print "Error: structure length ({}) != 0x17 + (num_supported_log_type_descriptors ({}) * length_log_type_descriptor({}))".format(self.length, self.num_supported_log_type_descriptors, self.length_log_type_descriptor)
> +                print "structure length = {}".format(self.length)
> +                print "num_supported_log_type_descriptors = {}".format(self.num_supported_log_type_descriptors)
> +                print "length_log_type_descriptor = {}".format(self.length_log_type_descriptor)
> +                self.decodeFailure = True
> +            self.add_field('descriptors', tuple(EventLogDescriptor.unpack(u) for i in range(self.num_supported_log_type_descriptors)), unpack.format_each("\n{!r}"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing SystemEventLog"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class EventLogDescriptor(unpack.Struct):
> +    @staticmethod
> +    def _unpack(u):
> +        _event_log_type_descriptors = {
> +            0x00: 'Reserved',
> +            0x01: 'Single-bit ECC memory error',
> +            0x02: 'Multi-bit ECC memory error',
> +            0x03: 'Parity memory error',
> +            0x04: 'Bus time-out',
> +            0x05: 'I/O Channel Check',
> +            0x06: 'Software NMI',
> +            0x07: 'POST Memory Resize',
> +            0x08: 'POST Error',
> +            0x09: 'PCI Parity Error',
> +            0x0A: 'PCI System Error',
> +            0x0B: 'CPU Failure',
> +            0x0C: 'EISA FailSafe Timer time-out',
> +            0x0D: 'Correctable memory log disabled',
> +            0x0E: 'Logging disabled for a specific Event Type - too many errors of the same type received in a short amount of time',
> +            0x0F: 'Reserved',
> +            0x10: 'System Limit Exceeded',
> +            0x11: 'Asynchronous hardware timer expired and issued a system reset',
> +            0x12: 'System configuration information',
> +            0x13: 'Hard-disk information',
> +            0x14: 'System reconfigured',
> +            0x15: 'Uncorrectable CPU-complex error',
> +            0x16: 'Log Area Reset/Cleared',
> +            0x17: 'System boot',
> +            xrange(0x18, 0x7F): 'Unused, available for assignment',
> +            xrange(0x80, 0xFE): 'Availalbe for system- and OEM-specific assignments',
> +            0xFF: 'End of log'
> +        }
> +        yield 'log_type', u.unpack_one('B'), unpack.format_table("{}", _event_log_type_descriptors)
> +        _event_log_format = {
> +            0x00: 'None',
> +            0x01: 'Handle',
> +            0x02: 'Multiple-Event',
> +            0x03: 'Multiple-Event Handle',
> +            0x04: 'POST Results Bitmap',
> +            0x05: 'System Management Type',
> +            0x06: 'Multiple-Event System Management Type',
> +            xrange(0x80, 0xFF): 'OEM assigned'
> +        }
> +        yield 'variable_data_format_type', u.unpack_one('B'), unpack.format_table("{}", _event_log_format)
> +
> +class PhysicalMemoryArray(SmbiosBaseStructure):
> +    smbios_structure_type = 16
> +
> +    def __init__(self, u, sm):
> +        super(PhysicalMemoryArray, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                _location_field = {
> +                    0x01: "Other",
> +                    0x02: "Unknown",
> +                    0x03: "System board or motherboard",
> +                    0x04: "ISA add-on card",
> +                    0x05: "EISA add-on card",
> +                    0x06: "PCI add-on card",
> +                    0x07: "MCA add-on card",
> +                    0x08: "PCMCIA add-on card",
> +                    0x09: "Proprietary add-on card",
> +                    0x0A: "NuBus",
> +                    0xA0: "PC-98/C20 add-on card",
> +                    0xA1: "PC-98/C24 add-on card",
> +                    0xA2: "PC-98/E add-on card",
> +                    0xA3: "PC-98/Local bus add-on card"
> +                    }
> +                self.add_field('location', u.unpack_one("B"), unpack.format_table("{}", _location_field))
> +            if self.length > 0x05:
> +                _use = {
> +                    0x01: "Other",
> +                    0x02: "Unknown",
> +                    0x03: "System memory",
> +                    0x04: "Video memory",
> +                    0x05: "Flash memory",
> +                    0x06: "Non-volatile RAM",
> +                    0x07: "Cache memory"
> +                    }
> +                self.add_field('use', u.unpack_one('B'), unpack.format_table("{}", _use))
> +            if self.length > 0x06:
> +                _error_correction = {
> +                    0x01: "Other",
> +                    0x02: "Unknown",
> +                    0x03: "None",
> +                    0x04: "Parity",
> +                    0x05: "Single-bit ECC",
> +                    0x06: "Multi-bit ECC",
> +                    0x07: "CRC"
> +                    }
> +                self.add_field('memory_error_correction', u.unpack_one('B'), unpack.format_table("{}", _error_correction))
> +            if self.length > 0x07:
> +                self.add_field('maximum_capacity', u.unpack_one('<I'))
> +            if self.length > 0x0B:
> +                self.add_field('memory_error_information_handle', u.unpack_one('<H'))
> +            if self.length > 0x0D:
> +                self.add_field('num_memory_devices', u.unpack_one('<H'))
> +            if self.length > 0x0F:
> +                self.add_field('extended_maximum_capacity', u.unpack_one('<Q'))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing PhysicalMemoryArray"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class MemoryDevice(SmbiosBaseStructure):
> +    smbios_structure_type = 17
> +
> +    def __init__(self, u, sm):
> +        super(MemoryDevice, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('physical_memory_array_handle', u.unpack_one("<H"))
> +            if self.length > 0x6:
> +                self.add_field('memory_error_information_handle', u.unpack_one("<H"))
> +            if self.length > 0x8:
> +                self.add_field('total_width', u.unpack_one("<H"))
> +            if self.length > 0xA:
> +                self.add_field('data_width', u.unpack_one("<H"))
> +            if self.length > 0xC:
> +                self.add_field('size', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +                _form_factors = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'SIMM',
> +                    0x04: 'SIP',
> +                    0x05: 'Chip',
> +                    0x06: 'DIP',
> +                    0x07: 'ZIP',
> +                    0x08: 'Proprietary Card',
> +                    0x09: 'DIMM',
> +                    0x0A: 'TSOP',
> +                    0x0B: 'Row of chips',
> +                    0x0C: 'RIMM',
> +                    0x0D: 'SODIMM',
> +                    0x0E: 'SRIMM',
> +                    0x0F: 'FB-DIMM'
> +                    }
> +                self.add_field('form_factor', u.unpack_one("B"), unpack.format_table("{}", _form_factors))
> +            if self.length > 0xF:
> +                self.add_field('device_set', u.unpack_one("B"))
> +            if self.length > 0x10:
> +                self.add_field('device_locator', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x11:
> +                self.add_field('bank_locator', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x12:
> +                _memory_types = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'DRAM',
> +                    0x04: 'EDRAM',
> +                    0x05: 'VRAM',
> +                    0x06: 'SRAM',
> +                    0x07: 'RAM',
> +                    0x08: 'ROM',
> +                    0x09: 'FLASH',
> +                    0x0A: 'EEPROM',
> +                    0x0B: 'FEPROM',
> +                    0x0C: 'EPROM',
> +                    0x0D: 'CDRAM',
> +                    0x0E: '3DRAM',
> +                    0x0F: 'SDRAM',
> +                    0x10: 'SGRAM',
> +                    0x11: 'RDRAM',
> +                    0x12: 'DDR',
> +                    0x13: 'DDR2',
> +                    0x14: 'DDR2 FB-DIMM',
> +                    xrange(0x15, 0x17): 'Reserved',
> +                    0x18: 'DDR3',
> +                    0x19: 'FBD2'
> +                    }
> +                self.add_field('memory_type', u.unpack_one("B"), unpack.format_table("{}", _memory_types))
> +            if self.length > 0x13:
> +                self.add_field('type_detail', u.unpack_one('<H'))
> +            if self.length > 0x15:
> +                self.add_field('speed', u.unpack_one("<H"))
> +            if self.length > 0x17:
> +                self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x18:
> +                self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x19:
> +                self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x1A:
> +                self.add_field('part_number', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x1B:
> +                self.add_field('attributes', u.unpack_one("B"))
> +                self.add_field('rank', bitfields.getbits(self.attributes, 3, 0), "attributes[3:0]={}")
> +            if self.length > 0x1C:
> +                if self.size == 0x7FFF:
> +                    self.add_field('extended_size', u.unpack_one('<I'))
> +                    self.add_field('mem_size', bitfields.getbits(self.type_detail, 30, 0), "type_detail[30:0]={}")
> +                else:
> +                    u.skip(4)
> +            if self.length > 0x20:
> +                self.add_field('configured_memory_clock_speed', u.unpack_one("<H"))
> +            if self.length > 0x22:
> +                self.add_field('minimum_voltage', u.unpack_one("<H"))
> +            if self.length > 0x24:
> +                self.add_field('maximum_voltage', u.unpack_one("<H"))
> +            if self.length > 0x26:
> +                self.add_field('configured_voltage', u.unpack_one("<H"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing MemoryDevice"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class MemoryErrorInfo32Bit(SmbiosBaseStructure):
> +    smbios_structure_type = 18
> +
> +    def __init__(self, u, sm):
> +        super(MemoryErrorInfo32Bit, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                _error_types = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'OK',
> +                    0x04: 'Bad read',
> +                    0x05: 'Parity error',
> +                    0x06: 'Single-bit error',
> +                    0x07: 'Double-bit error',
> +                    0x08: 'Multi-bit error',
> +                    0x09: 'Nibble error',
> +                    0x0A: 'Checksum error',
> +                    0x0B: 'CRC error',
> +                    0x0C: 'Corrected single-bit error',
> +                    0x0D: 'Corrected error',
> +                    0x0E: 'Uncorrectable error'
> +                    }
> +                self.add_field('error_type', u.unpack_one("B"), unpack.format_table("{}", _error_types))
> +            if self.length > 0x5:
> +                 _error_granularity_field = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Device level',
> +                    0x04: 'Memory partition level'
> +                    }
> +                 self.add_field('error_granularity', u.unpack_one("B"), unpack.format_table("{}", _error_granularity_field))
> +            if self.length > 0x6:
> +                _error_operation_field = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Read',
> +                    0x04: 'Write',
> +                    0x05: 'Partial write'
> +                    }
> +                self.add_field('error_operation', u.unpack_one("B"), unpack.format_table("{}", _error_operation_field))
> +            if self.length > 0x7:
> +                self.add_field('vendor_syndrome', u.unpack_one("<I"))
> +            if self.length > 0xB:
> +                self.add_field('memory_array_error_address', u.unpack_one("<I"))
> +            if self.length > 0xF:
> +                self.add_field('device_error_address', u.unpack_one("<I"))
> +            if self.length > 0x13:
> +                self.add_field('error_resolution', u.unpack_one("<I"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing MemoryErrorInfo32Bit"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class MemoryArrayMappedAddress(SmbiosBaseStructure):
> +    smbios_structure_type = 19
> +
> +    def __init__(self, u, sm):
> +        super(MemoryArrayMappedAddress, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('starting_address', u.unpack_one("<I"))
> +                # if FFFF FFFF: address stored in Extended Starting Address
> +            if self.length > 0x8:
> +                self.add_field('ending_address', u.unpack_one("<I"))
> +            if self.length > 0xC:
> +                self.add_field('memory_array_handle', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +                self.add_field('partition_width', u.unpack_one("B"))
> +            if self.length > 0xF:
> +                # valid if starting_address = FFFF FFFF
> +                if self.starting_address == 0xFFFFFFFF:
> +                    self.add_field('extended_starting_address', u.unpack_one("<Q"))
> +                    if self.length > 0x17:
> +                        self.add_field('extended_ending_address', u.unpack_one("<Q"))
> +                else:
> +                    u.skip(16)
> +
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing MemoryArrayMappedAddress"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class MemoryDeviceMappedAddress(SmbiosBaseStructure):
> +    smbios_structure_type = 20
> +
> +    def __init__(self, u, sm):
> +        super(MemoryDeviceMappedAddress, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('starting_address', u.unpack_one("<I"))
> +                # if FFFF FFFF: address stored in Extended Starting Address
> +            if self.length > 0x8:
> +                self.add_field('ending_address', u.unpack_one("<I"))
> +            if self.length > 0xC:
> +                self.add_field('memory_device_handle', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +                self.add_field('memory_array_mapped_address_handle', u.unpack_one("<H"))
> +            if self.length > 0x10:
> +                self.add_field('partition_row_position', u.unpack_one("B"))
> +            if self.length > 0x11:
> +                self.add_field('interleave_position', u.unpack_one("B"))
> +            if self.length > 0x12:
> +                self.add_field('interleave_data_depth', u.unpack_one("B"))
> +            if self.length > 0x13:
> +                # valid if starting_address = FFFF FFFF
> +                if self.starting_address == 0xFFFFFFFF:
> +                    self.add_field('extended_starting_address', u.unpack_one("<Q"))
> +                    if self.length > 0x1B:
> +                        self.add_field('extended_ending_address', u.unpack_one("<Q"))
> +                else:
> +                    u.skip(16)
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing MemoryDeviceMappedAddress"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class BuiltInPointingDevice(SmbiosBaseStructure):
> +    smbios_structure_type = 21
> +
> +    def __init__(self, u, sm):
> +        super(BuiltInPointingDevice, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                _pointing_device_types = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Mouse',
> +                    0x04: 'Track Ball',
> +                    0x05: 'Track Point',
> +                    0x06: 'Glide Point',
> +                    0x07: 'Touch Pad',
> +                    0x08: 'Touch Screen',
> +                    0x09: 'Optical Sensor'
> +                    }
> +                self.add_field('pointing_device_type', u.unpack_one("B"), unpack.format_table("{}", _pointing_device_types))
> +            if self.length > 0x5:
> +                _interfaces = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Serial',
> +                    0x04: 'PS/2',
> +                    0x05: 'Infared',
> +                    0x06: 'HP-HIL',
> +                    0x07: 'Bus mouse',
> +                    0x08: 'ADB (Apple Desktop Bus)',
> +                    0x09: 'Bus mouse DB-9',
> +                    0x0A: 'Bus mouse micro-DIN',
> +                    0x0B: 'USB'
> +                    }
> +                self.add_field('interface', u.unpack_one("B"), unpack.format_table("{}", _interfaces))
> +            if self.length > 0x6:
> +                self.add_field('num_buttons', u.unpack_one("B"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing BuiltInPointingDevice"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class PortableBattery(SmbiosBaseStructure):
> +    smbios_structure_type = 22
> +
> +    def __init__(self, u, sm):
> +        super(PortableBattery, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('location', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x5:
> +                self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x6:
> +                self.add_field('manufacturer_date', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x7:
> +                self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x8:
> +                self.add_field('device_name', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x9:
> +                _device_chemistry = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Lead Acid',
> +                    0x04: 'Nickel Cadmium',
> +                    0x05: 'Nickel metal hydride',
> +                    0x06: 'Lithium-ion',
> +                    0x07: 'Zinc air',
> +                    0x08: 'Lithium Polymer'
> +                    }
> +                self.add_field('device_chemistry', u.unpack_one("B"), unpack.format_table("{}", _device_chemistry))
> +            if self.length > 0xA:
> +                self.add_field('design_capacity', u.unpack_one("<H"))
> +            if self.length > 0xC:
> +                self.add_field('design_voltage', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +                self.add_field('sbds_version_number', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0xF:
> +                self.add_field('max_error_battery_data', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x10:
> +                if self.serial_number == 0:
> +                    self.add_field('sbds_serial_number', u.unpack_one("<H"))
> +                else:
> +                    u.skip(2)
> +            if self.length > 0x12:
> +                if self.manufacturer_date == 0:
> +                    self.add_field('sbds_manufacture_date', u.unpack_one("<H"))
> +                    self.add_field('year_biased_by_1980', bitfields.getbits(self.sbds_manufacture_date, 15, 9), "sbds_manufacture_date[15:9]={}")
> +                    self.add_field('month', bitfields.getbits(self.sbds_manufacture_date, 8, 5), "sbds_manufacture_date[8:5]={}")
> +                    self.add_field('date', bitfields.getbits(self.sbds_manufacture_date, 4, 0), "sbds_manufacture_date[4:0]={}")
> +                else:
> +                    u.skip(2)
> +            if self.length > 0x14:
> +                if self.device_chemistry == 0x02:
> +                    self.add_field('sbds_device_chemistry', u.unpack_one("B"), self.fmtstr)
> +                else:
> +                    u.skip(1)
> +            if self.length > 0x15:
> +                self.add_field('design_capacity_multiplier', u.unpack_one("B"))
> +            if self.length > 0x16:
> +                self.add_field('oem_specific', u.unpack_one("<I"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing PortableBattery"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class SystemReset(SmbiosBaseStructure):
> +    smbios_structure_type = 23
> +
> +    def __init__(self, u, sm):
> +        super(SystemReset, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('capabilities', u.unpack_one("B"))
> +                self.add_field('contains_watchdog_timer', bool(bitfields.getbits(self.capabilities, 5)), "capabilities[5]={}")
> +                _boot_option = {
> +                    0b00: 'Reserved, do not use',
> +                    0b01: 'Operating System',
> +                    0b10: 'System utilities',
> +                    0b11: 'Do not reboot'
> +                    }
> +                self.add_field('boot_option_on_limit', bitfields.getbits(self.capabilities, 4, 3), unpack.format_table("capabilities[4:3]={}", _boot_option))
> +                self.add_field('boot_option_after_watchdog_reset', bitfields.getbits(self.capabilities, 2, 1), unpack.format_table("capabilities[2:1]={}", _boot_option))
> +                self.add_field('system_reset_enabled_by_user', bool(bitfields.getbits(self.capabilities, 0)), "capabilities[0]={}")
> +            if self.length > 0x5:
> +                self.add_field('reset_count', u.unpack_one("<H"))
> +            if self.length > 0x5:
> +                self.add_field('reset_limit', u.unpack_one("<H"))
> +            if self.length > 0x9:
> +                self.add_field('timer_interval', u.unpack_one("<H"))
> +            if self.length > 0xB:
> +                self.add_field('timeout', u.unpack_one("<H"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing SystemReset"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class HardwareSecurity(SmbiosBaseStructure):
> +    smbios_structure_type = 24
> +
> +    def __init__(self, u, sm):
> +        super(HardwareSecurity, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('hardware_security_settings', u.unpack_one("B"))
> +                _status = {
> +                    0x00: 'Disabled',
> +                    0x01: 'Enabled',
> +                    0x02: 'Not Implemented',
> +                    0x03: 'Unknown'
> +                    }
> +                self.add_field('power_on_password_status', bitfields.getbits(self.hardware_security_settings, 7, 6), unpack.format_table("hardware_security_settings[7:6]={}", _status))
> +                self.add_field('keyboard_password_status', bitfields.getbits(self.hardware_security_settings, 5, 4), unpack.format_table("hardware_security_settings[5:4]={}", _status))
> +                self.add_field('admin_password_status', bitfields.getbits(self.hardware_security_settings, 3, 2), unpack.format_table("hardware_security_settings0[3:2]={}", _status))
> +                self.add_field('front_panel_reset_status', bitfields.getbits(self.hardware_security_settings, 1, 0), unpack.format_table("hardware_security_settings[1:0]={}", _status))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing HardwareSecurity"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class SystemPowerControls(SmbiosBaseStructure):
> +    smbios_structure_type = 25
> +
> +    def __init__(self, u, sm):
> +        super(SystemPowerControls, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('next_scheduled_poweron_month', u.unpack_one("B"))
> +                self.add_field('next_scheduled_poweron_day_of_month', u.unpack_one("B"))
> +                self.add_field('next_scheduled_poweron_hour', u.unpack_one("B"))
> +                self.add_field('next_scheduled_poweron_minute', u.unpack_one("B"))
> +                self.add_field('next_scheduled_poweron_second', u.unpack_one("B"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing SystemPowerControls"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class VoltageProbe(SmbiosBaseStructure):
> +    smbios_structure_type = 26
> +
> +    def __init__(self, u, sm):
> +        super(VoltageProbe, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('description', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x5:
> +                self.add_field('location_and_status', u.unpack_one("B"))
> +                _status = {
> +                    0b001: 'Other',
> +                    0b010: 'Unknown',
> +                    0b011: 'OK',
> +                    0b100: 'Non-critical',
> +                    0b101: 'Critical',
> +                    0b110: 'Non-recoverable'
> +                    }
> +                _location = {
> +                    0b00001: 'Other',
> +                    0b00010: 'Unknown',
> +                    0b00011: 'Processor',
> +                    0b00100: 'Disk',
> +                    0b00101: 'Peripheral Bay',
> +                    0b00110: 'System Management Module',
> +                    0b00111: 'Motherboard',
> +                    0b01000: 'Memory Module',
> +                    0b01001: 'Processor Module',
> +                    0b01010: 'Power Unit',
> +                    0b01011: 'Add-in Card'
> +                    }
> +                self.add_field('status', bitfields.getbits(self.location_and_status, 7, 5), unpack.format_table("location_and_status[7:5]={}", _status))
> +                self.add_field('location', bitfields.getbits(self.location_and_status, 4, 0), unpack.format_table("location_and_status[4:0]={}", _location))
> +            if self.length > 0x6:
> +                self.add_field('max_value', u.unpack_one("<H"))
> +            if self.length > 0x8:
> +                self.add_field('min_value', u.unpack_one("<H"))
> +            if self.length > 0xA:
> +                self.add_field('resolution', u.unpack_one("<H"))
> +            if self.length > 0xC:
> +                self.add_field('tolerance', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +                self.add_field('accuracy', u.unpack_one("<H"))
> +            if self.length > 0x10:
> +                self.add_field('oem_defined', u.unpack_one("<I"))
> +            if self.length > 0x14:
> +                self.add_field('nominal_value', u.unpack_one("<H"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing VoltageProbe"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class CoolingDevice(SmbiosBaseStructure):
> +    smbios_structure_type = 27
> +
> +    def __init__(self, u, sm):
> +        super(CoolingDevice, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('temperature_probe_handle', u.unpack_one("<H"))
> +            if self.length > 0x6:
> +                self.add_field('device_type_and_status', u.unpack_one("B"))
> +                _status = {
> +                    0b001: 'Other',
> +                    0b010: 'Unknown',
> +                    0b011: 'OK',
> +                    0b100: 'Non-critical',
> +                    0b101: 'Critical',
> +                    0b110: 'Non-recoverable'
> +                    }
> +                _type = {
> +                    0b00001: 'Other',
> +                    0b00010: 'Unknown',
> +                    0b00011: 'Fan',
> +                    0b00100: 'Centrifugal Blower',
> +                    0b00101: 'Chip Fan',
> +                    0b00110: 'Cabinet Fan',
> +                    0b00111: 'Power Supply Fan',
> +                    0b01000: 'Heat Pipe',
> +                    0b01001: 'Integrated Refrigeration',
> +                    0b10000: 'Active Cooling',
> +                    0b10001: 'Passive Cooling'
> +                    }
> +                self.add_field('status', bitfields.getbits(self.device_type_and_status, 7, 5), unpack.format_table("device_type_and_status[7:5]={}", _status))
> +                self.add_field('device_type', bitfields.getbits(self.device_type_and_status, 4, 0), unpack.format_table("device_type_and_status[4:0]={}", _type))
> +            if self.length > 0x7:
> +                self.add_field('cooling_unit_group', u.unpack_one("B"))
> +            if self.length > 0x8:
> +                self.add_field('OEM_defined', u.unpack_one("<I"))
> +            if self.length > 0xC:
> +                self.add_field('nominal_speed', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +               self.add_field('description', u.unpack_one("B"), self.fmtstr)
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing CoolingDevice"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class TemperatureProbe(SmbiosBaseStructure):
> +    smbios_structure_type = 28
> +
> +    def __init__(self, u, sm):
> +        super(TemperatureProbe, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('description', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x5:
> +                self.add_field('location_and_status', u.unpack_one("B"))
> +                _status = {
> +                    0b001: 'Other',
> +                    0b010: 'Unknown',
> +                    0b011: 'OK',
> +                    0b100: 'Non-critical',
> +                    0b101: 'Critical',
> +                    0b110: 'Non-recoverable'
> +                    }
> +                _location = {
> +                    0b00001: 'Other',
> +                    0b00010: 'Unknown',
> +                    0b00011: 'Processor',
> +                    0b00100: 'Disk',
> +                    0b00101: 'Peripheral Bay',
> +                    0b00110: 'System Management Module',
> +                    0b00111: 'Motherboard',
> +                    0b01000: 'Memory Module',
> +                    0b01001: 'Processor Module',
> +                    0b01010: 'Power Unit',
> +                    0b01011: 'Add-in Card',
> +                    0b01100: 'Front Panel Board',
> +                    0b01101: 'Back Panel Board',
> +                    0b01110: 'Power System Board',
> +                    0b01111: 'Drive Back Plane'
> +                    }
> +                self.add_field('status', bitfields.getbits(self.location_and_status, 7, 5), unpack.format_table("location_and_status[7:5]={}", _status))
> +                self.add_field('location', bitfields.getbits(self.location_and_status, 4, 0), unpack.format_table("location_and_status[4:0]={}", _location))
> +            if self.length > 0x6:
> +                self.add_field('maximum_value', u.unpack_one("<H"))
> +            if self.length > 0x8:
> +                self.add_field('minimum_value', u.unpack_one("<H"))
> +            if self.length > 0xA:
> +                self.add_field('resolution', u.unpack_one("<H"))
> +            if self.length > 0xC:
> +                self.add_field('tolerance', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +                self.add_field('accuracy', u.unpack_one("<H"))
> +            if self.length > 0x10:
> +                self.add_field('OEM_defined', u.unpack_one("<I"))
> +            if self.length > 0x14:
> +                self.add_field('nominal_value', u.unpack_one("<H"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing TemperatureProbe"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class ElectricalCurrentProbe(SmbiosBaseStructure):
> +    smbios_structure_type = 29
> +
> +    def __init__(self, u, sm):
> +        super(ElectricalCurrentProbe, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('description', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x5:
> +                self.add_field('location_and_status', u.unpack_one("B"))
> +                _status = {
> +                    0b001: 'Other',
> +                    0b010: 'Unknown',
> +                    0b011: 'OK',
> +                    0b100: 'Non-critical',
> +                    0b101: 'Critical',
> +                    0b110: 'Non-recoverable'
> +                    }
> +                _location = {
> +                    0b00001: 'Other',
> +                    0b00010: 'Unknown',
> +                    0b00011: 'Processor',
> +                    0b00100: 'Disk',
> +                    0b00101: 'Peripheral Bay',
> +                    0b00110: 'System Management Module',
> +                    0b00111: 'Motherboard',
> +                    0b01000: 'Memory Module',
> +                    0b01001: 'Processor Module',
> +                    0b01010: 'Power Unit',
> +                    0b01011: 'Add-in Card',
> +                    0b01100: 'Front Panel Board',
> +                    0b01101: 'Back Panel Board',
> +                    0b01110: 'Power System Board',
> +                    0b01111: 'Drive Back Plane'
> +                    }
> +                self.add_field('status', bitfields.getbits(self.location_and_status, 7, 5), unpack.format_table("location_and_status[7:5]={}", _status))
> +                self.add_field('location', bitfields.getbits(self.location_and_status, 4, 0), unpack.format_table("location_and_status[4:0]={}", _location))
> +            if self.length > 0x6:
> +                self.add_field('maximum_value', u.unpack_one("<H"))
> +            if self.length > 0x8:
> +                self.add_field('minimum_value', u.unpack_one("<H"))
> +            if self.length > 0xA:
> +                self.add_field('resolution', u.unpack_one("<H"))
> +            if self.length > 0xC:
> +                self.add_field('tolerance', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +                self.add_field('accuracy', u.unpack_one("<H"))
> +            if self.length > 0x10:
> +                self.add_field('OEM_defined', u.unpack_one("<I"))
> +            if self.length > 0x14:
> +                self.add_field('nominal_value', u.unpack_one("<H"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing ElectricalCurrentProbe"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class OutOfBandRemoteAccess(SmbiosBaseStructure):
> +    smbios_structure_type = 30
> +
> +    def __init__(self, u, sm):
> +        super(OutOfBandRemoteAccess, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('manufacturer_name', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x5:
> +                self.add_field('connections', u.unpack_one("B"))
> +                self.add_field('outbound_connection_enabled', bool(bitfields.getbits(self.connections, 1)), "connections[1]={}")
> +                self.add_field('inbound_connection_enabled', bool(bitfields.getbits(self.connections, 0)), "connections[0]={}")
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing OutOfBandRemoteAccess"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class BootIntegrityServicesEntryPoint(SmbiosBaseStructure):
> +    smbios_structure_type = 31
> +
> +class SystemBootInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 32
> +
> +    def __init__(self, u, sm):
> +        super(SystemBootInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0xA:
> +                u.skip(6)
> +                _boot_status = {
> +                    0: 'No errors detected',
> +                    1: 'No bootable media',
> +                    2: '"normal" operating system failed to load',
> +                    3: 'Firmware-detected hardware failure, including "unknown" failure types',
> +                    4: 'Operating system-detected hardware failure',
> +                    5: 'User-requested boot, usually through a keystroke',
> +                    6: 'System security violation',
> +                    7: 'Previously-requested image',
> +                    8: 'System watchdog timer expired, causing the system to reboot',
> +                    xrange(9,127): 'Reserved for future assignment',
> +                    xrange(128, 191): 'Vendor/OEM-specific implementations',
> +                    xrange(192, 255): 'Product-specific implementations'
> +                    }
> +                self.add_field('boot_status', u.unpack_one("B"), unpack.format_table("{}", _boot_status))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing SystemBootInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class MemoryErrorInfo64Bit(SmbiosBaseStructure):
> +    smbios_structure_type = 33
> +
> +    def __init__(self, u, sm):
> +        super(MemoryErrorInfo64Bit, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                _error_types = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'OK',
> +                    0x04: 'Bad read',
> +                    0x05: 'Parity error',
> +                    0x06: 'Single-bit error',
> +                    0x07: 'Double-bit error',
> +                    0x08: 'Multi-bit error',
> +                    0x09: 'Nibble error',
> +                    0x0A: 'Checksum error',
> +                    0x0B: 'CRC error',
> +                    0x0C: 'Corrected single-bit error',
> +                    0x0D: 'Corrected error',
> +                    0x0E: 'Uncorrectable error'
> +                    }
> +                self.add_field('error_type', u.unpack_one("B"), unpack.format_table("{}", _error_types))
> +            if self.length > 0x5:
> +                 _error_granularity_field = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Device level',
> +                    0x04: 'Memory partition level'
> +                    }
> +                 self.add_field('error_granularity', u.unpack_one("B"), unpack.format_table("{}", _error_granularity_field))
> +            if self.length > 0x6:
> +                _error_operation_field = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Read',
> +                    0x04: 'Write',
> +                    0x05: 'Partial write'
> +                    }
> +                self.add_field('error_operation', u.unpack_one("B"), unpack.format_table("{}", _error_operation_field))
> +            if self.length > 0x7:
> +                self.add_field('vendor_syndrome', u.unpack_one("<I"))
> +            if self.length > 0xB:
> +                self.add_field('memory_array_error_address', u.unpack_one("<Q"))
> +            if self.length > 0xF:
> +                self.add_field('device_error_address', u.unpack_one("<Q"))
> +            if self.length > 0x13:
> +                self.add_field('error_resolution', u.unpack_one("<Q"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing MemoryErrorInfo64Bit"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class ManagementDevice(SmbiosBaseStructure):
> +    smbios_structure_type = 34
> +
> +    def __init__(self, u, sm):
> +        super(ManagementDevice, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('description', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x5:
> +                _type = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'National Semiconductor LM75',
> +                    0x04: 'National Semiconductor LM78',
> +                    0x05: 'National Semiconductor LM79',
> +                    0x06: 'National Semiconductor LM80',
> +                    0x07: 'National Semiconductor LM81',
> +                    0x08: 'Analog Devices ADM9240',
> +                    0x09: 'Dallas Semiconductor DS1780',
> +                    0x0A: 'Maxim 1617',
> +                    0x0B: 'Genesys GL518SM',
> +                    0x0C: 'Winbond W83781D',
> +                    0x0D: 'Holtek HT82H791'
> +                    }
> +                self.add_field('device_type', u.unpack_one("B"), unpack.format_table("{}", _type))
> +            if self.length > 0x6:
> +                self.add_field('address', u.unpack_one("<I"))
> +            if self.length > 0xA:
> +                 _address_type = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'I/O Port',
> +                    0x04: 'Memory',
> +                    0x05: 'SM Bus'
> +                    }
> +                 self.add_field('address_type', u.unpack_one("B"), unpack.format_table("{}", _address_type))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing ManagementDevice"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class ManagementDeviceComponent(SmbiosBaseStructure):
> +    smbios_structure_type = 35
> +
> +    def __init__(self, u, sm):
> +        super(ManagementDeviceComponent, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('description', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x5:
> +                self.add_field('management_device_handle', u.unpack_one("<H"))
> +            if self.length > 0x7:
> +                self.add_field('component_handle', u.unpack_one("<H"))
> +            if self.length > 0x9:
> +                self.add_field('threshold_handle', u.unpack_one("<H"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing ManagementDeviceComponent"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class ManagementDeviceThresholdData(SmbiosBaseStructure):
> +    smbios_structure_type = 36
> +
> +    def __init__(self, u, sm):
> +        super(ManagementDeviceThresholdData, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('lower_threshold_noncritical', u.unpack_one("<H"))
> +            if self.length > 0x6:
> +                self.add_field('upper_threshold_noncritical', u.unpack_one("<H"))
> +            if self.length > 0x8:
> +                self.add_field('lower_threshold_critical', u.unpack_one("<H"))
> +            if self.length > 0xA:
> +                self.add_field('upper_threshold_critical', u.unpack_one("<H"))
> +            if self.length > 0xC:
> +                self.add_field('lower_threshold_nonrecoverable', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +                self.add_field('upper_threshold_nonrecoverable', u.unpack_one("<H"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing ManagementDeviceThresholdData"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class MemoryChannel(SmbiosBaseStructure):
> +    smbios_structure_type = 37
> +
> +    def __init__(self, u, sm):
> +        super(MemoryChannel, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                _channel_type = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'RamBus',
> +                    0x04: 'SyncLink'
> +                    }
> +                self.add_field('channel_type', u.unpack_one("B"), unpack.format_table("{}", _channel_type))
> +            if self.length > 0x6:
> +                self.add_field('max_channel_load', u.unpack_one("B"))
> +            if self.length > 0x8:
> +                self.add_field('memory_device_count', u.unpack_one("B"))
> +            if self.length > 0xA:
> +                self.add_field('memory_device_load', u.unpack_one("B"))
> +            if self.length > 0xC:
> +                self.add_field('memory_device_handle', u.unpack_one("<H"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing MemoryChannel"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class IPMIDeviceInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 38
> +
> +    def __init__(self, u, sm):
> +        super(IPMIDeviceInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            _interface_type = {
> +                0x00: 'Unknown',
> +                0x01: 'KCS: Keyboard Controller Style',
> +                0x02: 'SMIC: Server Management Interface Chip',
> +                0x03: 'BT: Block Transfer',
> +                xrange(0x04, 0xFF): 'Reserved'
> +                }
> +            self.add_field('interface_type', u.unpack_one("B"), unpack.format_table("{}", _interface_type))
> +            self.add_field('ipmi_specification_revision', u.unpack_one("B"))
> +            self.add_field('msd_revision', bitfields.getbits(self.ipmi_specification_revision, 7, 4), "ipmi_specification_revision[7:4]={}")
> +            self.add_field('lsd_revision', bitfields.getbits(self.ipmi_specification_revision, 3, 0), "ipmi_specification_revision[3:0]={}")
> +
> +            self.add_field('i2c_slave_address', u.unpack_one("B"))
> +            self.add_field('nv_storage_device_address', u.unpack_one("B"))
> +            self.add_field('base_address', u.unpack_one("<Q"))
> +            # if lsb is 1, address is in IO space. otherwise, memory-mapped
> +            self.add_field('base_address_modifier_interrupt_info', u.unpack_one("B"))
> +            _reg_spacing = {
> +                0b00: 'Interface registers are on successive byte boundaries',
> +                0b01: 'Interface registers are on 32-bit boundaries',
> +                0b10: 'Interface registers are on 16-byte boundaries',
> +                0b11: 'Reserved'
> +                }
> +            self.add_field('register_spacing', bitfields.getbits(self.base_address_modifier_interrupt_info, 7, 6), unpack.format_table("base_address_modifier_interrupt_info[7:6]={}", _reg_spacing))
> +            self.add_field('ls_bit_for_addresses', bitfields.getbits(self.base_address_modifier_interrupt_info, 4), "base_address_modifier_interrupt_info[4]={}")
> +            self.add_field('interrupt_info_specified', bool(bitfields.getbits(self.base_address_modifier_interrupt_info, 3)), "base_address_modifier_interrupt_info[3]={}")
> +            _polarity = {
> +                0: 'active low',
> +                1: 'active high'
> +                }
> +            self.add_field('interrupt_polarity', bitfields.getbits(self.base_address_modifier_interrupt_info, 1), unpack.format_table("base_address_modifier_interrupt_info[1]={}", _polarity))
> +            _interrupt_trigger = {
> +                0: 'edge',
> +                1: 'level'
> +                }
> +            self.add_field('interrupt_trigger_mode', bitfields.getbits(self.base_address_modifier_interrupt_info, 0), unpack.format_table("base_address_modifier_interrupt_info[0]={}", _interrupt_trigger))
> +            self.add_field('interrupt_number', u.unpack_one("B"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing IPMIDeviceInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class SystemPowerSupply(SmbiosBaseStructure):
> +    smbios_structure_type = 39
> +
> +    def __init__(self, u, sm):
> +        super(SystemPowerSupply, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('power_unit_group', u.unpack_one("B"))
> +            if self.length > 0x5:
> +                self.add_field('location', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x6:
> +                self.add_field('device_name', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x7:
> +                self.add_field('manufacturer', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x8:
> +                self.add_field('serial_number', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x9:
> +                self.add_field('asset_tag', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0xA:
> +                self.add_field('model_part_number', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0xB:
> +                self.add_field('revision_level', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0xC:
> +                self.add_field('max_power_capacity', u.unpack_one("<H"))
> +            if self.length > 0xE:
> +                self.add_field('power_supply_characteristics', u.unpack_one("<H"))
> +                _dmtf_power_supply_type = {
> +                    0b001: 'Other',
> +                    0b010: 'Unknown',
> +                    0b011: 'Linear',
> +                    0b100: 'Switching',
> +                    0b101: 'Battery',
> +                    0b110: 'UPS',
> +                    0b111: 'Converter',
> +                    0b1000: 'Regulator',
> +                    xrange(0b1001, 0b1111): 'Reserved'
> +                    }
> +                self.add_field('dmtf_power_supply_type', bitfields.getbits(self.power_supply_characteristics, 13, 10), unpack.format_table("power_supply_characteristics[13:10]={}", _dmtf_power_supply_type))
> +                _status = {
> +                    0b001: 'Other',
> +                    0b010: 'Unknown',
> +                    0b011: 'OK',
> +                    0b100: 'Non-critical',
> +                    0b101: 'Critical; power supply has failed and has been taken off-line'
> +                    }
> +                self.add_field('status', bitfields.getbits(self.power_supply_characteristics, 9, 7), unpack.format_table("power_supply_characteristics[9:7]={}", _status))
> +                _dmtf_input_voltage_range_switching = {
> +                    0b001: 'Other',
> +                    0b010: 'Unknown',
> +                    0b011: 'Manual',
> +                    0b100: 'Auto-switch',
> +                    0b101: 'Wide range',
> +                    0b110: 'Not applicable',
> +                    xrange(0b0111, 0b1111): 'Reserved'
> +                    }
> +                self.add_field('dmtf_input_voltage_range_switching', bitfields.getbits(self.power_supply_characteristics, 6, 3), unpack.format_table("power_supply_characteristics[6:3]={}", _dmtf_input_voltage_range_switching))
> +                self.add_field('power_supply_unplugged', bool(bitfields.getbits(self.power_supply_characteristics, 2)), "power_supply_characteristics[2]={}")
> +                self.add_field('power_supply_present', bool(bitfields.getbits(self.power_supply_characteristics, 1)), "power_supply_characteristics[1]={}")
> +                self.add_field('power_supply_hot_replaceable', bool(bitfields.getbits(self.power_supply_characteristics, 0)), "power_supply_characteristics[0]={}")
> +            if self.length > 0x10:
> +                self.add_field('input_voltage_probe_handle', u.unpack_one("<H"))
> +            if self.length > 0x12:
> +                self.add_field('cooling_device_handle', u.unpack_one("<H"))
> +            if self.length > 0x14:
> +                self.add_field('input_current_probe_handle', u.unpack_one("<H"))
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing SystemPowerSupply"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class AdditionalInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 40
> +
> +    def __init__(self, u, sm):
> +        super(AdditionalInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('num_additional_information_entries', u.unpack_one("B"))
> +            if self.length > 0x5:
> +                self.add_field('additional_information_entry_length', u.unpack_one("B"))
> +                self.add_field('referenced_handle', u.unpack_one("<H"))
> +                self.add_field('referenced_offset', u.unpack_one("B"))
> +                self.add_field('string', u.unpack_one("B"), self.fmtstr)
> +                self.add_field('value', u.unpack_rest())
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing AdditionalInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class OnboardDevicesExtendedInformation(SmbiosBaseStructure):
> +    smbios_structure_type = 41
> +
> +    def __init__(self, u, sm):
> +        super(OnboardDevicesExtendedInformation, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                self.add_field('reference_designation', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0x5:
> +                self.add_field('device_type', u.unpack_one("B"))
> +                self.add_field('device_enabled', bool(bitfields.getbits(self.device_type, 7)), "device_type[7]={}")
> +                _device_types = {
> +                    0x01: 'Other',
> +                    0x02: 'Unknown',
> +                    0x03: 'Video',
> +                    0x04: 'SCSI Controller',
> +                    0x05: 'Ethernet',
> +                    0x06: 'Token Ring',
> +                    0x07: 'Sound',
> +                    0x08: 'PATA Controller',
> +                    0x09: 'SATA Controller',
> +                    0x0A: 'SAS Controller'
> +                    }
> +                self.add_field('type_of_device', bitfields.getbits(self.device_type, 6, 0), unpack.format_table("device_type[6:0]={}", _device_types))
> +            if self.length > 0x6:
> +                self.add_field('device_type_instance', u.unpack_one("B"))
> +            if self.length > 0x7:
> +                self.add_field('segment_group_number', u.unpack_one("<H"))
> +            if self.length > 0x9:
> +                self.add_field('bus_number', u.unpack_one("B"), self.fmtstr)
> +            if self.length > 0xA:
> +                self.add_field('device_and_function_number', u.unpack_one("B"))
> +                self.add_field('device_number', bitfields.getbits(self.device_type, 7, 3), "device_and_function_number[7:3]={}")
> +                self.add_field('function_number', bitfields.getbits(self.device_type, 2, 0), "device_and_function_number[2:0]={}")
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing OnboardDevicesExtendedInformation"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class ManagementControllerHostInterface(SmbiosBaseStructure):
> +    smbios_structure_type = 42
> +
> +    def __init__(self, u, sm):
> +        super(ManagementControllerHostInterface, self).__init__(u, sm)
> +        u = self.u
> +        try:
> +            if self.length > 0x4:
> +                _interface_types = {
> +                    0x00: 'Reserved',
> +                    0x01: 'Reserved',
> +                    0x02: 'KCS: Keyboard Controller Style',
> +                    0x03: '8250 UART Register Compatible',
> +                    0x04: '16450 UART Register Compatible',
> +                    0x05: '16550/16550A UART Register Compatible',
> +                    0x06: '16650/16650A UART Register Compatible',
> +                    0x07: '16750/16750A UART Register Compatible',
> +                    0x08: '16850/16850A UART Register Compatible',
> +                    0xF0: 'OEM'
> +                    }
> +                self.add_field('interface_type', u.unpack_one("B"), unpack.format_table("{}", _interface_types))
> +            if self.length > 0x5:
> +                self.add_field('mc_host_interface_data', u.unpack_rest(), self.fmtstr)
> +        except:
> +            self.decodeFailure = True
> +            print "Error parsing ManagementControllerHostInterface"
> +            import traceback
> +            traceback.print_exc()
> +        self.fini()
> +
> +class Inactive(SmbiosBaseStructure):
> +    smbios_structure_type = 126
> +
> +    def __init__(self, u, sm):
> +        super(Inactive, self).__init__(u, sm)
> +        self.fini()
> +
> +class EndOfTable(SmbiosBaseStructure):
> +    smbios_structure_type = 127
> +
> +    def __init__(self, u, sm):
> +        super(EndOfTable, self).__init__(u, sm)
> +        self.fini()
> +
> +class SmbiosStructureUnknown(SmbiosBaseStructure):
> +    smbios_structure_type = None
> +
> +    def __init__(self, u, sm):
> +        super(SmbiosStructureUnknown, self).__init__(u, sm)
> +        self.fini()
> +
> +_smbios_structures = [
> +    BIOSInformation,
> +    SystemInformation,
> +    BaseboardInformation,
> +    SystemEnclosure,
> +    ProcessorInformation,
> +    MemoryControllerInformation,
> +    MemoryModuleInformation,
> +    CacheInformation,
> +    PortConnectorInfo,
> +    SystemSlots,
> +    OnBoardDevicesInformation,
> +    OEMStrings,
> +    SystemConfigOptions,
> +    BIOSLanguageInformation,
> +    GroupAssociations,
> +    SystemEventLog,
> +    PhysicalMemoryArray,
> +    MemoryDevice,
> +    MemoryErrorInfo32Bit,
> +    MemoryArrayMappedAddress,
> +    MemoryDeviceMappedAddress,
> +    BuiltInPointingDevice,
> +    PortableBattery,
> +    SystemReset,
> +    HardwareSecurity,
> +    SystemPowerControls,
> +    VoltageProbe,
> +    CoolingDevice,
> +    TemperatureProbe,
> +    ElectricalCurrentProbe,
> +    OutOfBandRemoteAccess,
> +    BootIntegrityServicesEntryPoint,
> +    SystemBootInformation,
> +    MemoryErrorInfo64Bit,
> +    ManagementDevice,
> +    ManagementDeviceComponent,
> +    ManagementDeviceThresholdData,
> +    MemoryChannel,
> +    IPMIDeviceInformation,
> +    SystemPowerSupply,
> +    AdditionalInformation,
> +    OnboardDevicesExtendedInformation,
> +    ManagementControllerHostInterface,
> +    Inactive,
> +    EndOfTable,
> +    SmbiosStructureUnknown, # Must always come last
> +]
> +
> +def log_smbios_info():
> +    with redirect.logonly():
> +        try:
> +            sm = SMBIOS()
> +            print
> +            if sm is None:
> +                print "No SMBIOS structures found"
> +                return
> +            output = {}
> +            known_types = (0, 1)
> +            for sm_struct in sm.structures:
> +                if sm_struct.type in known_types:
> +                    output.setdefault(sm_struct.type, []).append(sm_struct)
> +                    if len(output) == len(known_types):
> +                        break
> +
> +            print "SMBIOS information:"
> +            for key in sorted(known_types):
> +                for s in output.get(key, ["No structure of type {} found".format(key)]):
> +                    print ttypager._wrap("{}: {}".format(key, s))
> +        except:
> +            print "Error parsing SMBIOS information:"
> +            import traceback
> +            traceback.print_exc()
> +
> +def dump_raw():
> +    try:
> +        sm = SMBIOS()
> +        if sm:
> +            s = "SMBIOS -- Raw bytes and structure decode.\n\n"
> +
> +            s += str(sm.header) + '\n'
> +            s += bits.dumpmem(sm._header_memory) + '\n'
> +
> +            s += "Raw bytes for the SMBIOS structures\n"
> +            s += bits.dumpmem(sm._structure_memory) + '\n'
> +
> +            for sm_struct in sm.structures:
> +                s += str(sm_struct) + '\n'
> +                s += bits.dumpmem(sm_struct.raw_data)
> +
> +                s += "Strings:\n"
> +                for n in range(1, len(getattr(sm_struct, "strings", [])) + 1):
> +                    s += str(sm_struct.fmtstr(n)) + '\n'
> +                s += bits.dumpmem(sm_struct.raw_strings) + '\n'
> +        else:
> +            s = "No SMBIOS structures found"
> +        ttypager.ttypager_wrap(s, indent=False)
> +    except:
> +        print "Error parsing SMBIOS information:"
> +        import traceback
> +        traceback.print_exc()
> +
> +def dump():
> +    try:
> +        sm = SMBIOS()
> +        if sm:
> +            s = str(sm)
> +        else:
> +            s = "No SMBIOS structures found"
> +        ttypager.ttypager_wrap(s, indent=False)
> +    except:
> +        print "Error parsing SMBIOS information:"
> +        import traceback
> +        traceback.print_exc()
> +
> +def annex_a_conformance():
> +    try:
> +        sm = SMBIOS()
> +
> +        # check: 1. The table anchor string "_SM_" is present in the address range 0xF0000 to 0xFFFFF on a 16-byte bound
> +
> +        def table_entry_point_verification():
> +            ''' Verify table entry-point'''
> +            if (sm.header.length < 0x1F):
> +                print "Failure: Table entry-point - The entry-point Length must be at least 0x1F"
> +            if sm.header.checksum != 0:
> +                print "Failure: Table entry-point - The entry-point checksum must evaluate to 0"
> +            if ((sm.header.major_version < 2) and (sm.header.minor_version < 4)):
> +                print "Failure: Table entry-point - SMBIOS version must be at least 2.4"
> +            if (sm.header.intermediate_anchor_string == '_DMI_'):
> +                print "Failure: Table entry-point - The Intermediate Anchor String must be '_DMI_'"
> +            if (sm.header.intermediate_checksum != 0):
> +                print "Failure: Table entry-point - The Intermediate checksum must evaluate to 0"
> +
> +        #check: 3. The structure-table is traversable and conforms to the entry-point specifications:
> +
> +        def req_structures():
> +            '''Checks for required structures and corresponding data'''
> +            types_present = [sm.structures[x].smbios_structure_type for x in range(len(sm.structures))]
> +            required = [0, 1, 4, 7, 9, 16, 17, 19, 31, 32]
> +            for s in required:
> +                if s not in set(types_present):
> +                    print "Failure: Type {} required but not found".format(s)
> +
> +                else:
> +                    if s == 0:
> +                        if types_present.count(s) > 1:
> +                            print "Failure: Type {} - One and only one structure of this type must be present.".format(s)
> +                        if sm.structure_type(s).length < 0x18:
> +                            print "Failure: Type {} - The structure Length field must be at least 0x18".format(s)
> +                        if sm.structure_type(s).version is None:
> +                            print "Failure: Type {} - BIOS Version string must be present and non-null.".format(s)
> +                        if sm.structure_type(s).release_date is None:
> +                            print "Failure: Type {} - BIOS Release Date string must be present, non-null, and include a 4-digit year".format(s)
> +                        if bitfields.getbits(sm.structure_type(s).characteristics, 3, 0) != 0 or bitfields.getbits(sm.structure_type(s).characteristics, 31, 4) == 0:
> +                            print "Failure: Type {} - BIOS Characteristics: bits 3:0 must all be 0, and at least one of bits 31:4 must be set to 1.".format(s)
> +                    elif s == 1:
> +                        if types_present.count(s) > 1:
> +                            print "Failure: Type {} - One and only one structure of this type must be present.".format(s)
> +                        if sm.structure_type(s).length < 0x1B:
> +                            print "Failure: Type {} - The structure Length field must be at least 0x1B".format(s)
> +                        if sm.structure_type(s).manufacturer == None:
> +                            print "Failure: Type {} - Manufacturer string must be present and non-null.".format(s)
> +                        if sm.structure_type(s).product_name == None:
> +                            print "Failure: Type {} - Product Name string must be present and non-null".format(s)
> +                        if sm.structure_type(s).uuid == '00000000 00000000' and sm.structure_type(s).uuid == 'FFFFFFFF FFFFFFFF':
> +                            print "Failure: Type {} - UUID field must be neither 00000000 00000000 nor FFFFFFFF FFFFFFFF.".format(s)
> +                        if sm.structure_type(s).wakeup_type == 00 and sm.structure_type(s).wakeup_type == 0x02:
> +                            print "Failure: Type {} - Wake-up Type field must be neither 00h (Reserved) nor 02h (Unknown).".format(s)
> +                    # continue for remaining required types
> +
> +        # check remaining conformance guidelines
> +
> +        table_entry_point_verification()
> +        req_structures()
> +    except:
> +        print "Error checking ANNEX A conformance guidelines"
> +        import traceback
> +        traceback.print_exc()
> diff --git a/tests/qtest/acpi-bits/bits-tests/testacpi.py b/tests/qtest/acpi-bits/bits-tests/testacpi.py
> new file mode 100644
> index 0000000000..9ec452f330
> --- /dev/null
> +++ b/tests/qtest/acpi-bits/bits-tests/testacpi.py
> @@ -0,0 +1,283 @@
> +# Copyright (c) 2015, Intel Corporation
> +# All rights reserved.
> +#
> +# Redistribution and use in source and binary forms, with or without
> +# modification, are permitted provided that the following conditions are met:
> +#
> +#     * Redistributions of source code must retain the above copyright notice,
> +#       this list of conditions and the following disclaimer.
> +#     * Redistributions in binary form must reproduce the above copyright notice,
> +#       this list of conditions and the following disclaimer in the documentation
> +#       and/or other materials provided with the distribution.
> +#     * Neither the name of Intel Corporation nor the names of its contributors
> +#       may be used to endorse or promote products derived from this software
> +#       without specific prior written permission.
> +#
> +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
> +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
> +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
> +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
> +# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
> +# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
> +# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
> +# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
> +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> +
> +"""Tests for ACPI"""
> +
> +import acpi
> +import bits
> +import bits.mwait
> +import struct
> +import testutil
> +import testsuite
> +import time
> +
> +def register_tests():
> +    testsuite.add_test("ACPI _MAT (Multiple APIC Table Entry) under Processor objects", test_mat, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI DSDT (Differentiated System Description Table)", test_dsdt, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI FACP (Fixed ACPI Description Table)", test_facp, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI HPET (High Precision Event Timer Table)", test_hpet, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI MADT (Multiple APIC Description Table)", test_apic, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI MPST (Memory Power State Table)", test_mpst, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI RSDP (Root System Description Pointer Structure)", test_rsdp, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI XSDT (Extended System Description Table)", test_xsdt, submenu="ACPI Tests")
> +
> +def test_mat():
> +    cpupaths = acpi.get_cpupaths()
> +    apic = acpi.parse_apic()
> +    procid_apicid = apic.procid_apicid
> +    uid_x2apicid = apic.uid_x2apicid
> +    for cpupath in cpupaths:
> +        # Find the ProcId defined by the processor object
> +        processor = acpi.evaluate(cpupath)
> +        # Find the UID defined by the processor object's _UID method
> +        uid = acpi.evaluate(cpupath + "._UID")
> +        mat_buffer = acpi.evaluate(cpupath + "._MAT")
> +        if mat_buffer is None:
> +            continue
> +        # Process each _MAT subtable
> +        mat = acpi._MAT(mat_buffer)
> +        for index, subtable in enumerate(mat):
> +            if subtable.subtype == acpi.MADT_TYPE_LOCAL_APIC:
> +                if subtable.flags.bits.enabled:
> +                    testsuite.test("{} Processor declaration ProcId = _MAT ProcId".format(cpupath), processor.ProcId == subtable.proc_id)
> +                    testsuite.print_detail("{} ProcId ({:#02x}) != _MAT ProcId ({:#02x})".format(cpupath, processor.ProcId, subtable.proc_id))
> +                    testsuite.print_detail("Processor Declaration: {}".format(processor))
> +                    testsuite.print_detail("_MAT entry[{}]: {}".format(index, subtable))
> +                    if testsuite.test("{} with local APIC in _MAT has local APIC in MADT".format(cpupath), processor.ProcId in procid_apicid):
> +                        testsuite.test("{} ApicId derived using Processor declaration ProcId = _MAT ApicId".format(cpupath), procid_apicid[processor.ProcId] == subtable.apic_id)
> +                        testsuite.print_detail("{} ApicId derived from MADT ({:#02x}) != _MAT ApicId ({:#02x})".format(cpupath, procid_apicid[processor.ProcId], subtable.apic_id))
> +                        testsuite.print_detail("Processor Declaration: {}".format(processor))
> +                        testsuite.print_detail("_MAT entry[{}]: {}".format(index, subtable))
> +            if subtable.subtype == acpi.MADT_TYPE_LOCAL_X2APIC:
> +                if subtable.flags.bits.enabled:
> +                    if testsuite.test("{} with x2Apic in _MAT has _UID".format(cpupath), uid is not None):
> +                        testsuite.test("{}._UID = _MAT UID".format(cpupath), uid == subtable.uid)
> +                        testsuite.print_detail("{}._UID ({:#x}) != _MAT UID ({:#x})".format(cpupath, uid, subtable.uid))
> +                        testsuite.print_detail("_MAT entry[{}]: {}".format(index, subtable))
> +                    if testsuite.test("{} with _MAT x2Apic has x2Apic in MADT".format(cpupath), subtable.uid in uid_x2apicid):
> +                        testsuite.test("{} x2ApicId derived from MADT using UID = _MAT x2ApicId".format(cpupath), uid_x2apicid[subtable.uid] == subtable.x2apicid)
> +                        testsuite.print_detail("{} x2ApicId derived from MADT ({:#02x}) != _MAT x2ApicId ({:#02x})".format(cpupath, uid_x2apicid[subtable.uid], subtable.x2apicid))
> +                        testsuite.print_detail("_MAT entry[{}]: {}".format(index, subtable))
> +
> +def test_pss():
> +    uniques = acpi.parse_cpu_method("_PSS")
> +    # We special-case None here to avoid a double-failure for CPUs without a _PSS
> +    testsuite.test("_PSS must be identical for all CPUs", len(uniques) <= 1 or (len(uniques) == 2 and None in uniques))
> +    for pss, cpupaths in uniques.iteritems():
> +        if not testsuite.test("_PSS must exist", pss is not None):
> +            testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
> +            testsuite.print_detail('No _PSS exists')
> +            continue
> +
> +        if not testsuite.test("_PSS must not be empty", pss.pstates):
> +            testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
> +            testsuite.print_detail('_PSS is empty')
> +            continue
> +
> +        testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
> +        for index, pstate in enumerate(pss.pstates):
> +            testsuite.print_detail("P[{}]: {}".format(index, pstate))
> +
> +        testsuite.test("_PSS must contain at most 16 Pstates", len(pss.pstates) <= 16)
> +        testsuite.test("_PSS must have no duplicate Pstates", len(pss.pstates) == len(set(pss.pstates)))
> +
> +        frequencies = [p.core_frequency for p in pss.pstates]
> +        testsuite.test("_PSS must list Pstates in descending order of frequency", frequencies == sorted(frequencies, reverse=True))
> +
> +        testsuite.test("_PSS must have Pstates with no duplicate frequencies", len(frequencies) == len(set(frequencies)))
> +
> +        dissipations = [p.power for p in pss.pstates]
> +        testsuite.test("_PSS must list Pstates in descending order of power dissipation", dissipations == sorted(dissipations, reverse=True))
> +
> +def test_pstates():
> +    """Execute and verify frequency for each Pstate in the _PSS"""
> +    IA32_PERF_CTL = 0x199
> +    with bits.mwait.use_hint(), bits.preserve_msr(IA32_PERF_CTL):
> +        cpupath_procid = acpi.find_procid()
> +        cpupath_uid = acpi.find_uid()
> +        apic = acpi.parse_apic()
> +        procid_apicid = apic.procid_apicid
> +        uid_x2apicid = apic.uid_x2apicid
> +        def cpupath_apicid(cpupath):
> +            if procid_apicid is not None:
> +                procid = cpupath_procid.get(cpupath, None)
> +                if procid is not None:
> +                    apicid = procid_apicid.get(procid, None)
> +                    if apicid is not None:
> +                        return apicid
> +            if uid_x2apicid is not None:
> +                uid = cpupath_uid.get(cpupath, None)
> +                if uid is not None:
> +                    apicid = uid_x2apicid.get(uid, None)
> +                    if apicid is not None:
> +                        return apicid
> +            return bits.cpus()[0]
> +
> +        bclk = testutil.adjust_to_nearest(bits.bclk(), 100.0/12) * 1000000
> +
> +        uniques = acpi.parse_cpu_method("_PSS")
> +        for pss, cpupaths in uniques.iteritems():
> +            if not testsuite.test("_PSS must exist", pss is not None):
> +                testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
> +                testsuite.print_detail('No _PSS exists')
> +                continue
> +
> +            for n, pstate in enumerate(pss.pstates):
> +                for cpupath in cpupaths:
> +                    apicid = cpupath_apicid(cpupath)
> +                    if apicid is None:
> +                        print 'Failed to find apicid for cpupath {}'.format(cpupath)
> +                        continue
> +                    bits.wrmsr(apicid, IA32_PERF_CTL, pstate.control)
> +
> +                # Detecting Turbo frequency requires at least 2 pstates
> +                # since turbo frequency = max non-turbo frequency + 1
> +                turbo = False
> +                if len(pss.pstates) >= 2:
> +                    turbo = (n == 0 and pstate.core_frequency == (pss.pstates[1].core_frequency + 1))
> +                    if turbo:
> +                        # Needs to busywait, not sleep
> +                        start = time.time()
> +                        while (time.time() - start < 2):
> +                            pass
> +
> +                for duration in (0.1, 1.0):
> +                    frequency_data = bits.cpu_frequency(duration)
> +                    # Abort the test if no cpu frequency is not available
> +                    if frequency_data is None:
> +                        continue
> +                    aperf = frequency_data[1]
> +                    aperf = testutil.adjust_to_nearest(aperf, bclk/2)
> +                    aperf = int(aperf / 1000000)
> +                    if turbo:
> +                        if aperf >= pstate.core_frequency:
> +                            break
> +                    else:
> +                        if aperf == pstate.core_frequency:
> +                            break
> +
> +                if turbo:
> +                    testsuite.test("P{}: Turbo measured frequency {} >= expected {} MHz".format(n, aperf, pstate.core_frequency), aperf >= pstate.core_frequency)
> +                else:
> +                    testsuite.test("P{}: measured frequency {} MHz == expected {} MHz".format(n, aperf, pstate.core_frequency), aperf == pstate.core_frequency)
> +
> +def test_psd_thread_scope():
> +    uniques = acpi.parse_cpu_method("_PSD")
> +    if not testsuite.test("_PSD (P-State Dependency) must exist for each processor", None not in uniques):
> +        testsuite.print_detail(acpi.factor_commonprefix(uniques[None]))
> +        testsuite.print_detail('No _PSD exists')
> +        return
> +    unique_num_dependencies = {}
> +    unique_num_entries = {}
> +    unique_revision = {}
> +    unique_domain = {}
> +    unique_coordination_type = {}
> +    unique_num_processors = {}
> +    for value, cpupaths in uniques.iteritems():
> +        unique_num_dependencies.setdefault(len(value.dependencies), []).extend(cpupaths)
> +        unique_num_entries.setdefault(value.dependencies[0].num_entries, []).extend(cpupaths)
> +        unique_revision.setdefault(value.dependencies[0].revision, []).extend(cpupaths)
> +        unique_domain.setdefault(value.dependencies[0].domain, []).extend(cpupaths)
> +        unique_coordination_type.setdefault(value.dependencies[0].coordination_type, []).extend(cpupaths)
> +        unique_num_processors.setdefault(value.dependencies[0].num_processors, []).extend(cpupaths)
> +    def detail(d, fmt):
> +        for value, cpupaths in sorted(d.iteritems(), key=(lambda (k,v): v)):
> +            testsuite.print_detail(acpi.factor_commonprefix(cpupaths))
> +            testsuite.print_detail(fmt.format(value))
> +
> +    testsuite.test('Dependency count for each processor must be 1', unique_num_dependencies.keys() == [1])
> +    detail(unique_num_dependencies, 'Dependency count for each processor = {} (Expected 1)')
> +    testsuite.test('_PSD.num_entries must be 5', unique_num_entries.keys() == [5])
> +    detail(unique_num_entries, 'num_entries = {} (Expected 5)')
> +    testsuite.test('_PSD.revision must be 0', unique_revision.keys() == [0])
> +    detail(unique_revision, 'revision = {}')
> +    testsuite.test('_PSD.coordination_type must be 0xFE (HW_ALL)', unique_coordination_type.keys() == [0xfe])
> +    detail(unique_coordination_type, 'coordination_type = {:#x} (Expected 0xFE HW_ALL)')
> +    testsuite.test('_PSD.domain must be unique (thread-scoped) for each processor', len(unique_domain) == len(acpi.get_cpupaths()))
> +    detail(unique_domain, 'domain = {:#x} (Expected a unique value for each processor)')
> +    testsuite.test('_PSD.num_processors must be 1', unique_num_processors.keys() == [1])
> +    detail(unique_num_processors, 'num_processors = {} (Expected 1)')
> +
> +def test_table_checksum(data):
> +    csum = sum(ord(c) for c in data) % 0x100
> +    testsuite.test('ACPI table cumulative checksum must equal 0', csum == 0)
> +    testsuite.print_detail("Cumulative checksum = {} (Expected 0)".format(csum))
> +
> +def test_apic():
> +    data = acpi.get_table("APIC")
> +    if data is None:
> +        return
> +    test_table_checksum(data)
> +    apic = acpi.parse_apic()
> +
> +def test_dsdt():
> +    data = acpi.get_table("DSDT")
> +    if data is None:
> +        return
> +    test_table_checksum(data)
> +
> +def test_facp():
> +    data = acpi.get_table("FACP")
> +    if data is None:
> +        return
> +    test_table_checksum(data)
> +    facp = acpi.parse_facp()
> +
> +def test_hpet():
> +    data = acpi.get_table("HPET")
> +    if data is None:
> +        return
> +    test_table_checksum(data)
> +    hpet = acpi.parse_hpet()
> +
> +def test_mpst():
> +    data = acpi.get_table("MPST")
> +    if data is None:
> +        return
> +    test_table_checksum(data)
> +    mpst = acpi.MPST(data)
> +
> +def test_rsdp():
> +    data = acpi.get_table("RSD PTR ")
> +    if data is None:
> +        return
> +
> +    # Checksum the first 20 bytes per ACPI 1.0
> +    csum = sum(ord(c) for c in data[:20]) % 0x100
> +    testsuite.test('ACPI 1.0 table first 20 bytes cummulative checksum must equal 0', csum == 0)
> +    testsuite.print_detail("Cummulative checksum = {} (Expected 0)".format(csum))
> +
> +    test_table_checksum(data)
> +    rsdp = acpi.parse_rsdp()
> +
> +def test_xsdt():
> +    data = acpi.get_table("XSDT")
> +    if data is None:
> +        return
> +    test_table_checksum(data)
> +    xsdt = acpi.parse_xsdt()
> diff --git a/tests/qtest/acpi-bits/bits-tests/testcpuid.py b/tests/qtest/acpi-bits/bits-tests/testcpuid.py
> new file mode 100644
> index 0000000000..ac55d912e1
> --- /dev/null
> +++ b/tests/qtest/acpi-bits/bits-tests/testcpuid.py
> @@ -0,0 +1,83 @@
> +# Copyright (c) 2012, Intel Corporation
> +# All rights reserved.
> +#
> +# Redistribution and use in source and binary forms, with or without
> +# modification, are permitted provided that the following conditions are met:
> +#
> +#     * Redistributions of source code must retain the above copyright notice,
> +#       this list of conditions and the following disclaimer.
> +#     * Redistributions in binary form must reproduce the above copyright notice,
> +#       this list of conditions and the following disclaimer in the documentation
> +#       and/or other materials provided with the distribution.
> +#     * Neither the name of Intel Corporation nor the names of its contributors
> +#       may be used to endorse or promote products derived from this software
> +#       without specific prior written permission.
> +#
> +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
> +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
> +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
> +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
> +# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
> +# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
> +# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
> +# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
> +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Could you do me a favour and add a SPDX license identifier in front of the 
license boilerplate codes?

It's not required for the QEMU project, but at least I normally have 
troubles to find out what such longer license boilertexts mean ... is it a 
BSD license, is it a MIT license, something completely else...? A short SPDX 
line at the beginning helps to identify the license at a first glance.

  Thomas

Re: [PATCH 04/12] acpi/tests/bits: initial commit of test scripts that are run by biosbits

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 09:24:19AM +0200, Thomas Huth wrote:

... 3000 lines of irrelevant code ...


> > diff --git a/tests/qtest/acpi-bits/bits-tests/testcpuid.py b/tests/qtest/acpi-bits/bits-tests/testcpuid.py
> > new file mode 100644
> > index 0000000000..ac55d912e1
> > --- /dev/null
> > +++ b/tests/qtest/acpi-bits/bits-tests/testcpuid.py
> > @@ -0,0 +1,83 @@
> > +# Copyright (c) 2012, Intel Corporation
> > +# All rights reserved.
> > +#
> > +# Redistribution and use in source and binary forms, with or without
> > +# modification, are permitted provided that the following conditions are met:
> > +#
> > +#     * Redistributions of source code must retain the above copyright notice,
> > +#       this list of conditions and the following disclaimer.
> > +#     * Redistributions in binary form must reproduce the above copyright notice,
> > +#       this list of conditions and the following disclaimer in the documentation
> > +#       and/or other materials provided with the distribution.
> > +#     * Neither the name of Intel Corporation nor the names of its contributors
> > +#       may be used to endorse or promote products derived from this software
> > +#       without specific prior written permission.
> > +#
> > +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
> > +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
> > +# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
> > +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
> > +# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
> > +# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
> > +# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
> > +# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> > +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
> > +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> 
> Could you do me a favour and add a SPDX license identifier in front of the
> license boilerplate codes?
> 
> It's not required for the QEMU project, but at least I normally have
> troubles to find out what such longer license boilertexts mean ... is it a
> BSD license, is it a MIT license, something completely else...? A short SPDX
> line at the beginning helps to identify the license at a first glance.
> 
>  Thomas

Pls remember to cut out irrelevant text when you respond.

Thanks,
-- 
MST

[PATCH 05/12] acpi/tests/bits: disable acpi PSS tests that are failing in biosbits

[Ani Sinha]

PSS tests in acpi test suite seems to be failing in biosbits. This is because
the test is unable to find PSS support in QEMU bios. Let us disable
them for now so that make check does not fail. We can fix the tests and
re-enable them later.

Example failure:

---- ACPI _PSS (Pstate) table conformance tests ----
[assert] _PSS must exist FAIL
  \_SB_.CPUS.C000
  No _PSS exists
Summary: 1 passed, 1 failed
---- ACPI _PSS (Pstate) runtime tests ----
[assert] _PSS must exist FAIL
  \_SB_.CPUS.C000
  No _PSS exists
Summary: 0 passed, 1 failed

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/acpi-bits/bits-tests/testacpi.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/qtest/acpi-bits/bits-tests/testacpi.py b/tests/qtest/acpi-bits/bits-tests/testacpi.py
index 9ec452f330..dbc150076e 100644
--- a/tests/qtest/acpi-bits/bits-tests/testacpi.py
+++ b/tests/qtest/acpi-bits/bits-tests/testacpi.py
@@ -36,8 +36,8 @@
 
 def register_tests():
     testsuite.add_test("ACPI _MAT (Multiple APIC Table Entry) under Processor objects", test_mat, submenu="ACPI Tests")
-    testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests")
-    testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests")
+#    testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests")
+#    testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests")
     testsuite.add_test("ACPI DSDT (Differentiated System Description Table)", test_dsdt, submenu="ACPI Tests")
     testsuite.add_test("ACPI FACP (Fixed ACPI Description Table)", test_facp, submenu="ACPI Tests")
     testsuite.add_test("ACPI HPET (High Precision Event Timer Table)", test_hpet, submenu="ACPI Tests")
-- 
2.25.1

[PATCH 06/12] acpi/tests/bits: add smilatency test suite from bits in order to disable it

[Ani Sinha]

smilatency tests does not reliably pass every time it is run from QEMU. This
change adds the test file unchanged from bits so that the next change can
disable the test.

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/acpi-bits/bits-tests/meson.build  |   2 +-
 .../qtest/acpi-bits/bits-tests/smilatency.py  | 102 ++++++++++++++++++
 2 files changed, 103 insertions(+), 1 deletion(-)
 create mode 100644 tests/qtest/acpi-bits/bits-tests/smilatency.py

diff --git a/tests/qtest/acpi-bits/bits-tests/meson.build b/tests/qtest/acpi-bits/bits-tests/meson.build
index 3056731a53..06bca75d99 100644
--- a/tests/qtest/acpi-bits/bits-tests/meson.build
+++ b/tests/qtest/acpi-bits/bits-tests/meson.build
@@ -1,4 +1,4 @@
-test_files = ['smbios.py', 'testacpi.py', 'testcpuid.py']
+test_files = ['smbios.py', 'smilatency.py', 'testacpi.py', 'testcpuid.py']
 
 copytestfiles = custom_target('copy test files',
   input : test_files,
diff --git a/tests/qtest/acpi-bits/bits-tests/smilatency.py b/tests/qtest/acpi-bits/bits-tests/smilatency.py
new file mode 100644
index 0000000000..fb1b7228e3
--- /dev/null
+++ b/tests/qtest/acpi-bits/bits-tests/smilatency.py
@@ -0,0 +1,102 @@
+# Copyright (c) 2015, Intel Corporation
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+#     * Redistributions of source code must retain the above copyright notice,
+#       this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright notice,
+#       this list of conditions and the following disclaimer in the documentation
+#       and/or other materials provided with the distribution.
+#     * Neither the name of Intel Corporation nor the names of its contributors
+#       may be used to endorse or promote products derived from this software
+#       without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""SMI latency test."""
+
+import bits
+from collections import namedtuple
+import testsuite
+import time
+import usb
+
+def register_tests():
+    testsuite.add_test("SMI latency test", smi_latency);
+    testsuite.add_test("SMI latency test with USB disabled via BIOS handoff", test_with_usb_disabled, runall=False);
+
+def smi_latency():
+    MSR_SMI_COUNT = 0x34
+
+    print "Warning: touching the keyboard can affect the results of this test."
+
+    tsc_per_sec = bits.tsc_per_sec()
+    tsc_per_usec = tsc_per_sec / (1000 * 1000)
+    bins = [long(tsc_per_usec * 10**i) for i in range(9)]
+    bin_descs = [
+        "0     < t <=   1us",
+        "1us   < t <=  10us",
+        "10us  < t <= 100us",
+        "100us < t <=   1ms",
+        "1ms   < t <=  10ms",
+        "10ms  < t <= 100ms",
+        "100ms < t <=   1s ",
+        "1s    < t <=  10s ",
+        "10s   < t <= 100s ",
+        "100s  < t         ",
+    ]
+
+    print "Starting test. Wait here, I will be back in 15 seconds."
+    (max_latency, smi_count_delta, bins) = bits.smi_latency(long(15 * tsc_per_sec), bins)
+    BinType = namedtuple('BinType', ("max", "total", "count", "times"))
+    bins = [BinType(*b) for b in bins]
+
+    testsuite.test("SMI latency < 150us to minimize risk of OS timeouts", max_latency / tsc_per_usec <= 150)
+    if not testsuite.show_detail():
+        return
+
+    for bin, desc in zip(bins, bin_descs):
+        if bin.count == 0:
+            continue
+        testsuite.print_detail("{}; average = {}; count = {}".format(desc, bits.format_tsc(bin.total/bin.count), bin.count))
+        deltas = (bits.format_tsc(t2 - t1) for t1,t2 in zip(bin.times, bin.times[1:]))
+        testsuite.print_detail(" Times between first few observations: {}".format(" ".join("{:>6}".format(delta) for delta in deltas)))
+
+    if smi_count_delta is not None:
+        testsuite.print_detail("{} SMI detected using MSR_SMI_COUNT (MSR {:#x})".format(smi_count_delta, MSR_SMI_COUNT))
+
+    testsuite.print_detail("Summary of impact: observed maximum latency = {}".format(bits.format_tsc(max_latency)))
+
+def test_with_usb_disabled():
+    if usb.handoff_to_os():
+        smi_latency()
+
+def average_io_smi(port, value, count):
+    def f():
+        tsc_start = bits.rdtsc()
+        bits.outb(port, value)
+        return bits.rdtsc() - tsc_start
+    counts = [f() for i in range(count)]
+    return sum(counts)/len(counts)
+
+def time_io_smi(port=0xb2, value=0, count=1000):
+    count_for_estimate = 10
+    start = time.time()
+    average_io_smi(port, value, count_for_estimate)
+    avg10 = time.time() - start
+    estimate = avg10 * count/count_for_estimate
+    if estimate > 1:
+        print "Running test, estimated time: {}s".format(int(estimate))
+    average = average_io_smi(port, value, count)
+    print "Average of {} SMIs (via outb, port={:#x}, value={:#x}): {}".format(count, port, value, bits.format_tsc(average))
-- 
2.25.1

[PATCH 07/12] acpi/tests/bits: disable smilatency test since it does not pass everytime

[Ani Sinha]

smilatency test is latency sensitive and does not pass deterministically when
run in QEMU environment under biosbits. Disable the test suite for now.

Example failure:

==== SMI latency test ====
Warning: touching the keyboard can affect the results of this test.
Starting test. Wait here, I will be back in 15 seconds.
[assert] SMI latency < 150us to minimize risk of OS timeouts FAIL
  1us   < t <=  10us; average = 1372ns; count = 10912449
   Times between first few observations:  176us 1646ns 1441ns 1450ns 1462ns
  10us  < t <= 100us; average = 16us; count = 1187
   Times between first few observations:   15ms 3148us 5856us   49ms   33ms
  100us < t <=   1ms; average = 259us; count = 8
   Times between first few observations:  111ms 2227ms 1779ms  999ms  219ms
  0 SMI detected using MSR_SMI_COUNT (MSR 0x34)
  Summary of impact: observed maximum latency = 298us
Summary: 0 passed, 1 failed

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/acpi-bits/bits-tests/smilatency.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/tests/qtest/acpi-bits/bits-tests/smilatency.py b/tests/qtest/acpi-bits/bits-tests/smilatency.py
index fb1b7228e3..53b5f820a5 100644
--- a/tests/qtest/acpi-bits/bits-tests/smilatency.py
+++ b/tests/qtest/acpi-bits/bits-tests/smilatency.py
@@ -33,8 +33,9 @@
 import usb
 
 def register_tests():
-    testsuite.add_test("SMI latency test", smi_latency);
-    testsuite.add_test("SMI latency test with USB disabled via BIOS handoff", test_with_usb_disabled, runall=False);
+    pass
+    # testsuite.add_test("SMI latency test", smi_latency);
+    # testsuite.add_test("SMI latency test with USB disabled via BIOS handoff", test_with_usb_disabled, runall=False);
 
 def smi_latency():
     MSR_SMI_COUNT = 0x34
-- 
2.25.1

[PATCH 08/12] acpi/tests/bits: add biosbits config file for running bios tests

[Ani Sinha]

This change adds initial biosbits config file that instructs biosbits to run
bios test suits in batch mode. Additionally acpi and smbios structures are also
dumped.

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/acpi-bits/bits-config/bits-cfg.txt | 18 ++++++++++++++++++
 tests/qtest/acpi-bits/bits-config/meson.build  | 11 +++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 tests/qtest/acpi-bits/bits-config/bits-cfg.txt
 create mode 100644 tests/qtest/acpi-bits/bits-config/meson.build

diff --git a/tests/qtest/acpi-bits/bits-config/bits-cfg.txt b/tests/qtest/acpi-bits/bits-config/bits-cfg.txt
new file mode 100644
index 0000000000..8010804453
--- /dev/null
+++ b/tests/qtest/acpi-bits/bits-config/bits-cfg.txt
@@ -0,0 +1,18 @@
+# BITS configuration file
+[bits]
+
+# To run BITS in batch mode, set batch to a list of one or more of the
+# following keywords; BITS will then run all of the requested operations, then
+# save the log file to disk.
+#
+# test: Run the full BITS testsuite.
+# acpi: Dump all ACPI structures.
+# smbios: Dump all SMBIOS structures.
+#
+# Leave batch set to an empty string to disable batch mode.
+# batch =
+
+# Uncomment the following to run all available batch operations
+# please take a look at boot/python/init.py in bits zip file
+# to see how these options are parsed and used.
+batch = test acpi smbios
diff --git a/tests/qtest/acpi-bits/bits-config/meson.build b/tests/qtest/acpi-bits/bits-config/meson.build
new file mode 100644
index 0000000000..bbd7a940dc
--- /dev/null
+++ b/tests/qtest/acpi-bits/bits-config/meson.build
@@ -0,0 +1,11 @@
+config_files = ['bits-cfg.txt']
+
+copycfgfiles = custom_target('copy cfg files',
+  input : config_files,
+  output :  config_files,
+  command : ['cp', '@INPUT@', '@OUTPUT@'],
+  install : true,
+  install_dir : 'bits-config',
+  build_by_default : true)
+
+other_deps += copycfgfiles
-- 
2.25.1

[PATCH 09/12] acpi/tests/bits: add acpi and smbios python tests that uses biosbits

[Ani Sinha]

This change adds python based qtest framework that can be used to run
qtests from within a virtual environment. A bash script creates the virtual
environment and then runs the python based tests from within that environment.
All dependent python packages are installed in the virtual environment using
pip module. QEMU python test modules are also available in the environment for
spawning the QEMU based VMs.

It also introduces QEMU acpi/smbios biosbits python test script which is run
from within the python virtual environment.

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/acpi-bits/acpi-bits-test-venv.sh |  59 ++++
 tests/qtest/acpi-bits/acpi-bits-test.py      | 327 +++++++++++++++++++
 tests/qtest/acpi-bits/meson.build            |  39 +++
 tests/qtest/acpi-bits/requirements.txt       |   1 +
 4 files changed, 426 insertions(+)
 create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh
 create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py
 create mode 100644 tests/qtest/acpi-bits/meson.build
 create mode 100644 tests/qtest/acpi-bits/requirements.txt

diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
new file mode 100644
index 0000000000..124e03ce18
--- /dev/null
+++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# Generates a python virtual environment for the test to run.
+# Then runs python test scripts from within that virtual environment.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# Author: Ani Sinha <ani@anisinha.ca>
+
+set -e
+
+MYPATH=$(realpath ${BASH_SOURCE:-$0})
+MYDIR=$(dirname $MYPATH)
+
+if [ -z "$QTEST_SOURCE_ROOT" ]; then
+    echo -n "Please set QTEST_SOURCE_ROOT env pointing"
+    echo " to the root of the qemu source tree."
+    echo -n "This is required so that the test can find the "
+    echo "python modules that it needs for execution."
+    exit 1
+fi
+SRCDIR=$QTEST_SOURCE_ROOT
+TESTSCRIPTS=("acpi-bits-test.py")
+PIPCMD="-m pip -q --disable-pip-version-check"
+# we need to save the old value of PWD before we do a change-dir later
+QTEST_PWD=$PWD
+
+TESTS_PYTHON=/usr/bin/python3
+TESTS_VENV_REQ=requirements.txt
+
+# sadly for pip -e and -t options do not work together.
+# please see https://github.com/pypa/pip/issues/562
+cd $MYDIR
+
+$TESTS_PYTHON -m venv .
+$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/
+[ -f $TESTS_VENV_REQ ] && \
+    $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ
+
+# venv is activated at this point.
+
+# run the test
+for testscript in ${TESTSCRIPTS[@]} ; do
+    export QTEST_PWD; python3 $testscript
+done
+
+cd $QTEST_PWD
+
+exit 0
diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py
new file mode 100644
index 0000000000..673567bf8e
--- /dev/null
+++ b/tests/qtest/acpi-bits/acpi-bits-test.py
@@ -0,0 +1,327 @@
+#!/usr/bin/env python3
+# group: rw quick
+# Exercize QEMU generated ACPI/SMBIOS tables using biosbits,
+# https://biosbits.org/
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# Some parts are slightly taken from qtest.py and iotests.py
+#
+# Authors:
+#  Ani Sinha <ani@anisinha.ca>
+
+# pylint: disable=invalid-name
+
+"""
+QEMU bios tests using biosbits available at
+https://biosbits.org/.
+"""
+
+import logging
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tarfile
+import tempfile
+import time
+import unittest
+import zipfile
+from typing import (
+    List,
+    Optional,
+    Sequence,
+)
+from tap import TAPTestRunner
+from qemu.machine import QEMUMachine
+
+QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY')
+QTEST_PWD = os.getenv('QTEST_PWD')
+
+def get_arch():
+    """finds the arch from the qemu binary name"""
+    match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG)
+    if match:
+        return match.group(1)
+    return 'x86_64'
+
+ARCH = get_arch()
+
+class QEMUBitsMachine(QEMUMachine):
+    """
+    A QEMU VM, with isa-debugcon enabled and bits iso passed
+    using -cdrom to QEMU commandline.
+    """
+    def __init__(self,
+                 binary: str,
+                 args: Sequence[str] = (),
+                 wrapper: Sequence[str] = (),
+                 name: Optional[str] = None,
+                 base_temp_dir: str = "/var/tmp",
+                 debugcon_log: str = "debugcon-log.txt",
+                 debugcon_addr: str = "0x403",
+                 sock_dir: Optional[str] = None,
+                 qmp_timer: Optional[float] = None):
+        # pylint: disable=too-many-arguments
+
+        if name is None:
+            name = "qemu-bits-%d" % os.getpid()
+        if sock_dir is None:
+            sock_dir = base_temp_dir
+        super().__init__(binary, args, wrapper=wrapper, name=name,
+                         base_temp_dir=base_temp_dir,
+                         sock_dir=sock_dir, qmp_timer=qmp_timer)
+        self.debugcon_log = debugcon_log
+        self.debugcon_addr = debugcon_addr
+        self.base_temp_dir = base_temp_dir
+
+    @property
+    def _base_args(self) -> List[str]:
+        args = super()._base_args
+        args.extend([
+            '-chardev',
+            'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir,
+                                                     self.debugcon_log),
+            '-device',
+            'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr,
+        ])
+        return args
+
+    def base_args(self):
+        """return the base argument to QEMU binary"""
+        return self._base_args
+
+class AcpiBitsTest(unittest.TestCase):
+    """ACPI and SMBIOS tests using biosbits."""
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._vm = None
+        self._workdir = None
+        self._bitsver = 2095
+        self._debugcon_addr = '0x403'
+        self._debugcon_log = 'debugcon-log.txt'
+        logging.basicConfig(level=logging.INFO)
+
+    def copy_bits_config(self):
+        """ copies the bios bits config file into bits.
+        """
+        config_file = 'bits-cfg.txt'
+        qemu_bits_config_dir = os.path.join(os.getcwd(), 'bits-config')
+        target_config_dir = os.path.join(self._workdir,
+                                         'bits-%d' %self._bitsver, 'boot')
+        self.assertTrue(os.path.exists(qemu_bits_config_dir))
+        self.assertTrue(os.path.exists(target_config_dir))
+        self.assertTrue(os.access(os.path.join(qemu_bits_config_dir,
+                                               config_file), os.R_OK))
+        shutil.copy2(os.path.join(qemu_bits_config_dir, config_file),
+                     target_config_dir)
+        logging.info('copied config file %s to %s',
+                     config_file, target_config_dir)
+
+    def copy_test_scripts(self):
+        """copies the python test scripts into bits. """
+        qemu_test_dir = os.path.join(os.getcwd(), 'bits-tests')
+        target_test_dir = os.path.join(self._workdir, 'bits-%d' %self._bitsver,
+                                       'boot', 'python')
+
+        self.assertTrue(os.path.exists(qemu_test_dir))
+        self.assertTrue(os.path.exists(target_test_dir))
+
+        for filename in os.listdir(qemu_test_dir):
+            if os.path.isfile(os.path.join(qemu_test_dir, filename)) and \
+               filename.endswith('.py'):
+                shutil.copy2(os.path.join(qemu_test_dir, filename),
+                             target_test_dir)
+                logging.info('copied test file %s to %s',
+                             filename, target_test_dir)
+
+                # now remove the pyc test file if it exists, otherwise the
+                # changes in the python test script won't be executed.
+                testfile_pyc = os.path.splitext(filename)[0] + '.pyc'
+                if os.access(os.path.join(target_test_dir, testfile_pyc),
+                             os.R_OK):
+                    os.remove(os.path.join(target_test_dir, testfile_pyc))
+                    logging.info('removed compiled file %s',
+                                 os.path.join(target_test_dir, testfile_pyc))
+
+    def fix_mkrescue(self, mkrescue):
+        """ grub-mkrescue is a bash script with two variables, 'prefix' and
+            'libdir'. They must be pointed to the right location so that the
+            iso can be generated appropriately. We point the two variables to
+            the directory where we have extracted our pre-built bits grub
+            tarball.
+        """
+        grub_x86_64_mods = os.path.join(self._workdir, 'bits-grub',
+                                        'grub-inst-x86_64-efi')
+        grub_i386_mods = os.path.join(self._workdir, 'bits-grub',
+                                      'grub-inst')
+
+        self.assertTrue(os.path.exists(grub_x86_64_mods))
+        self.assertTrue(os.path.exists(grub_i386_mods))
+
+        new_script = ""
+        with open(mkrescue, 'r') as filehandle:
+            orig_script = filehandle.read()
+            new_script = re.sub('(^prefix=)(.*)',
+                                r'\1"%s"' %grub_x86_64_mods,
+                                orig_script, flags=re.M)
+            new_script = re.sub('(^libdir=)(.*)', r'\1"%s/lib"' %grub_i386_mods,
+                                new_script, flags=re.M)
+
+        with open(mkrescue, 'w') as filehandle:
+            filehandle.write(new_script)
+
+    def generate_bits_iso(self):
+        """ Uses grub-mkrescue to generate a fresh bits iso with the python
+            test scripts
+        """
+        bits_dir = os.path.join(self._workdir, 'bits-%d' %self._bitsver)
+        iso_file = os.path.join(self._workdir, 'bits-%d.iso' %self._bitsver)
+        mkrescue_script = os.path.join(self._workdir,
+                                       'bits-grub', 'grub-inst-x86_64-efi',
+                                       'bin', 'grub-mkrescue')
+
+        self.assertTrue(os.access(mkrescue_script, os.R_OK | os.W_OK))
+
+        self.fix_mkrescue(mkrescue_script)
+
+        logging.info('calling grub-mkrescue to generate the biosbits iso ...')
+
+        if os.getenv('V'):
+            subprocess.check_call([mkrescue_script, '-o', iso_file, bits_dir],
+                                  stdout=subprocess.DEVNULL)
+        else:
+            subprocess.check_call([mkrescue_script, '-o', iso_file, bits_dir],
+                                  stderr=subprocess.DEVNULL,
+                                  stdout=subprocess.DEVNULL)
+
+        self.assertTrue(os.access(iso_file, os.R_OK))
+
+        logging.info('iso file %s successfully generated.', iso_file)
+
+    def setUp(self):
+        self._workdir = tempfile.mkdtemp(prefix='acpi-bits-',
+                                         suffix='.tmp')
+
+        logging.info('generated working dir: %s', self._workdir)
+
+        # extract the bits software in the temp working directory
+        bits_zip_file = os.path.join(os.getcwd(), 'prebuilt', 'bits-%d.zip'
+                                     %self._bitsver)
+        grub_tar_file = os.path.join(os.getcwd(), 'prebuilt',
+                                     'bits-%d-grub.tar.gz' %self._bitsver)
+
+        with zipfile.ZipFile(bits_zip_file, 'r') as zref:
+            zref.extractall(self._workdir)
+
+        with tarfile.open(grub_tar_file, 'r') as tarball:
+            tarball.extractall(self._workdir)
+
+        self.copy_test_scripts()
+        self.copy_bits_config()
+        self.generate_bits_iso()
+
+    def parse_log(self):
+        """parse the log generated by running bits tests and
+           check for failures.
+        """
+        debugconf = os.path.join(self._workdir, self._debugcon_log)
+        log = ""
+        with open(debugconf, 'r') as filehandle:
+            log = filehandle.read()
+
+        if os.getenv('V'):
+            print('\nlogs from biosbits follows:')
+            print('==========================================\n\n')
+            print(log)
+            print('\n==========================================\n')
+
+        matchiter = re.finditer(r'(.*Summary: )(\d+ passed), (\d+ failed).*',
+                                log)
+        for match in matchiter:
+            # verify that no test cases failed.
+            self.assertEqual(match.group(3).split()[0], '0',
+                             'Some bits tests seems to have failed. ' \
+                             'Set V=1 in the environment to get the entire ' \
+                             'log from bits.')
+
+    def tearDown(self):
+        if self._vm:
+            self.assertFalse(not self._vm.is_running)
+        logging.info('removing the work directory %s', self._workdir)
+        shutil.rmtree(self._workdir)
+
+    def test_acpi_smbios_bits(self):
+        """The main test case implementaion."""
+
+        qemu_bin = QTESTQEMUPROG
+        iso_file = os.path.join(self._workdir, 'bits-%d.iso' %self._bitsver)
+
+        # QTESTQEMUPROG could be relative to the current directory
+        if not os.access(QTESTQEMUPROG, os.X_OK):
+            qemu_bin = os.path.join(QTEST_PWD, QTESTQEMUPROG)
+
+        logging.info('QEMU binary used: %s', qemu_bin)
+
+        self.assertTrue(os.access(qemu_bin, os.X_OK))
+        self.assertTrue(os.access(iso_file, os.R_OK))
+
+        self._vm = QEMUBitsMachine(binary=qemu_bin,
+                                   base_temp_dir=self._workdir,
+                                   debugcon_log=self._debugcon_log,
+                                   debugcon_addr=self._debugcon_addr)
+
+        self._vm.add_args('-cdrom', '%s' %iso_file)
+
+        args = " ".join(str(arg) for arg in self._vm.base_args()) + \
+            " " + " ".join(str(arg) for arg in self._vm.args)
+
+        logging.info("launching QEMU vm with the following arguments: %s",
+                     args)
+
+        self._vm.launch()
+        # biosbits has been configured to run all the specified test suites
+        # in batch mode and then automatically initiate a vm shutdown.
+        # sleep for maximum of one minute
+        max_sleep_time = time.monotonic() + 60
+        while self._vm.is_running() and time.monotonic() < max_sleep_time:
+            time.sleep(1)
+
+        self.assertFalse(time.monotonic() > max_sleep_time,
+                         'The VM seems to have failed to shutdown in time')
+
+        self.parse_log()
+
+def execute_unittest(argv: List[str], debug: bool = False,
+                     runner: TAPTestRunner = None) -> None:
+    """Executes unittests within the calling module."""
+
+    unittest.main(argv=argv,
+                  testRunner=runner,
+                  verbosity=2 if debug else 1,
+                  warnings=None if sys.warnoptions else 'ignore')
+
+def main():
+    """ The main function where execution begins. """
+
+    assert QTESTQEMUPROG is not None, \
+        "Environment variable QTEST_QEMU_BINARY required."
+
+    runner = TAPTestRunner()
+    runner.set_stream(True)
+    runner.set_format("%s/acpi-bits-test" %ARCH)
+    execute_unittest(sys.argv, False, runner)
+
+main()
diff --git a/tests/qtest/acpi-bits/meson.build b/tests/qtest/acpi-bits/meson.build
new file mode 100644
index 0000000000..95c096914f
--- /dev/null
+++ b/tests/qtest/acpi-bits/meson.build
@@ -0,0 +1,39 @@
+xorriso = find_program('xorriso', required: true)
+if not xorriso.found()
+  message('xorriso not found ... disabled bits acpi tests.')
+  subdir_done()
+endif
+
+# biosbits qtests are currenly only supported on x86_64 platforms.
+qtests_x86_64 += ['acpi-bits-test']
+qtests +=  {'acpi-bits-test' : files('acpi-bits-test.py'),}
+slow_qtests += {'acpi-bits-test': 120, }
+
+subdir('prebuilt')
+subdir('bits-tests')
+subdir('bits-config')
+
+test_files = ['acpi-bits-test.py']
+requirements = 'requirements.txt'
+
+copytestfiles = custom_target('copy test files',
+  input : test_files,
+  output :  test_files,
+  command : ['cp', '@INPUT@', '@OUTPUT@'],
+  install : false,
+  build_by_default : true)
+
+requirementsfiles = custom_target('copy py req files',
+  input : requirements,
+  output : requirements,
+  command : ['cp', '@INPUT@', '@OUTPUT@'],
+  install : false,
+  build_by_default : true)
+
+other_deps += [copytestfiles,requirementsfiles]
+
+qtest_executables += {
+    'acpi-bits-test': configure_file(copy:true,
+                                     input:'acpi-bits-test-venv.sh',
+				     output:'acpi-bits-test')
+}
diff --git a/tests/qtest/acpi-bits/requirements.txt b/tests/qtest/acpi-bits/requirements.txt
new file mode 100644
index 0000000000..00cdad09ef
--- /dev/null
+++ b/tests/qtest/acpi-bits/requirements.txt
@@ -0,0 +1 @@
+tap.py
-- 
2.25.1

Re: [PATCH 09/12] acpi/tests/bits: add acpi and smbios python tests that uses biosbits

[Thomas Huth]

On 27/06/2022 09.28, Ani Sinha wrote:
> This change adds python based qtest framework that can be used to run
> qtests from within a virtual environment. A bash script creates the virtual
> environment and then runs the python based tests from within that environment.
> All dependent python packages are installed in the virtual environment using
> pip module. QEMU python test modules are also available in the environment for
> spawning the QEMU based VMs.
> 
> It also introduces QEMU acpi/smbios biosbits python test script which is run
> from within the python virtual environment.
> 
> Signed-off-by: Ani Sinha <ani@anisinha.ca>
> ---
>   tests/qtest/acpi-bits/acpi-bits-test-venv.sh |  59 ++++
>   tests/qtest/acpi-bits/acpi-bits-test.py      | 327 +++++++++++++++++++
>   tests/qtest/acpi-bits/meson.build            |  39 +++
>   tests/qtest/acpi-bits/requirements.txt       |   1 +
>   4 files changed, 426 insertions(+)
>   create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh
>   create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py
>   create mode 100644 tests/qtest/acpi-bits/meson.build
>   create mode 100644 tests/qtest/acpi-bits/requirements.txt
> 
> diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
> new file mode 100644
> index 0000000000..124e03ce18
> --- /dev/null
> +++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
> @@ -0,0 +1,59 @@
> +#!/usr/bin/env bash
> +# Generates a python virtual environment for the test to run.
> +# Then runs python test scripts from within that virtual environment.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +#
> +# Author: Ani Sinha <ani@anisinha.ca>
> +
> +set -e
> +
> +MYPATH=$(realpath ${BASH_SOURCE:-$0})
> +MYDIR=$(dirname $MYPATH)
> +
> +if [ -z "$QTEST_SOURCE_ROOT" ]; then
> +    echo -n "Please set QTEST_SOURCE_ROOT env pointing"
> +    echo " to the root of the qemu source tree."
> +    echo -n "This is required so that the test can find the "
> +    echo "python modules that it needs for execution."
> +    exit 1
> +fi
> +SRCDIR=$QTEST_SOURCE_ROOT
> +TESTSCRIPTS=("acpi-bits-test.py")
> +PIPCMD="-m pip -q --disable-pip-version-check"
> +# we need to save the old value of PWD before we do a change-dir later
> +QTEST_PWD=$PWD
> +
> +TESTS_PYTHON=/usr/bin/python3
> +TESTS_VENV_REQ=requirements.txt
> +
> +# sadly for pip -e and -t options do not work together.
> +# please see https://github.com/pypa/pip/issues/562
> +cd $MYDIR
> +
> +$TESTS_PYTHON -m venv .
> +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/
> +[ -f $TESTS_VENV_REQ ] && \
> +    $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ
> +
> +# venv is activated at this point.
> +
> +# run the test
> +for testscript in ${TESTSCRIPTS[@]} ; do
> +    export QTEST_PWD; python3 $testscript
> +done
> +
> +cd $QTEST_PWD
> +
> +exit 0
> diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py
> new file mode 100644
> index 0000000000..673567bf8e
> --- /dev/null
> +++ b/tests/qtest/acpi-bits/acpi-bits-test.py
> @@ -0,0 +1,327 @@
> +#!/usr/bin/env python3
> +# group: rw quick
> +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits,
> +# https://biosbits.org/
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +#
> +# Some parts are slightly taken from qtest.py and iotests.py
> +#
> +# Authors:
> +#  Ani Sinha <ani@anisinha.ca>
> +
> +# pylint: disable=invalid-name
> +
> +"""
> +QEMU bios tests using biosbits available at
> +https://biosbits.org/.
> +"""
> +
> +import logging
> +import os
> +import re
> +import shutil
> +import subprocess
> +import sys
> +import tarfile
> +import tempfile
> +import time
> +import unittest
> +import zipfile
> +from typing import (
> +    List,
> +    Optional,
> +    Sequence,
> +)
> +from tap import TAPTestRunner
> +from qemu.machine import QEMUMachine
> +
> +QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY')
> +QTEST_PWD = os.getenv('QTEST_PWD')
> +
> +def get_arch():
> +    """finds the arch from the qemu binary name"""
> +    match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG)
> +    if match:
> +        return match.group(1)
> +    return 'x86_64'
> +
> +ARCH = get_arch()
> +
> +class QEMUBitsMachine(QEMUMachine):
> +    """
> +    A QEMU VM, with isa-debugcon enabled and bits iso passed
> +    using -cdrom to QEMU commandline.
> +    """
> +    def __init__(self,
> +                 binary: str,
> +                 args: Sequence[str] = (),
> +                 wrapper: Sequence[str] = (),
> +                 name: Optional[str] = None,
> +                 base_temp_dir: str = "/var/tmp",
> +                 debugcon_log: str = "debugcon-log.txt",
> +                 debugcon_addr: str = "0x403",
> +                 sock_dir: Optional[str] = None,
> +                 qmp_timer: Optional[float] = None):
> +        # pylint: disable=too-many-arguments
> +
> +        if name is None:
> +            name = "qemu-bits-%d" % os.getpid()
> +        if sock_dir is None:
> +            sock_dir = base_temp_dir
> +        super().__init__(binary, args, wrapper=wrapper, name=name,
> +                         base_temp_dir=base_temp_dir,
> +                         sock_dir=sock_dir, qmp_timer=qmp_timer)
> +        self.debugcon_log = debugcon_log
> +        self.debugcon_addr = debugcon_addr
> +        self.base_temp_dir = base_temp_dir
> +
> +    @property
> +    def _base_args(self) -> List[str]:
> +        args = super()._base_args
> +        args.extend([
> +            '-chardev',
> +            'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir,
> +                                                     self.debugcon_log),
> +            '-device',
> +            'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr,
> +        ])
> +        return args

So is this patch series *really* related to qtests? qtests are using a 
special "accelerator" mode of QEMU where it gets launched with the "-qtest" 
parameter. I can't see that you're using "-qtest" here or anywhere else, so 
this rather looks like another framework to me to run python-based QEMU 
tests (similar to the avocado tests).

Thus if this is really not related to qtests, may I suggest to move this 
into another folder instead? Maybe tests/pytests/acpi-bits or something similar?

  Thomas

Re: [PATCH 09/12] acpi/tests/bits: add acpi and smbios python tests that uses biosbits

[Ani Sinha]

On Tue, Jun 28, 2022 at 12:50 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 27/06/2022 09.28, Ani Sinha wrote:
> > This change adds python based qtest framework that can be used to run
> > qtests from within a virtual environment. A bash script creates the virtual
> > environment and then runs the python based tests from within that environment.
> > All dependent python packages are installed in the virtual environment using
> > pip module. QEMU python test modules are also available in the environment for
> > spawning the QEMU based VMs.
> >
> > It also introduces QEMU acpi/smbios biosbits python test script which is run
> > from within the python virtual environment.
> >
> > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > ---
> >   tests/qtest/acpi-bits/acpi-bits-test-venv.sh |  59 ++++
> >   tests/qtest/acpi-bits/acpi-bits-test.py      | 327 +++++++++++++++++++
> >   tests/qtest/acpi-bits/meson.build            |  39 +++
> >   tests/qtest/acpi-bits/requirements.txt       |   1 +
> >   4 files changed, 426 insertions(+)
> >   create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh
> >   create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py
> >   create mode 100644 tests/qtest/acpi-bits/meson.build
> >   create mode 100644 tests/qtest/acpi-bits/requirements.txt
> >
> > diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
> > new file mode 100644
> > index 0000000000..124e03ce18
> > --- /dev/null
> > +++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
> > @@ -0,0 +1,59 @@
> > +#!/usr/bin/env bash
> > +# Generates a python virtual environment for the test to run.
> > +# Then runs python test scripts from within that virtual environment.
> > +#
> > +# This program is free software; you can redistribute it and/or modify
> > +# it under the terms of the GNU General Public License as published by
> > +# the Free Software Foundation; either version 2 of the License, or
> > +# (at your option) any later version.
> > +#
> > +# This program is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> > +#
> > +# Author: Ani Sinha <ani@anisinha.ca>
> > +
> > +set -e
> > +
> > +MYPATH=$(realpath ${BASH_SOURCE:-$0})
> > +MYDIR=$(dirname $MYPATH)
> > +
> > +if [ -z "$QTEST_SOURCE_ROOT" ]; then
> > +    echo -n "Please set QTEST_SOURCE_ROOT env pointing"
> > +    echo " to the root of the qemu source tree."
> > +    echo -n "This is required so that the test can find the "
> > +    echo "python modules that it needs for execution."
> > +    exit 1
> > +fi
> > +SRCDIR=$QTEST_SOURCE_ROOT
> > +TESTSCRIPTS=("acpi-bits-test.py")
> > +PIPCMD="-m pip -q --disable-pip-version-check"
> > +# we need to save the old value of PWD before we do a change-dir later
> > +QTEST_PWD=$PWD
> > +
> > +TESTS_PYTHON=/usr/bin/python3
> > +TESTS_VENV_REQ=requirements.txt
> > +
> > +# sadly for pip -e and -t options do not work together.
> > +# please see https://github.com/pypa/pip/issues/562
> > +cd $MYDIR
> > +
> > +$TESTS_PYTHON -m venv .
> > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/
> > +[ -f $TESTS_VENV_REQ ] && \
> > +    $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ
> > +
> > +# venv is activated at this point.
> > +
> > +# run the test
> > +for testscript in ${TESTSCRIPTS[@]} ; do
> > +    export QTEST_PWD; python3 $testscript
> > +done
> > +
> > +cd $QTEST_PWD
> > +
> > +exit 0
> > diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py
> > new file mode 100644
> > index 0000000000..673567bf8e
> > --- /dev/null
> > +++ b/tests/qtest/acpi-bits/acpi-bits-test.py
> > @@ -0,0 +1,327 @@
> > +#!/usr/bin/env python3
> > +# group: rw quick
> > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits,
> > +# https://biosbits.org/
> > +#
> > +# This program is free software; you can redistribute it and/or modify
> > +# it under the terms of the GNU General Public License as published by
> > +# the Free Software Foundation; either version 2 of the License, or
> > +# (at your option) any later version.
> > +#
> > +# This program is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> > +#
> > +# Some parts are slightly taken from qtest.py and iotests.py
> > +#
> > +# Authors:
> > +#  Ani Sinha <ani@anisinha.ca>
> > +
> > +# pylint: disable=invalid-name
> > +
> > +"""
> > +QEMU bios tests using biosbits available at
> > +https://biosbits.org/.
> > +"""
> > +
> > +import logging
> > +import os
> > +import re
> > +import shutil
> > +import subprocess
> > +import sys
> > +import tarfile
> > +import tempfile
> > +import time
> > +import unittest
> > +import zipfile
> > +from typing import (
> > +    List,
> > +    Optional,
> > +    Sequence,
> > +)
> > +from tap import TAPTestRunner
> > +from qemu.machine import QEMUMachine
> > +
> > +QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY')
> > +QTEST_PWD = os.getenv('QTEST_PWD')
> > +
> > +def get_arch():
> > +    """finds the arch from the qemu binary name"""
> > +    match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG)
> > +    if match:
> > +        return match.group(1)
> > +    return 'x86_64'
> > +
> > +ARCH = get_arch()
> > +
> > +class QEMUBitsMachine(QEMUMachine):
> > +    """
> > +    A QEMU VM, with isa-debugcon enabled and bits iso passed
> > +    using -cdrom to QEMU commandline.
> > +    """
> > +    def __init__(self,
> > +                 binary: str,
> > +                 args: Sequence[str] = (),
> > +                 wrapper: Sequence[str] = (),
> > +                 name: Optional[str] = None,
> > +                 base_temp_dir: str = "/var/tmp",
> > +                 debugcon_log: str = "debugcon-log.txt",
> > +                 debugcon_addr: str = "0x403",
> > +                 sock_dir: Optional[str] = None,
> > +                 qmp_timer: Optional[float] = None):
> > +        # pylint: disable=too-many-arguments
> > +
> > +        if name is None:
> > +            name = "qemu-bits-%d" % os.getpid()
> > +        if sock_dir is None:
> > +            sock_dir = base_temp_dir
> > +        super().__init__(binary, args, wrapper=wrapper, name=name,
> > +                         base_temp_dir=base_temp_dir,
> > +                         sock_dir=sock_dir, qmp_timer=qmp_timer)
> > +        self.debugcon_log = debugcon_log
> > +        self.debugcon_addr = debugcon_addr
> > +        self.base_temp_dir = base_temp_dir
> > +
> > +    @property
> > +    def _base_args(self) -> List[str]:
> > +        args = super()._base_args
> > +        args.extend([
> > +            '-chardev',
> > +            'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir,
> > +                                                     self.debugcon_log),
> > +            '-device',
> > +            'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr,
> > +        ])
> > +        return args
>
> So is this patch series *really* related to qtests? qtests are using a
> special "accelerator" mode of QEMU where it gets launched with the "-qtest"
> parameter. I can't see that you're using "-qtest" here or anywhere else, so
> this rather looks like another framework to me to run python-based QEMU
> tests (similar to the avocado tests).

yes you are right. This does not use or need the qtest accelerator
because we are not inspecting the guest memory for anything.

>
> Thus if this is really not related to qtests, may I suggest to move this
> into another folder instead? Maybe tests/pytests/acpi-bits or something similar?

The problem I faced with this test is that it does not quite fall into
the qtest category. Nor does it fall into the integration test
category. I asked Igor and he suggested I use the qtest framework.
Should we invent a new class of tests then? How many such tests are we
going to have in the future?  Lets see what others think.

>
>   Thomas
>

Re: [PATCH 09/12] acpi/tests/bits: add acpi and smbios python tests that uses biosbits

[Thomas Huth]

On 28/06/2022 09.26, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 12:50 PM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 27/06/2022 09.28, Ani Sinha wrote:
>>> This change adds python based qtest framework that can be used to run
>>> qtests from within a virtual environment. A bash script creates the virtual
>>> environment and then runs the python based tests from within that environment.
>>> All dependent python packages are installed in the virtual environment using
>>> pip module. QEMU python test modules are also available in the environment for
>>> spawning the QEMU based VMs.
>>>
>>> It also introduces QEMU acpi/smbios biosbits python test script which is run
>>> from within the python virtual environment.
>>>
>>> Signed-off-by: Ani Sinha <ani@anisinha.ca>
>>> ---
>>>    tests/qtest/acpi-bits/acpi-bits-test-venv.sh |  59 ++++
>>>    tests/qtest/acpi-bits/acpi-bits-test.py      | 327 +++++++++++++++++++
>>>    tests/qtest/acpi-bits/meson.build            |  39 +++
>>>    tests/qtest/acpi-bits/requirements.txt       |   1 +
>>>    4 files changed, 426 insertions(+)
>>>    create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh
>>>    create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py
>>>    create mode 100644 tests/qtest/acpi-bits/meson.build
>>>    create mode 100644 tests/qtest/acpi-bits/requirements.txt
>>>
>>> diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
>>> new file mode 100644
>>> index 0000000000..124e03ce18
>>> --- /dev/null
>>> +++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
>>> @@ -0,0 +1,59 @@
>>> +#!/usr/bin/env bash
>>> +# Generates a python virtual environment for the test to run.
>>> +# Then runs python test scripts from within that virtual environment.
>>> +#
>>> +# This program is free software; you can redistribute it and/or modify
>>> +# it under the terms of the GNU General Public License as published by
>>> +# the Free Software Foundation; either version 2 of the License, or
>>> +# (at your option) any later version.
>>> +#
>>> +# This program is distributed in the hope that it will be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> +# GNU General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU General Public License
>>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> +#
>>> +# Author: Ani Sinha <ani@anisinha.ca>
>>> +
>>> +set -e
>>> +
>>> +MYPATH=$(realpath ${BASH_SOURCE:-$0})
>>> +MYDIR=$(dirname $MYPATH)
>>> +
>>> +if [ -z "$QTEST_SOURCE_ROOT" ]; then
>>> +    echo -n "Please set QTEST_SOURCE_ROOT env pointing"
>>> +    echo " to the root of the qemu source tree."
>>> +    echo -n "This is required so that the test can find the "
>>> +    echo "python modules that it needs for execution."
>>> +    exit 1
>>> +fi
>>> +SRCDIR=$QTEST_SOURCE_ROOT
>>> +TESTSCRIPTS=("acpi-bits-test.py")
>>> +PIPCMD="-m pip -q --disable-pip-version-check"
>>> +# we need to save the old value of PWD before we do a change-dir later
>>> +QTEST_PWD=$PWD
>>> +
>>> +TESTS_PYTHON=/usr/bin/python3
>>> +TESTS_VENV_REQ=requirements.txt
>>> +
>>> +# sadly for pip -e and -t options do not work together.
>>> +# please see https://github.com/pypa/pip/issues/562
>>> +cd $MYDIR
>>> +
>>> +$TESTS_PYTHON -m venv .
>>> +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/
>>> +[ -f $TESTS_VENV_REQ ] && \
>>> +    $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ
>>> +
>>> +# venv is activated at this point.
>>> +
>>> +# run the test
>>> +for testscript in ${TESTSCRIPTS[@]} ; do
>>> +    export QTEST_PWD; python3 $testscript
>>> +done
>>> +
>>> +cd $QTEST_PWD
>>> +
>>> +exit 0
>>> diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py
>>> new file mode 100644
>>> index 0000000000..673567bf8e
>>> --- /dev/null
>>> +++ b/tests/qtest/acpi-bits/acpi-bits-test.py
>>> @@ -0,0 +1,327 @@
>>> +#!/usr/bin/env python3
>>> +# group: rw quick
>>> +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits,
>>> +# https://biosbits.org/
>>> +#
>>> +# This program is free software; you can redistribute it and/or modify
>>> +# it under the terms of the GNU General Public License as published by
>>> +# the Free Software Foundation; either version 2 of the License, or
>>> +# (at your option) any later version.
>>> +#
>>> +# This program is distributed in the hope that it will be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> +# GNU General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU General Public License
>>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> +#
>>> +# Some parts are slightly taken from qtest.py and iotests.py
>>> +#
>>> +# Authors:
>>> +#  Ani Sinha <ani@anisinha.ca>
>>> +
>>> +# pylint: disable=invalid-name
>>> +
>>> +"""
>>> +QEMU bios tests using biosbits available at
>>> +https://biosbits.org/.
>>> +"""
>>> +
>>> +import logging
>>> +import os
>>> +import re
>>> +import shutil
>>> +import subprocess
>>> +import sys
>>> +import tarfile
>>> +import tempfile
>>> +import time
>>> +import unittest
>>> +import zipfile
>>> +from typing import (
>>> +    List,
>>> +    Optional,
>>> +    Sequence,
>>> +)
>>> +from tap import TAPTestRunner
>>> +from qemu.machine import QEMUMachine
>>> +
>>> +QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY')
>>> +QTEST_PWD = os.getenv('QTEST_PWD')
>>> +
>>> +def get_arch():
>>> +    """finds the arch from the qemu binary name"""
>>> +    match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG)
>>> +    if match:
>>> +        return match.group(1)
>>> +    return 'x86_64'
>>> +
>>> +ARCH = get_arch()
>>> +
>>> +class QEMUBitsMachine(QEMUMachine):
>>> +    """
>>> +    A QEMU VM, with isa-debugcon enabled and bits iso passed
>>> +    using -cdrom to QEMU commandline.
>>> +    """
>>> +    def __init__(self,
>>> +                 binary: str,
>>> +                 args: Sequence[str] = (),
>>> +                 wrapper: Sequence[str] = (),
>>> +                 name: Optional[str] = None,
>>> +                 base_temp_dir: str = "/var/tmp",
>>> +                 debugcon_log: str = "debugcon-log.txt",
>>> +                 debugcon_addr: str = "0x403",
>>> +                 sock_dir: Optional[str] = None,
>>> +                 qmp_timer: Optional[float] = None):
>>> +        # pylint: disable=too-many-arguments
>>> +
>>> +        if name is None:
>>> +            name = "qemu-bits-%d" % os.getpid()
>>> +        if sock_dir is None:
>>> +            sock_dir = base_temp_dir
>>> +        super().__init__(binary, args, wrapper=wrapper, name=name,
>>> +                         base_temp_dir=base_temp_dir,
>>> +                         sock_dir=sock_dir, qmp_timer=qmp_timer)
>>> +        self.debugcon_log = debugcon_log
>>> +        self.debugcon_addr = debugcon_addr
>>> +        self.base_temp_dir = base_temp_dir
>>> +
>>> +    @property
>>> +    def _base_args(self) -> List[str]:
>>> +        args = super()._base_args
>>> +        args.extend([
>>> +            '-chardev',
>>> +            'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir,
>>> +                                                     self.debugcon_log),
>>> +            '-device',
>>> +            'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr,
>>> +        ])
>>> +        return args
>>
>> So is this patch series *really* related to qtests? qtests are using a
>> special "accelerator" mode of QEMU where it gets launched with the "-qtest"
>> parameter. I can't see that you're using "-qtest" here or anywhere else, so
>> this rather looks like another framework to me to run python-based QEMU
>> tests (similar to the avocado tests).
> 
> yes you are right. This does not use or need the qtest accelerator
> because we are not inspecting the guest memory for anything.
> 
>>
>> Thus if this is really not related to qtests, may I suggest to move this
>> into another folder instead? Maybe tests/pytests/acpi-bits or something similar?
> 
> The problem I faced with this test is that it does not quite fall into
> the qtest category. Nor does it fall into the integration test
> category. I asked Igor and he suggested I use the qtest framework.

My python-foo really lacks, so putting python stuff under tests/qtest where 
I'm the maintainer is a really bad idea ;-)

> Should we invent a new class of tests then? How many such tests are we
> going to have in the future?  Lets see what others think.

I think we want to have another folder under tests/ for python-based tests 
that do not use the avocado framework. John is working on better python 
bindings, and since we switched almost all other tests to the meson test 
runner now, the avocado-based tests (with their own test runner) rather feel 
like an oddball in the qemu test suite now, so I asume we will have more 
python-based tests in the future which will work without avocado.

The difficult question is - as always - to come up with a good name...

  - tests/pytest/
  - tests/integration/
  - tests/pyqmygoose/
  - ...

... I don't have a real good clue, as you can see ;-)

  Thomas

Re: [PATCH 09/12] acpi/tests/bits: add acpi and smbios python tests that uses biosbits

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 12:56:52PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 12:50 PM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 27/06/2022 09.28, Ani Sinha wrote:
> > > This change adds python based qtest framework that can be used to run
> > > qtests from within a virtual environment. A bash script creates the virtual
> > > environment and then runs the python based tests from within that environment.
> > > All dependent python packages are installed in the virtual environment using
> > > pip module. QEMU python test modules are also available in the environment for
> > > spawning the QEMU based VMs.
> > >
> > > It also introduces QEMU acpi/smbios biosbits python test script which is run
> > > from within the python virtual environment.
> > >
> > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > ---
> > >   tests/qtest/acpi-bits/acpi-bits-test-venv.sh |  59 ++++
> > >   tests/qtest/acpi-bits/acpi-bits-test.py      | 327 +++++++++++++++++++
> > >   tests/qtest/acpi-bits/meson.build            |  39 +++
> > >   tests/qtest/acpi-bits/requirements.txt       |   1 +
> > >   4 files changed, 426 insertions(+)
> > >   create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh
> > >   create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py
> > >   create mode 100644 tests/qtest/acpi-bits/meson.build
> > >   create mode 100644 tests/qtest/acpi-bits/requirements.txt
> > >
> > > diff --git a/tests/qtest/acpi-bits/acpi-bits-test-venv.sh b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
> > > new file mode 100644
> > > index 0000000000..124e03ce18
> > > --- /dev/null
> > > +++ b/tests/qtest/acpi-bits/acpi-bits-test-venv.sh
> > > @@ -0,0 +1,59 @@
> > > +#!/usr/bin/env bash
> > > +# Generates a python virtual environment for the test to run.
> > > +# Then runs python test scripts from within that virtual environment.
> > > +#
> > > +# This program is free software; you can redistribute it and/or modify
> > > +# it under the terms of the GNU General Public License as published by
> > > +# the Free Software Foundation; either version 2 of the License, or
> > > +# (at your option) any later version.
> > > +#
> > > +# This program is distributed in the hope that it will be useful,
> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > > +# GNU General Public License for more details.
> > > +#
> > > +# You should have received a copy of the GNU General Public License
> > > +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> > > +#
> > > +# Author: Ani Sinha <ani@anisinha.ca>
> > > +
> > > +set -e
> > > +
> > > +MYPATH=$(realpath ${BASH_SOURCE:-$0})
> > > +MYDIR=$(dirname $MYPATH)
> > > +
> > > +if [ -z "$QTEST_SOURCE_ROOT" ]; then
> > > +    echo -n "Please set QTEST_SOURCE_ROOT env pointing"
> > > +    echo " to the root of the qemu source tree."
> > > +    echo -n "This is required so that the test can find the "
> > > +    echo "python modules that it needs for execution."
> > > +    exit 1
> > > +fi
> > > +SRCDIR=$QTEST_SOURCE_ROOT
> > > +TESTSCRIPTS=("acpi-bits-test.py")
> > > +PIPCMD="-m pip -q --disable-pip-version-check"
> > > +# we need to save the old value of PWD before we do a change-dir later
> > > +QTEST_PWD=$PWD
> > > +
> > > +TESTS_PYTHON=/usr/bin/python3
> > > +TESTS_VENV_REQ=requirements.txt
> > > +
> > > +# sadly for pip -e and -t options do not work together.
> > > +# please see https://github.com/pypa/pip/issues/562
> > > +cd $MYDIR
> > > +
> > > +$TESTS_PYTHON -m venv .
> > > +$TESTS_PYTHON $PIPCMD install -e $SRCDIR/python/
> > > +[ -f $TESTS_VENV_REQ ] && \
> > > +    $TESTS_PYTHON $PIPCMD install -r $TESTS_VENV_REQ
> > > +
> > > +# venv is activated at this point.
> > > +
> > > +# run the test
> > > +for testscript in ${TESTSCRIPTS[@]} ; do
> > > +    export QTEST_PWD; python3 $testscript
> > > +done
> > > +
> > > +cd $QTEST_PWD
> > > +
> > > +exit 0
> > > diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py
> > > new file mode 100644
> > > index 0000000000..673567bf8e
> > > --- /dev/null
> > > +++ b/tests/qtest/acpi-bits/acpi-bits-test.py
> > > @@ -0,0 +1,327 @@
> > > +#!/usr/bin/env python3
> > > +# group: rw quick
> > > +# Exercize QEMU generated ACPI/SMBIOS tables using biosbits,
> > > +# https://biosbits.org/
> > > +#
> > > +# This program is free software; you can redistribute it and/or modify
> > > +# it under the terms of the GNU General Public License as published by
> > > +# the Free Software Foundation; either version 2 of the License, or
> > > +# (at your option) any later version.
> > > +#
> > > +# This program is distributed in the hope that it will be useful,
> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > > +# GNU General Public License for more details.
> > > +#
> > > +# You should have received a copy of the GNU General Public License
> > > +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> > > +#
> > > +# Some parts are slightly taken from qtest.py and iotests.py
> > > +#
> > > +# Authors:
> > > +#  Ani Sinha <ani@anisinha.ca>
> > > +
> > > +# pylint: disable=invalid-name
> > > +
> > > +"""
> > > +QEMU bios tests using biosbits available at
> > > +https://biosbits.org/.
> > > +"""
> > > +
> > > +import logging
> > > +import os
> > > +import re
> > > +import shutil
> > > +import subprocess
> > > +import sys
> > > +import tarfile
> > > +import tempfile
> > > +import time
> > > +import unittest
> > > +import zipfile
> > > +from typing import (
> > > +    List,
> > > +    Optional,
> > > +    Sequence,
> > > +)
> > > +from tap import TAPTestRunner
> > > +from qemu.machine import QEMUMachine
> > > +
> > > +QTESTQEMUPROG = os.getenv('QTEST_QEMU_BINARY')
> > > +QTEST_PWD = os.getenv('QTEST_PWD')
> > > +
> > > +def get_arch():
> > > +    """finds the arch from the qemu binary name"""
> > > +    match = re.search('.*qemu-system-(.*)', QTESTQEMUPROG)
> > > +    if match:
> > > +        return match.group(1)
> > > +    return 'x86_64'
> > > +
> > > +ARCH = get_arch()
> > > +
> > > +class QEMUBitsMachine(QEMUMachine):
> > > +    """
> > > +    A QEMU VM, with isa-debugcon enabled and bits iso passed
> > > +    using -cdrom to QEMU commandline.
> > > +    """
> > > +    def __init__(self,
> > > +                 binary: str,
> > > +                 args: Sequence[str] = (),
> > > +                 wrapper: Sequence[str] = (),
> > > +                 name: Optional[str] = None,
> > > +                 base_temp_dir: str = "/var/tmp",
> > > +                 debugcon_log: str = "debugcon-log.txt",
> > > +                 debugcon_addr: str = "0x403",
> > > +                 sock_dir: Optional[str] = None,
> > > +                 qmp_timer: Optional[float] = None):
> > > +        # pylint: disable=too-many-arguments
> > > +
> > > +        if name is None:
> > > +            name = "qemu-bits-%d" % os.getpid()
> > > +        if sock_dir is None:
> > > +            sock_dir = base_temp_dir
> > > +        super().__init__(binary, args, wrapper=wrapper, name=name,
> > > +                         base_temp_dir=base_temp_dir,
> > > +                         sock_dir=sock_dir, qmp_timer=qmp_timer)
> > > +        self.debugcon_log = debugcon_log
> > > +        self.debugcon_addr = debugcon_addr
> > > +        self.base_temp_dir = base_temp_dir
> > > +
> > > +    @property
> > > +    def _base_args(self) -> List[str]:
> > > +        args = super()._base_args
> > > +        args.extend([
> > > +            '-chardev',
> > > +            'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir,
> > > +                                                     self.debugcon_log),
> > > +            '-device',
> > > +            'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr,
> > > +        ])
> > > +        return args
> >
> > So is this patch series *really* related to qtests? qtests are using a
> > special "accelerator" mode of QEMU where it gets launched with the "-qtest"
> > parameter. I can't see that you're using "-qtest" here or anywhere else, so
> > this rather looks like another framework to me to run python-based QEMU
> > tests (similar to the avocado tests).
> 
> yes you are right. This does not use or need the qtest accelerator
> because we are not inspecting the guest memory for anything.

Same is true for e.g. ./tests/qtest/bios-tables-test.c ...


> >
> > Thus if this is really not related to qtests, may I suggest to move this
> > into another folder instead? Maybe tests/pytests/acpi-bits or something similar?
> 
> The problem I faced with this test is that it does not quite fall into
> the qtest category. Nor does it fall into the integration test
> category. I asked Igor and he suggested I use the qtest framework.
> Should we invent a new class of tests then? How many such tests are we
> going to have in the future?  Lets see what others think.
> 
> >
> >   Thomas
> >

Re: [PATCH 09/12] acpi/tests/bits: add acpi and smbios python tests that uses biosbits

[Thomas Huth]

On 28/06/2022 11.55, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 12:56:52PM +0530, Ani Sinha wrote:
>> On Tue, Jun 28, 2022 at 12:50 PM Thomas Huth <thuth@redhat.com> wrote:
>>>
>>> On 27/06/2022 09.28, Ani Sinha wrote:
>>>> This change adds python based qtest framework that can be used to run
>>>> qtests from within a virtual environment. A bash script creates the virtual
>>>> environment and then runs the python based tests from within that environment.
>>>> All dependent python packages are installed in the virtual environment using
>>>> pip module. QEMU python test modules are also available in the environment for
>>>> spawning the QEMU based VMs.
>>>>
>>>> It also introduces QEMU acpi/smbios biosbits python test script which is run
>>>> from within the python virtual environment.
>>>>
>>>> Signed-off-by: Ani Sinha <ani@anisinha.ca>
>>>> ---
[...]
>>>> +class QEMUBitsMachine(QEMUMachine):
>>>> +    """
>>>> +    A QEMU VM, with isa-debugcon enabled and bits iso passed
>>>> +    using -cdrom to QEMU commandline.
>>>> +    """
>>>> +    def __init__(self,
>>>> +                 binary: str,
>>>> +                 args: Sequence[str] = (),
>>>> +                 wrapper: Sequence[str] = (),
>>>> +                 name: Optional[str] = None,
>>>> +                 base_temp_dir: str = "/var/tmp",
>>>> +                 debugcon_log: str = "debugcon-log.txt",
>>>> +                 debugcon_addr: str = "0x403",
>>>> +                 sock_dir: Optional[str] = None,
>>>> +                 qmp_timer: Optional[float] = None):
>>>> +        # pylint: disable=too-many-arguments
>>>> +
>>>> +        if name is None:
>>>> +            name = "qemu-bits-%d" % os.getpid()
>>>> +        if sock_dir is None:
>>>> +            sock_dir = base_temp_dir
>>>> +        super().__init__(binary, args, wrapper=wrapper, name=name,
>>>> +                         base_temp_dir=base_temp_dir,
>>>> +                         sock_dir=sock_dir, qmp_timer=qmp_timer)
>>>> +        self.debugcon_log = debugcon_log
>>>> +        self.debugcon_addr = debugcon_addr
>>>> +        self.base_temp_dir = base_temp_dir
>>>> +
>>>> +    @property
>>>> +    def _base_args(self) -> List[str]:
>>>> +        args = super()._base_args
>>>> +        args.extend([
>>>> +            '-chardev',
>>>> +            'file,path=%s,id=debugcon' %os.path.join(self.base_temp_dir,
>>>> +                                                     self.debugcon_log),
>>>> +            '-device',
>>>> +            'isa-debugcon,iobase=%s,chardev=debugcon' %self.debugcon_addr,
>>>> +        ])
>>>> +        return args
>>>
>>> So is this patch series *really* related to qtests? qtests are using a
>>> special "accelerator" mode of QEMU where it gets launched with the "-qtest"
>>> parameter. I can't see that you're using "-qtest" here or anywhere else, so
>>> this rather looks like another framework to me to run python-based QEMU
>>> tests (similar to the avocado tests).
>>
>> yes you are right. This does not use or need the qtest accelerator
>> because we are not inspecting the guest memory for anything.
> 
> Same is true for e.g. ./tests/qtest/bios-tables-test.c ...

No, bios-tables-test still uses functions like qtest_memread() and 
qtest_readb(), so it still uses the qtest framework, even if it's actually 
running code via TCG.

  Thomas

[PATCH 10/12] acpi/tests/bits: add acpi bits qtest directory in meson for running tests

[Ani Sinha]

added acpi-bits subdirectory in meson.build so that the tests in that
subdirectory can be run as a part of make check.

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/acpi-bits/acpi-bits-test.py | 4 ++--
 tests/qtest/meson.build                 | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/qtest/acpi-bits/acpi-bits-test.py b/tests/qtest/acpi-bits/acpi-bits-test.py
index 673567bf8e..ef4ace8028 100644
--- a/tests/qtest/acpi-bits/acpi-bits-test.py
+++ b/tests/qtest/acpi-bits/acpi-bits-test.py
@@ -244,9 +244,9 @@ def parse_log(self):
 
         if os.getenv('V'):
             print('\nlogs from biosbits follows:')
-            print('==========================================\n\n')
+            print('==========================================\n')
             print(log)
-            print('\n==========================================\n')
+            print('==========================================\n')
 
         matchiter = re.finditer(r'(.*Summary: )(\d+ passed), (\d+ failed).*',
                                 log)
diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index ad52f1c81b..2c72c42933 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -312,6 +312,8 @@ endif
 qtest_executables = {}
 other_deps = []
 
+subdir('acpi-bits')
+
 foreach dir : target_dirs
   if not dir.endswith('-softmmu')
     continue
-- 
2.25.1

[PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

The README file is added describing the directory structure and the purpose
of every file it contains. It also describes how to add new tests, make changes
to existing tests or bits config files or regenerate the bits software.

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
 1 file changed, 168 insertions(+)
 create mode 100644 tests/qtest/acpi-bits/README

diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
new file mode 100644
index 0000000000..97b15f1665
--- /dev/null
+++ b/tests/qtest/acpi-bits/README
@@ -0,0 +1,168 @@
+=============================================================================
+ACPI/SMBIOS QTESTS USING BIOSBITS
+=============================================================================
+
+Biosbits is a software written by Josh Triplett that can be downloaded by
+visiting https://biosbits.org/. The github codebase can be found here:
+https://github.com/biosbits/bits/tree/master. It is a software that exercizes
+the bios components such as acpi and smbios tables directly through acpica
+bios interpreter (a freely available C based library written by Intel,
+downloadable from https://acpica.org/ and is included with biosbits) without an
+operating system getting involved in between.
+There are several advantages to directly testing the bios in a real physical
+machine or VM as opposed to indirectly discovering bios issues through the
+operating system. For one thing, the OSes tend to hide bios problems from the
+end user. The other is that we have more control of what we wanted to test
+and how by directly using acpica interpreter on top of the bios on a running
+system. More details on the inspiration for developing biosbits and its real
+life uses can be found in (a) and (b).
+This directory contains QEMU qtests written in python that exercizes the QEMU
+bios components using biosbits and reports test failures.
+
+These tests use python virtual environment. In debian/ubuntu system, the tests
+would require python3.8-venv and python3-pip packages to be installed.
+
+A brief description of the contents of this directory follows:
+
+├── acpi-bits-test.py
+├── acpi-bits-test-venv.sh
+├── bits-config
+│   ├── bits-cfg.txt
+│   └── meson.build
+├── bits-tests
+│   ├── meson.build
+│   ├── smbios.py
+│   ├── smilatency.py
+│   ├── testacpi.py
+│   └── testcpuid.py
+├── meson.build
+├── prebuilt
+│   ├── bits-2095-grub.tar.gz
+│   ├── bits-2095.zip
+│   └── meson.build
+├── README
+└── requirements.txt
+
+acpi-bits:
+ - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
+   environment necessary for the actual python test script to run. Amongst
+   other things, it makes sure that QEMU python library is available within
+   that venv so that QEMU machines can be forked. QEMU python library can be
+   found within python/ directory in QEMU source.
+   After setting up the virtual environment, it runs the python test script
+   from within that environment.
+   If you want to enable verbose mode only for bits test and run make check,
+   one trick is to add V=1 before the call to execute the python script in
+   this file.
+ - acpi-bits-test.py: This is the main python test script that generates a
+   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
+   test failures. This is the script one would be interested in if they wanted
+   to add or change some component of the log parsing, add a new commandline to
+   how QEMU is spawned etc. Test writers typically would not need to modify
+   this script unless they wanted to enhance or change the log parsing for
+   their tests.
+ - requirements.txt: This text file contains the dependent python libraries
+   needed for the tests to run. If a new dependent library is needed, it would
+   be added here as a new entry and then acpi-bits-test-venv.sh would install
+   it when setting up the python virtual environment using pip.
+ - README: This text file.
+
+acpi-bits/bits-config:
+   This location contains biosbits config files that determine how the software
+   runs the tests.
+ - bits-config.txt: this is the biosbits config file that determines what tests
+   or actions are performed by bits. The description of the config options are
+   provided in the file itself.
+
+acpi-bits/prebuilt:
+   This location contains prebuilt biosbits binaries that are used to generate
+   the bits iso file for testing.
+ - bits-2095.zip: The contents from this zip archive are the main contents of
+    the iso file that are used for testing. This binary zip archive also
+    contains the full source of the bits software including the full text of
+    the license agreement listed here:
+    https://github.com/biosbits/bits/blob/master/COPYING
+    The source tarball can be found in this location in the zip file:
+    boot/src/bits-2095.tar.gz
+    The additional changes beyond those that are present in the official
+    biosbits github repository can be found here:
+    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
+
+    Basically these changes can be categorized into the following:
+    (a) changes to make sure biosbits builds with the latest gcc compiler
+    (gcc 9.4).
+    (b) upgrade of acpica to the latest version (march 2022).
+    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
+    debugcon can be used to collect the logs.
+    (d) send a gub halt command to shutdown the VM once all the tests have been
+    executed.
+
+    This zip file is automatically generated by the bits build system. It can
+    be found in the bits build directory and it is suffixed by the bits build
+    number.
+    Normally, there is no need to make any alteration to this zip archive
+    unless one wanted to change the bits software itself (for example to add
+    a new config option or change actions of existing options or change the
+    debug IO port etc). When such changes are made and a new biosbits software
+    is needed to be generated, please refresh this zip archive as well as the
+    grub tarball at the same time. Tests will break if changes are made to bits
+    that are incompatible with existing behavior. So care must be taken to make
+    sure that the change is compatible with existing bits software as much as
+    possible. When a new config option is introduced for example, bits must
+    be upadated here first before introducing the new config option through
+    a later commit.
+ - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
+    scripts and modules to the prebuilt directory. These prebuilt grub
+    artifacts are required in order to generate a bootable bits iso file that
+    can run tests.
+    In order to generate this tar archive, please put the following two
+    directories that can be found in the bits build directory in a single
+    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
+    bits that generated the archive:
+
+    grub-inst-x86_64-efi
+    grub-inst
+
+    This version should be the same as the version number of bits that generated
+    bits-<n>.zip file. In other words, the two files must be in sync and should
+    come from the same build of biosbits of the same version. Things may still
+    work if they come from different versions but mixing version numbers are
+    not recommended.
+    There is normally no need to regenerate this archive unless some fixes or
+    changes have gone into the grub that are part of biosbits.
+
+acpi-bits/bits-tests:
+   This directory contains biosbits python based tests that are run from within
+   the biosbits environment in the spawned VM. New additions of test cases can
+   be made in the appropriate test file. For example, new acpi tests can go
+   into testacpi.py and one would call testsuite.add_test() to register the new
+   test so that it gets executed as a part of the ACPI tests.
+   It might be occassionally necessary to disable some subtests or add a new
+   test that belongs to a test suite not already present in this directory. To
+   do this, please extract the bits source from the zip file mentioned above.
+   Copy the test suite/script that needs modification (addition of new tests
+   or disabling them) from boot/python location of the extracted directory
+   into this directory.
+
+   step (a): copy unmodified test script to this directory.
+   step (b): update meson.build and add this file to the list.
+   Commit (a) and (b) together in the same commit.
+
+   step (c): perform modifications to the test.
+   Commit (c) in a separate commit.
+
+   The test framework will then use your modified test script to run the test.
+   No further changes would be needed. Please check the logs to make sure that
+   appropriate changes have taken effect.
+
+meson.build files makes sure that the bits qtests are appropriately integrated
+into the QEMU qtest framework and are run with "make check-qtest".
+Currently, the bits test is configured to run only for x86_64 architectures. Bits
+has been built only for this architecture.
+
+
+Author: Ani Sinha <ani@anisinha.ca>
+
+References:
+(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
+(b) https://www.youtube.com/watch?v=36QIepyUuhg
-- 
2.25.1

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Michael S. Tsirkin]

On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> The README file is added describing the directory structure and the purpose
> of every file it contains. It also describes how to add new tests, make changes
> to existing tests or bits config files or regenerate the bits software.
> 
> Signed-off-by: Ani Sinha <ani@anisinha.ca>
> ---
>  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
>  1 file changed, 168 insertions(+)
>  create mode 100644 tests/qtest/acpi-bits/README
> 
> diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> new file mode 100644
> index 0000000000..97b15f1665
> --- /dev/null
> +++ b/tests/qtest/acpi-bits/README
> @@ -0,0 +1,168 @@
> +=============================================================================
> +ACPI/SMBIOS QTESTS USING BIOSBITS
> +=============================================================================
> +
> +Biosbits is a software written by Josh Triplett that can be downloaded by
> +visiting https://biosbits.org/. The github codebase can be found here:
> +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> +the bios components such as acpi and smbios tables directly through acpica
> +bios interpreter (a freely available C based library written by Intel,
> +downloadable from https://acpica.org/ and is included with biosbits) without an
> +operating system getting involved in between.
> +There are several advantages to directly testing the bios in a real physical
> +machine or VM as opposed to indirectly discovering bios issues through the
> +operating system. For one thing, the OSes tend to hide bios problems from the
> +end user. The other is that we have more control of what we wanted to test
> +and how by directly using acpica interpreter on top of the bios on a running
> +system. More details on the inspiration for developing biosbits and its real
> +life uses can be found in (a) and (b).
> +This directory contains QEMU qtests written in python that exercizes the QEMU
> +bios components using biosbits and reports test failures.
> +
> +These tests use python virtual environment. In debian/ubuntu system, the tests
> +would require python3.8-venv and python3-pip packages to be installed.

Why do we mess with venv and pip? Certainly possible but
what's wrong with using distro provided packages?

> +
> +A brief description of the contents of this directory follows:
> +
> +├── acpi-bits-test.py
> +├── acpi-bits-test-venv.sh
> +├── bits-config
> +│   ├── bits-cfg.txt
> +│   └── meson.build
> +├── bits-tests
> +│   ├── meson.build
> +│   ├── smbios.py
> +│   ├── smilatency.py
> +│   ├── testacpi.py
> +│   └── testcpuid.py
> +├── meson.build
> +├── prebuilt
> +│   ├── bits-2095-grub.tar.gz
> +│   ├── bits-2095.zip
> +│   └── meson.build
> +├── README
> +└── requirements.txt
> +
> +acpi-bits:
> + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> +   environment necessary for the actual python test script to run. Amongst
> +   other things, it makes sure that QEMU python library is available within
> +   that venv so that QEMU machines can be forked. QEMU python library can be
> +   found within python/ directory in QEMU source.
> +   After setting up the virtual environment, it runs the python test script
> +   from within that environment.
> +   If you want to enable verbose mode only for bits test and run make check,
> +   one trick is to add V=1 before the call to execute the python script in
> +   this file.
> + - acpi-bits-test.py: This is the main python test script that generates a
> +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> +   test failures. This is the script one would be interested in if they wanted
> +   to add or change some component of the log parsing, add a new commandline to
> +   how QEMU is spawned etc. Test writers typically would not need to modify
> +   this script unless they wanted to enhance or change the log parsing for
> +   their tests.
> + - requirements.txt: This text file contains the dependent python libraries
> +   needed for the tests to run. If a new dependent library is needed, it would
> +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> +   it when setting up the python virtual environment using pip.
> + - README: This text file.
> +
> +acpi-bits/bits-config:
> +   This location contains biosbits config files that determine how the software
> +   runs the tests.
> + - bits-config.txt: this is the biosbits config file that determines what tests
> +   or actions are performed by bits. The description of the config options are
> +   provided in the file itself.
> +
> +acpi-bits/prebuilt:
> +   This location contains prebuilt biosbits binaries that are used to generate
> +   the bits iso file for testing.
> + - bits-2095.zip: The contents from this zip archive are the main contents of
> +    the iso file that are used for testing. This binary zip archive also
> +    contains the full source of the bits software including the full text of
> +    the license agreement listed here:
> +    https://github.com/biosbits/bits/blob/master/COPYING
> +    The source tarball can be found in this location in the zip file:
> +    boot/src/bits-2095.tar.gz
> +    The additional changes beyond those that are present in the official
> +    biosbits github repository can be found here:
> +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> +
> +    Basically these changes can be categorized into the following:
> +    (a) changes to make sure biosbits builds with the latest gcc compiler
> +    (gcc 9.4).
> +    (b) upgrade of acpica to the latest version (march 2022).
> +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> +    debugcon can be used to collect the logs.
> +    (d) send a gub halt command to shutdown the VM once all the tests have been
> +    executed.
> +
> +    This zip file is automatically generated by the bits build system. It can
> +    be found in the bits build directory and it is suffixed by the bits build
> +    number.
> +    Normally, there is no need to make any alteration to this zip archive
> +    unless one wanted to change the bits software itself (for example to add
> +    a new config option or change actions of existing options or change the
> +    debug IO port etc). When such changes are made and a new biosbits software
> +    is needed to be generated, please refresh this zip archive as well as the
> +    grub tarball at the same time. Tests will break if changes are made to bits
> +    that are incompatible with existing behavior. So care must be taken to make
> +    sure that the change is compatible with existing bits software as much as
> +    possible. When a new config option is introduced for example, bits must
> +    be upadated here first before introducing the new config option through
> +    a later commit.

I don't think playing with source tarballs is a reasonable work environment.
Let's use submodules just like e.g. firmware does?



> + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> +    scripts and modules to the prebuilt directory. These prebuilt grub
> +    artifacts are required in order to generate a bootable bits iso file that
> +    can run tests.
> +    In order to generate this tar archive, please put the following two
> +    directories that can be found in the bits build directory in a single
> +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> +    bits that generated the archive:
> +
> +    grub-inst-x86_64-efi
> +    grub-inst
> +
> +    This version should be the same as the version number of bits that generated
> +    bits-<n>.zip file. In other words, the two files must be in sync and should
> +    come from the same build of biosbits of the same version. Things may still
> +    work if they come from different versions but mixing version numbers are
> +    not recommended.
> +    There is normally no need to regenerate this archive unless some fixes or
> +    changes have gone into the grub that are part of biosbits.
> +
> +acpi-bits/bits-tests:
> +   This directory contains biosbits python based tests that are run from within
> +   the biosbits environment in the spawned VM. New additions of test cases can
> +   be made in the appropriate test file. For example, new acpi tests can go
> +   into testacpi.py and one would call testsuite.add_test() to register the new
> +   test so that it gets executed as a part of the ACPI tests.
> +   It might be occassionally necessary to disable some subtests or add a new
> +   test that belongs to a test suite not already present in this directory. To
> +   do this, please extract the bits source from the zip file mentioned above.
> +   Copy the test suite/script that needs modification (addition of new tests
> +   or disabling them) from boot/python location of the extracted directory
> +   into this directory.
> +
> +   step (a): copy unmodified test script to this directory.
> +   step (b): update meson.build and add this file to the list.
> +   Commit (a) and (b) together in the same commit.
> +
> +   step (c): perform modifications to the test.
> +   Commit (c) in a separate commit.
> +
> +   The test framework will then use your modified test script to run the test.
> +   No further changes would be needed. Please check the logs to make sure that
> +   appropriate changes have taken effect.
> +
> +meson.build files makes sure that the bits qtests are appropriately integrated
> +into the QEMU qtest framework and are run with "make check-qtest".
> +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> +has been built only for this architecture.
> +
> +
> +Author: Ani Sinha <ani@anisinha.ca>
> +
> +References:
> +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> -- 
> 2.25.1

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > The README file is added describing the directory structure and the purpose
> > of every file it contains. It also describes how to add new tests, make changes
> > to existing tests or bits config files or regenerate the bits software.
> >
> > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > ---
> >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> >  1 file changed, 168 insertions(+)
> >  create mode 100644 tests/qtest/acpi-bits/README
> >
> > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > new file mode 100644
> > index 0000000000..97b15f1665
> > --- /dev/null
> > +++ b/tests/qtest/acpi-bits/README
> > @@ -0,0 +1,168 @@
> > +=============================================================================
> > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > +=============================================================================
> > +
> > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > +visiting https://biosbits.org/. The github codebase can be found here:
> > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > +the bios components such as acpi and smbios tables directly through acpica
> > +bios interpreter (a freely available C based library written by Intel,
> > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > +operating system getting involved in between.
> > +There are several advantages to directly testing the bios in a real physical
> > +machine or VM as opposed to indirectly discovering bios issues through the
> > +operating system. For one thing, the OSes tend to hide bios problems from the
> > +end user. The other is that we have more control of what we wanted to test
> > +and how by directly using acpica interpreter on top of the bios on a running
> > +system. More details on the inspiration for developing biosbits and its real
> > +life uses can be found in (a) and (b).
> > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > +bios components using biosbits and reports test failures.
> > +
> > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > +would require python3.8-venv and python3-pip packages to be installed.
>
> Why do we mess with venv and pip? Certainly possible but
> what's wrong with using distro provided packages?

There are two things:
(a) We are already using pip and venv for our avocado based
integration tests. Look for TESTS_VENV_DIR in Makefile.include under
tests.
(b) the venv is primarily needed because I wanted to take advantage of
our rich python library that handles QEMU based machines. There are
python qtest libraries as well. These are well tested and used with
integration tests and I wanted to keep the test part of the code
simple by simply reusing them. however, in order to use them, we need
a venv environment within which these qemu python libraries are
installed. Integration tests does the same thing.

A note about my language of choice - python. I gave a lot of thoughts
on this. We do not do a lot of stuff here. All we do is:
(a) generate bits iso.
(b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
(c) collect and analyze logs.

We are not inspecting guest memory or manipulating devices or pci
buses. We do not need the power of C here. We need something that is
simple to write, easy to maintain and understand and can deal with
things like manipulating text files and configs easily. Python seems a
better fit for the role.

>
> > +
> > +A brief description of the contents of this directory follows:
> > +
> > +├── acpi-bits-test.py
> > +├── acpi-bits-test-venv.sh
> > +├── bits-config
> > +│   ├── bits-cfg.txt
> > +│   └── meson.build
> > +├── bits-tests
> > +│   ├── meson.build
> > +│   ├── smbios.py
> > +│   ├── smilatency.py
> > +│   ├── testacpi.py
> > +│   └── testcpuid.py
> > +├── meson.build
> > +├── prebuilt
> > +│   ├── bits-2095-grub.tar.gz
> > +│   ├── bits-2095.zip
> > +│   └── meson.build
> > +├── README
> > +└── requirements.txt
> > +
> > +acpi-bits:
> > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > +   environment necessary for the actual python test script to run. Amongst
> > +   other things, it makes sure that QEMU python library is available within
> > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > +   found within python/ directory in QEMU source.
> > +   After setting up the virtual environment, it runs the python test script
> > +   from within that environment.
> > +   If you want to enable verbose mode only for bits test and run make check,
> > +   one trick is to add V=1 before the call to execute the python script in
> > +   this file.
> > + - acpi-bits-test.py: This is the main python test script that generates a
> > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > +   test failures. This is the script one would be interested in if they wanted
> > +   to add or change some component of the log parsing, add a new commandline to
> > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > +   this script unless they wanted to enhance or change the log parsing for
> > +   their tests.
> > + - requirements.txt: This text file contains the dependent python libraries
> > +   needed for the tests to run. If a new dependent library is needed, it would
> > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > +   it when setting up the python virtual environment using pip.
> > + - README: This text file.
> > +
> > +acpi-bits/bits-config:
> > +   This location contains biosbits config files that determine how the software
> > +   runs the tests.
> > + - bits-config.txt: this is the biosbits config file that determines what tests
> > +   or actions are performed by bits. The description of the config options are
> > +   provided in the file itself.
> > +
> > +acpi-bits/prebuilt:
> > +   This location contains prebuilt biosbits binaries that are used to generate
> > +   the bits iso file for testing.
> > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > +    the iso file that are used for testing. This binary zip archive also
> > +    contains the full source of the bits software including the full text of
> > +    the license agreement listed here:
> > +    https://github.com/biosbits/bits/blob/master/COPYING
> > +    The source tarball can be found in this location in the zip file:
> > +    boot/src/bits-2095.tar.gz
> > +    The additional changes beyond those that are present in the official
> > +    biosbits github repository can be found here:
> > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > +
> > +    Basically these changes can be categorized into the following:
> > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > +    (gcc 9.4).
> > +    (b) upgrade of acpica to the latest version (march 2022).
> > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > +    debugcon can be used to collect the logs.
> > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > +    executed.
> > +
> > +    This zip file is automatically generated by the bits build system. It can
> > +    be found in the bits build directory and it is suffixed by the bits build
> > +    number.
> > +    Normally, there is no need to make any alteration to this zip archive
> > +    unless one wanted to change the bits software itself (for example to add
> > +    a new config option or change actions of existing options or change the
> > +    debug IO port etc). When such changes are made and a new biosbits software
> > +    is needed to be generated, please refresh this zip archive as well as the
> > +    grub tarball at the same time. Tests will break if changes are made to bits
> > +    that are incompatible with existing behavior. So care must be taken to make
> > +    sure that the change is compatible with existing bits software as much as
> > +    possible. When a new config option is introduced for example, bits must
> > +    be upadated here first before introducing the new config option through
> > +    a later commit.
>
> I don't think playing with source tarballs is a reasonable work environment.

I agree. However, we do not do much with the source tarball. It is
there as part of bits to satisfy the license requirement. If we need
to manipulate any test script that is in the source file, we would
copy it over and keep it in the bits-test directory and change it
there.

> Let's use submodules just like e.g. firmware does?

Yes I also proposed this to Igor on IRC. We can/maybe need to figure
out a place to store the bits source if we think my github is not the
best place. However, we need the source only if/when we need to
rebuild bits. I suspect it won't be too often if at all.

>
>
>
> > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > +    artifacts are required in order to generate a bootable bits iso file that
> > +    can run tests.
> > +    In order to generate this tar archive, please put the following two
> > +    directories that can be found in the bits build directory in a single
> > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > +    bits that generated the archive:
> > +
> > +    grub-inst-x86_64-efi
> > +    grub-inst
> > +
> > +    This version should be the same as the version number of bits that generated
> > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > +    come from the same build of biosbits of the same version. Things may still
> > +    work if they come from different versions but mixing version numbers are
> > +    not recommended.
> > +    There is normally no need to regenerate this archive unless some fixes or
> > +    changes have gone into the grub that are part of biosbits.
> > +
> > +acpi-bits/bits-tests:
> > +   This directory contains biosbits python based tests that are run from within
> > +   the biosbits environment in the spawned VM. New additions of test cases can
> > +   be made in the appropriate test file. For example, new acpi tests can go
> > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > +   test so that it gets executed as a part of the ACPI tests.
> > +   It might be occassionally necessary to disable some subtests or add a new
> > +   test that belongs to a test suite not already present in this directory. To
> > +   do this, please extract the bits source from the zip file mentioned above.
> > +   Copy the test suite/script that needs modification (addition of new tests
> > +   or disabling them) from boot/python location of the extracted directory
> > +   into this directory.
> > +
> > +   step (a): copy unmodified test script to this directory.
> > +   step (b): update meson.build and add this file to the list.
> > +   Commit (a) and (b) together in the same commit.
> > +
> > +   step (c): perform modifications to the test.
> > +   Commit (c) in a separate commit.
> > +
> > +   The test framework will then use your modified test script to run the test.
> > +   No further changes would be needed. Please check the logs to make sure that
> > +   appropriate changes have taken effect.
> > +
> > +meson.build files makes sure that the bits qtests are appropriately integrated
> > +into the QEMU qtest framework and are run with "make check-qtest".
> > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > +has been built only for this architecture.
> > +
> > +
> > +Author: Ani Sinha <ani@anisinha.ca>
> > +
> > +References:
> > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > --
> > 2.25.1
>

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > The README file is added describing the directory structure and the purpose
> > > of every file it contains. It also describes how to add new tests, make changes
> > > to existing tests or bits config files or regenerate the bits software.
> > >
> > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > ---
> > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > >  1 file changed, 168 insertions(+)
> > >  create mode 100644 tests/qtest/acpi-bits/README
> > >
> > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > new file mode 100644
> > > index 0000000000..97b15f1665
> > > --- /dev/null
> > > +++ b/tests/qtest/acpi-bits/README
> > > @@ -0,0 +1,168 @@
> > > +=============================================================================
> > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > +=============================================================================
> > > +
> > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > +the bios components such as acpi and smbios tables directly through acpica
> > > +bios interpreter (a freely available C based library written by Intel,
> > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > +operating system getting involved in between.
> > > +There are several advantages to directly testing the bios in a real physical
> > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > +end user. The other is that we have more control of what we wanted to test
> > > +and how by directly using acpica interpreter on top of the bios on a running
> > > +system. More details on the inspiration for developing biosbits and its real
> > > +life uses can be found in (a) and (b).
> > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > +bios components using biosbits and reports test failures.
> > > +
> > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > +would require python3.8-venv and python3-pip packages to be installed.
> >
> > Why do we mess with venv and pip? Certainly possible but
> > what's wrong with using distro provided packages?
> 
> There are two things:
> (a) We are already using pip and venv for our avocado based
> integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> tests.
> (b) the venv is primarily needed because I wanted to take advantage of
> our rich python library that handles QEMU based machines. There are
> python qtest libraries as well. These are well tested and used with
> integration tests and I wanted to keep the test part of the code
> simple by simply reusing them. however, in order to use them, we need
> a venv environment within which these qemu python libraries are
> installed. Integration tests does the same thing.
> 
> A note about my language of choice - python. I gave a lot of thoughts
> on this. We do not do a lot of stuff here. All we do is:
> (a) generate bits iso.
> (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> (c) collect and analyze logs.
> 
> We are not inspecting guest memory or manipulating devices or pci
> buses. We do not need the power of C here. We need something that is
> simple to write, easy to maintain and understand and can deal with
> things like manipulating text files and configs easily. Python seems a
> better fit for the role.

No problem with that. So that's venv. But do we need pip and pulling
packages from the net during testing?

> >
> > > +
> > > +A brief description of the contents of this directory follows:
> > > +
> > > +├── acpi-bits-test.py
> > > +├── acpi-bits-test-venv.sh
> > > +├── bits-config
> > > +│   ├── bits-cfg.txt
> > > +│   └── meson.build
> > > +├── bits-tests
> > > +│   ├── meson.build
> > > +│   ├── smbios.py
> > > +│   ├── smilatency.py
> > > +│   ├── testacpi.py
> > > +│   └── testcpuid.py
> > > +├── meson.build
> > > +├── prebuilt
> > > +│   ├── bits-2095-grub.tar.gz
> > > +│   ├── bits-2095.zip
> > > +│   └── meson.build
> > > +├── README
> > > +└── requirements.txt
> > > +
> > > +acpi-bits:
> > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > +   environment necessary for the actual python test script to run. Amongst
> > > +   other things, it makes sure that QEMU python library is available within
> > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > +   found within python/ directory in QEMU source.
> > > +   After setting up the virtual environment, it runs the python test script
> > > +   from within that environment.
> > > +   If you want to enable verbose mode only for bits test and run make check,
> > > +   one trick is to add V=1 before the call to execute the python script in
> > > +   this file.
> > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > +   test failures. This is the script one would be interested in if they wanted
> > > +   to add or change some component of the log parsing, add a new commandline to
> > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > +   this script unless they wanted to enhance or change the log parsing for
> > > +   their tests.
> > > + - requirements.txt: This text file contains the dependent python libraries
> > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > +   it when setting up the python virtual environment using pip.
> > > + - README: This text file.
> > > +
> > > +acpi-bits/bits-config:
> > > +   This location contains biosbits config files that determine how the software
> > > +   runs the tests.
> > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > +   or actions are performed by bits. The description of the config options are
> > > +   provided in the file itself.
> > > +
> > > +acpi-bits/prebuilt:
> > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > +   the bits iso file for testing.
> > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > +    the iso file that are used for testing. This binary zip archive also
> > > +    contains the full source of the bits software including the full text of
> > > +    the license agreement listed here:
> > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > +    The source tarball can be found in this location in the zip file:
> > > +    boot/src/bits-2095.tar.gz
> > > +    The additional changes beyond those that are present in the official
> > > +    biosbits github repository can be found here:
> > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > +
> > > +    Basically these changes can be categorized into the following:
> > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > +    (gcc 9.4).
> > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > +    debugcon can be used to collect the logs.
> > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > +    executed.
> > > +
> > > +    This zip file is automatically generated by the bits build system. It can
> > > +    be found in the bits build directory and it is suffixed by the bits build
> > > +    number.
> > > +    Normally, there is no need to make any alteration to this zip archive
> > > +    unless one wanted to change the bits software itself (for example to add
> > > +    a new config option or change actions of existing options or change the
> > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > +    is needed to be generated, please refresh this zip archive as well as the
> > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > +    that are incompatible with existing behavior. So care must be taken to make
> > > +    sure that the change is compatible with existing bits software as much as
> > > +    possible. When a new config option is introduced for example, bits must
> > > +    be upadated here first before introducing the new config option through
> > > +    a later commit.
> >
> > I don't think playing with source tarballs is a reasonable work environment.
> 
> I agree. However, we do not do much with the source tarball. It is
> there as part of bits to satisfy the license requirement. If we need
> to manipulate any test script that is in the source file, we would
> copy it over and keep it in the bits-test directory and change it
> there.
> 
> > Let's use submodules just like e.g. firmware does?
> 
> Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> out a place to store the bits source if we think my github is not the
> best place. However, we need the source only if/when we need to
> rebuild bits. I suspect it won't be too often if at all.

Why not on git.qemu.org ?

> >
> >
> >
> > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > +    artifacts are required in order to generate a bootable bits iso file that
> > > +    can run tests.
> > > +    In order to generate this tar archive, please put the following two
> > > +    directories that can be found in the bits build directory in a single
> > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > +    bits that generated the archive:
> > > +
> > > +    grub-inst-x86_64-efi
> > > +    grub-inst
> > > +
> > > +    This version should be the same as the version number of bits that generated
> > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > +    come from the same build of biosbits of the same version. Things may still
> > > +    work if they come from different versions but mixing version numbers are
> > > +    not recommended.
> > > +    There is normally no need to regenerate this archive unless some fixes or
> > > +    changes have gone into the grub that are part of biosbits.
> > > +
> > > +acpi-bits/bits-tests:
> > > +   This directory contains biosbits python based tests that are run from within
> > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > +   test so that it gets executed as a part of the ACPI tests.
> > > +   It might be occassionally necessary to disable some subtests or add a new
> > > +   test that belongs to a test suite not already present in this directory. To
> > > +   do this, please extract the bits source from the zip file mentioned above.
> > > +   Copy the test suite/script that needs modification (addition of new tests
> > > +   or disabling them) from boot/python location of the extracted directory
> > > +   into this directory.
> > > +
> > > +   step (a): copy unmodified test script to this directory.
> > > +   step (b): update meson.build and add this file to the list.
> > > +   Commit (a) and (b) together in the same commit.
> > > +
> > > +   step (c): perform modifications to the test.
> > > +   Commit (c) in a separate commit.
> > > +
> > > +   The test framework will then use your modified test script to run the test.
> > > +   No further changes would be needed. Please check the logs to make sure that
> > > +   appropriate changes have taken effect.
> > > +
> > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > +has been built only for this architecture.
> > > +
> > > +
> > > +Author: Ani Sinha <ani@anisinha.ca>
> > > +
> > > +References:
> > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > --
> > > 2.25.1
> >

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > The README file is added describing the directory structure and the purpose
> > > > of every file it contains. It also describes how to add new tests, make changes
> > > > to existing tests or bits config files or regenerate the bits software.
> > > >
> > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > ---
> > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 168 insertions(+)
> > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > >
> > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > new file mode 100644
> > > > index 0000000000..97b15f1665
> > > > --- /dev/null
> > > > +++ b/tests/qtest/acpi-bits/README
> > > > @@ -0,0 +1,168 @@
> > > > +=============================================================================
> > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > +=============================================================================
> > > > +
> > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > +bios interpreter (a freely available C based library written by Intel,
> > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > +operating system getting involved in between.
> > > > +There are several advantages to directly testing the bios in a real physical
> > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > +end user. The other is that we have more control of what we wanted to test
> > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > +system. More details on the inspiration for developing biosbits and its real
> > > > +life uses can be found in (a) and (b).
> > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > +bios components using biosbits and reports test failures.
> > > > +
> > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > +would require python3.8-venv and python3-pip packages to be installed.
> > >
> > > Why do we mess with venv and pip? Certainly possible but
> > > what's wrong with using distro provided packages?
> >
> > There are two things:
> > (a) We are already using pip and venv for our avocado based
> > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > tests.
> > (b) the venv is primarily needed because I wanted to take advantage of
> > our rich python library that handles QEMU based machines. There are
> > python qtest libraries as well. These are well tested and used with
> > integration tests and I wanted to keep the test part of the code
> > simple by simply reusing them. however, in order to use them, we need
> > a venv environment within which these qemu python libraries are
> > installed. Integration tests does the same thing.
> >
> > A note about my language of choice - python. I gave a lot of thoughts
> > on this. We do not do a lot of stuff here. All we do is:
> > (a) generate bits iso.
> > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > (c) collect and analyze logs.
> >
> > We are not inspecting guest memory or manipulating devices or pci
> > buses. We do not need the power of C here. We need something that is
> > simple to write, easy to maintain and understand and can deal with
> > things like manipulating text files and configs easily. Python seems a
> > better fit for the role.
>
> No problem with that. So that's venv. But do we need pip and pulling
> packages from the net during testing?

We do that too. See requirements.txt in tests/
Following two are downloaded:
avocado-framework==88.1
pycdlib==1.11.0

Also see this line in Makefie.include:

$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))

>
> > >
> > > > +
> > > > +A brief description of the contents of this directory follows:
> > > > +
> > > > +├── acpi-bits-test.py
> > > > +├── acpi-bits-test-venv.sh
> > > > +├── bits-config
> > > > +│   ├── bits-cfg.txt
> > > > +│   └── meson.build
> > > > +├── bits-tests
> > > > +│   ├── meson.build
> > > > +│   ├── smbios.py
> > > > +│   ├── smilatency.py
> > > > +│   ├── testacpi.py
> > > > +│   └── testcpuid.py
> > > > +├── meson.build
> > > > +├── prebuilt
> > > > +│   ├── bits-2095-grub.tar.gz
> > > > +│   ├── bits-2095.zip
> > > > +│   └── meson.build
> > > > +├── README
> > > > +└── requirements.txt
> > > > +
> > > > +acpi-bits:
> > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > +   environment necessary for the actual python test script to run. Amongst
> > > > +   other things, it makes sure that QEMU python library is available within
> > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > +   found within python/ directory in QEMU source.
> > > > +   After setting up the virtual environment, it runs the python test script
> > > > +   from within that environment.
> > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > +   this file.
> > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > +   test failures. This is the script one would be interested in if they wanted
> > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > +   their tests.
> > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > +   it when setting up the python virtual environment using pip.
> > > > + - README: This text file.
> > > > +
> > > > +acpi-bits/bits-config:
> > > > +   This location contains biosbits config files that determine how the software
> > > > +   runs the tests.
> > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > +   or actions are performed by bits. The description of the config options are
> > > > +   provided in the file itself.
> > > > +
> > > > +acpi-bits/prebuilt:
> > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > +   the bits iso file for testing.
> > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > +    the iso file that are used for testing. This binary zip archive also
> > > > +    contains the full source of the bits software including the full text of
> > > > +    the license agreement listed here:
> > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > +    The source tarball can be found in this location in the zip file:
> > > > +    boot/src/bits-2095.tar.gz
> > > > +    The additional changes beyond those that are present in the official
> > > > +    biosbits github repository can be found here:
> > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > +
> > > > +    Basically these changes can be categorized into the following:
> > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > +    (gcc 9.4).
> > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > +    debugcon can be used to collect the logs.
> > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > +    executed.
> > > > +
> > > > +    This zip file is automatically generated by the bits build system. It can
> > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > +    number.
> > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > +    unless one wanted to change the bits software itself (for example to add
> > > > +    a new config option or change actions of existing options or change the
> > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > +    sure that the change is compatible with existing bits software as much as
> > > > +    possible. When a new config option is introduced for example, bits must
> > > > +    be upadated here first before introducing the new config option through
> > > > +    a later commit.
> > >
> > > I don't think playing with source tarballs is a reasonable work environment.
> >
> > I agree. However, we do not do much with the source tarball. It is
> > there as part of bits to satisfy the license requirement. If we need
> > to manipulate any test script that is in the source file, we would
> > copy it over and keep it in the bits-test directory and change it
> > there.
> >
> > > Let's use submodules just like e.g. firmware does?
> >
> > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > out a place to store the bits source if we think my github is not the
> > best place. However, we need the source only if/when we need to
> > rebuild bits. I suspect it won't be too often if at all.
>
> Why not on git.qemu.org ?

Sounds good to me. Do I get push access? :-)

>
> > >
> > >
> > >
> > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > +    can run tests.
> > > > +    In order to generate this tar archive, please put the following two
> > > > +    directories that can be found in the bits build directory in a single
> > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > +    bits that generated the archive:
> > > > +
> > > > +    grub-inst-x86_64-efi
> > > > +    grub-inst
> > > > +
> > > > +    This version should be the same as the version number of bits that generated
> > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > +    come from the same build of biosbits of the same version. Things may still
> > > > +    work if they come from different versions but mixing version numbers are
> > > > +    not recommended.
> > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > +    changes have gone into the grub that are part of biosbits.
> > > > +
> > > > +acpi-bits/bits-tests:
> > > > +   This directory contains biosbits python based tests that are run from within
> > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > +   test that belongs to a test suite not already present in this directory. To
> > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > +   or disabling them) from boot/python location of the extracted directory
> > > > +   into this directory.
> > > > +
> > > > +   step (a): copy unmodified test script to this directory.
> > > > +   step (b): update meson.build and add this file to the list.
> > > > +   Commit (a) and (b) together in the same commit.
> > > > +
> > > > +   step (c): perform modifications to the test.
> > > > +   Commit (c) in a separate commit.
> > > > +
> > > > +   The test framework will then use your modified test script to run the test.
> > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > +   appropriate changes have taken effect.
> > > > +
> > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > +has been built only for this architecture.
> > > > +
> > > > +
> > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > +
> > > > +References:
> > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > --
> > > > 2.25.1
> > >
>

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > The README file is added describing the directory structure and the purpose
> > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > to existing tests or bits config files or regenerate the bits software.
> > > > >
> > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > ---
> > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > >  1 file changed, 168 insertions(+)
> > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > >
> > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > new file mode 100644
> > > > > index 0000000000..97b15f1665
> > > > > --- /dev/null
> > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > @@ -0,0 +1,168 @@
> > > > > +=============================================================================
> > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > +=============================================================================
> > > > > +
> > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > +operating system getting involved in between.
> > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > +life uses can be found in (a) and (b).
> > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > +bios components using biosbits and reports test failures.
> > > > > +
> > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > >
> > > > Why do we mess with venv and pip? Certainly possible but
> > > > what's wrong with using distro provided packages?
> > >
> > > There are two things:
> > > (a) We are already using pip and venv for our avocado based
> > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > tests.
> > > (b) the venv is primarily needed because I wanted to take advantage of
> > > our rich python library that handles QEMU based machines. There are
> > > python qtest libraries as well. These are well tested and used with
> > > integration tests and I wanted to keep the test part of the code
> > > simple by simply reusing them. however, in order to use them, we need
> > > a venv environment within which these qemu python libraries are
> > > installed. Integration tests does the same thing.
> > >
> > > A note about my language of choice - python. I gave a lot of thoughts
> > > on this. We do not do a lot of stuff here. All we do is:
> > > (a) generate bits iso.
> > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > (c) collect and analyze logs.
> > >
> > > We are not inspecting guest memory or manipulating devices or pci
> > > buses. We do not need the power of C here. We need something that is
> > > simple to write, easy to maintain and understand and can deal with
> > > things like manipulating text files and configs easily. Python seems a
> > > better fit for the role.
> >
> > No problem with that. So that's venv. But do we need pip and pulling
> > packages from the net during testing?
> 
> We do that too. See requirements.txt in tests/
> Following two are downloaded:
> avocado-framework==88.1
> pycdlib==1.11.0
> 
> Also see this line in Makefie.include:
> 
> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))

Right but that's avocado since it pulls lots of stuff from
the net anyway.
Are the libraries in question not packaged on major distros?

> >
> > > >
> > > > > +
> > > > > +A brief description of the contents of this directory follows:
> > > > > +
> > > > > +├── acpi-bits-test.py
> > > > > +├── acpi-bits-test-venv.sh
> > > > > +├── bits-config
> > > > > +│   ├── bits-cfg.txt
> > > > > +│   └── meson.build
> > > > > +├── bits-tests
> > > > > +│   ├── meson.build
> > > > > +│   ├── smbios.py
> > > > > +│   ├── smilatency.py
> > > > > +│   ├── testacpi.py
> > > > > +│   └── testcpuid.py
> > > > > +├── meson.build
> > > > > +├── prebuilt
> > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > +│   ├── bits-2095.zip
> > > > > +│   └── meson.build
> > > > > +├── README
> > > > > +└── requirements.txt
> > > > > +
> > > > > +acpi-bits:
> > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > +   found within python/ directory in QEMU source.
> > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > +   from within that environment.
> > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > +   this file.
> > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > +   their tests.
> > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > +   it when setting up the python virtual environment using pip.
> > > > > + - README: This text file.
> > > > > +
> > > > > +acpi-bits/bits-config:
> > > > > +   This location contains biosbits config files that determine how the software
> > > > > +   runs the tests.
> > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > +   or actions are performed by bits. The description of the config options are
> > > > > +   provided in the file itself.
> > > > > +
> > > > > +acpi-bits/prebuilt:
> > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > +   the bits iso file for testing.
> > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > +    contains the full source of the bits software including the full text of
> > > > > +    the license agreement listed here:
> > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > +    The source tarball can be found in this location in the zip file:
> > > > > +    boot/src/bits-2095.tar.gz
> > > > > +    The additional changes beyond those that are present in the official
> > > > > +    biosbits github repository can be found here:
> > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > +
> > > > > +    Basically these changes can be categorized into the following:
> > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > +    (gcc 9.4).
> > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > +    debugcon can be used to collect the logs.
> > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > +    executed.
> > > > > +
> > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > +    number.
> > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > +    a new config option or change actions of existing options or change the
> > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > +    be upadated here first before introducing the new config option through
> > > > > +    a later commit.
> > > >
> > > > I don't think playing with source tarballs is a reasonable work environment.
> > >
> > > I agree. However, we do not do much with the source tarball. It is
> > > there as part of bits to satisfy the license requirement. If we need
> > > to manipulate any test script that is in the source file, we would
> > > copy it over and keep it in the bits-test directory and change it
> > > there.
> > >
> > > > Let's use submodules just like e.g. firmware does?
> > >
> > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > out a place to store the bits source if we think my github is not the
> > > best place. However, we need the source only if/when we need to
> > > rebuild bits. I suspect it won't be too often if at all.
> >
> > Why not on git.qemu.org ?
> 
> Sounds good to me. Do I get push access? :-)
> 
> >
> > > >
> > > >
> > > >
> > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > +    can run tests.
> > > > > +    In order to generate this tar archive, please put the following two
> > > > > +    directories that can be found in the bits build directory in a single
> > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > +    bits that generated the archive:
> > > > > +
> > > > > +    grub-inst-x86_64-efi
> > > > > +    grub-inst
> > > > > +
> > > > > +    This version should be the same as the version number of bits that generated
> > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > +    work if they come from different versions but mixing version numbers are
> > > > > +    not recommended.
> > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > +
> > > > > +acpi-bits/bits-tests:
> > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > +   into this directory.
> > > > > +
> > > > > +   step (a): copy unmodified test script to this directory.
> > > > > +   step (b): update meson.build and add this file to the list.
> > > > > +   Commit (a) and (b) together in the same commit.
> > > > > +
> > > > > +   step (c): perform modifications to the test.
> > > > > +   Commit (c) in a separate commit.
> > > > > +
> > > > > +   The test framework will then use your modified test script to run the test.
> > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > +   appropriate changes have taken effect.
> > > > > +
> > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > +has been built only for this architecture.
> > > > > +
> > > > > +
> > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > +
> > > > > +References:
> > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > --
> > > > > 2.25.1
> > > >
> >

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > The README file is added describing the directory structure and the purpose
> > > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > > to existing tests or bits config files or regenerate the bits software.
> > > > > >
> > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > > ---
> > > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > > >  1 file changed, 168 insertions(+)
> > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > >
> > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > > new file mode 100644
> > > > > > index 0000000000..97b15f1665
> > > > > > --- /dev/null
> > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > @@ -0,0 +1,168 @@
> > > > > > +=============================================================================
> > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > +=============================================================================
> > > > > > +
> > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > > +operating system getting involved in between.
> > > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > > +life uses can be found in (a) and (b).
> > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > > +bios components using biosbits and reports test failures.
> > > > > > +
> > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > > >
> > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > what's wrong with using distro provided packages?
> > > >
> > > > There are two things:
> > > > (a) We are already using pip and venv for our avocado based
> > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > tests.
> > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > our rich python library that handles QEMU based machines. There are
> > > > python qtest libraries as well. These are well tested and used with
> > > > integration tests and I wanted to keep the test part of the code
> > > > simple by simply reusing them. however, in order to use them, we need
> > > > a venv environment within which these qemu python libraries are
> > > > installed. Integration tests does the same thing.
> > > >
> > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > on this. We do not do a lot of stuff here. All we do is:
> > > > (a) generate bits iso.
> > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > > (c) collect and analyze logs.
> > > >
> > > > We are not inspecting guest memory or manipulating devices or pci
> > > > buses. We do not need the power of C here. We need something that is
> > > > simple to write, easy to maintain and understand and can deal with
> > > > things like manipulating text files and configs easily. Python seems a
> > > > better fit for the role.
> > >
> > > No problem with that. So that's venv. But do we need pip and pulling
> > > packages from the net during testing?
> >
> > We do that too. See requirements.txt in tests/
> > Following two are downloaded:
> > avocado-framework==88.1
> > pycdlib==1.11.0
> >
> > Also see this line in Makefie.include:
> >
> > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>
> Right but that's avocado since it pulls lots of stuff from
> the net anyway.
> Are the libraries in question not packaged on major distros?

Currently I only need this:
https://github.com/python-tap/tappy
which is the basic TAP processing library for python.

It seems its only installed through pip:
https://tappy.readthedocs.io/en/latest/

I do not think this is packaged by default. It's such a basic library
for parsing test output that maybe we can keep this somewhere within
the python src tree? Not sure ...

>
> > >
> > > > >
> > > > > > +
> > > > > > +A brief description of the contents of this directory follows:
> > > > > > +
> > > > > > +├── acpi-bits-test.py
> > > > > > +├── acpi-bits-test-venv.sh
> > > > > > +├── bits-config
> > > > > > +│   ├── bits-cfg.txt
> > > > > > +│   └── meson.build
> > > > > > +├── bits-tests
> > > > > > +│   ├── meson.build
> > > > > > +│   ├── smbios.py
> > > > > > +│   ├── smilatency.py
> > > > > > +│   ├── testacpi.py
> > > > > > +│   └── testcpuid.py
> > > > > > +├── meson.build
> > > > > > +├── prebuilt
> > > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > > +│   ├── bits-2095.zip
> > > > > > +│   └── meson.build
> > > > > > +├── README
> > > > > > +└── requirements.txt
> > > > > > +
> > > > > > +acpi-bits:
> > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > > +   found within python/ directory in QEMU source.
> > > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > > +   from within that environment.
> > > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > > +   this file.
> > > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > > +   their tests.
> > > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > > +   it when setting up the python virtual environment using pip.
> > > > > > + - README: This text file.
> > > > > > +
> > > > > > +acpi-bits/bits-config:
> > > > > > +   This location contains biosbits config files that determine how the software
> > > > > > +   runs the tests.
> > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > > +   or actions are performed by bits. The description of the config options are
> > > > > > +   provided in the file itself.
> > > > > > +
> > > > > > +acpi-bits/prebuilt:
> > > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > > +   the bits iso file for testing.
> > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > > +    contains the full source of the bits software including the full text of
> > > > > > +    the license agreement listed here:
> > > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > > +    The source tarball can be found in this location in the zip file:
> > > > > > +    boot/src/bits-2095.tar.gz
> > > > > > +    The additional changes beyond those that are present in the official
> > > > > > +    biosbits github repository can be found here:
> > > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > > +
> > > > > > +    Basically these changes can be categorized into the following:
> > > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > > +    (gcc 9.4).
> > > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > > +    debugcon can be used to collect the logs.
> > > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > > +    executed.
> > > > > > +
> > > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > > +    number.
> > > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > > +    a new config option or change actions of existing options or change the
> > > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > > +    be upadated here first before introducing the new config option through
> > > > > > +    a later commit.
> > > > >
> > > > > I don't think playing with source tarballs is a reasonable work environment.
> > > >
> > > > I agree. However, we do not do much with the source tarball. It is
> > > > there as part of bits to satisfy the license requirement. If we need
> > > > to manipulate any test script that is in the source file, we would
> > > > copy it over and keep it in the bits-test directory and change it
> > > > there.
> > > >
> > > > > Let's use submodules just like e.g. firmware does?
> > > >
> > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > > out a place to store the bits source if we think my github is not the
> > > > best place. However, we need the source only if/when we need to
> > > > rebuild bits. I suspect it won't be too often if at all.
> > >
> > > Why not on git.qemu.org ?
> >
> > Sounds good to me. Do I get push access? :-)
> >
> > >
> > > > >
> > > > >
> > > > >
> > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > > +    can run tests.
> > > > > > +    In order to generate this tar archive, please put the following two
> > > > > > +    directories that can be found in the bits build directory in a single
> > > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > > +    bits that generated the archive:
> > > > > > +
> > > > > > +    grub-inst-x86_64-efi
> > > > > > +    grub-inst
> > > > > > +
> > > > > > +    This version should be the same as the version number of bits that generated
> > > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > > +    work if they come from different versions but mixing version numbers are
> > > > > > +    not recommended.
> > > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > > +
> > > > > > +acpi-bits/bits-tests:
> > > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > > +   into this directory.
> > > > > > +
> > > > > > +   step (a): copy unmodified test script to this directory.
> > > > > > +   step (b): update meson.build and add this file to the list.
> > > > > > +   Commit (a) and (b) together in the same commit.
> > > > > > +
> > > > > > +   step (c): perform modifications to the test.
> > > > > > +   Commit (c) in a separate commit.
> > > > > > +
> > > > > > +   The test framework will then use your modified test script to run the test.
> > > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > > +   appropriate changes have taken effect.
> > > > > > +
> > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > > +has been built only for this architecture.
> > > > > > +
> > > > > > +
> > > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > > +
> > > > > > +References:
> > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > > --
> > > > > > 2.25.1
> > > > >
> > >
>

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > > The README file is added describing the directory structure and the purpose
> > > > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > > > to existing tests or bits config files or regenerate the bits software.
> > > > > > >
> > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > > > ---
> > > > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > > > >  1 file changed, 168 insertions(+)
> > > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > > >
> > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > > > new file mode 100644
> > > > > > > index 0000000000..97b15f1665
> > > > > > > --- /dev/null
> > > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > > @@ -0,0 +1,168 @@
> > > > > > > +=============================================================================
> > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > > +=============================================================================
> > > > > > > +
> > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > > > +operating system getting involved in between.
> > > > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > > > +life uses can be found in (a) and (b).
> > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > > > +bios components using biosbits and reports test failures.
> > > > > > > +
> > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > > > >
> > > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > > what's wrong with using distro provided packages?
> > > > >
> > > > > There are two things:
> > > > > (a) We are already using pip and venv for our avocado based
> > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > > tests.
> > > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > > our rich python library that handles QEMU based machines. There are
> > > > > python qtest libraries as well. These are well tested and used with
> > > > > integration tests and I wanted to keep the test part of the code
> > > > > simple by simply reusing them. however, in order to use them, we need
> > > > > a venv environment within which these qemu python libraries are
> > > > > installed. Integration tests does the same thing.
> > > > >
> > > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > > on this. We do not do a lot of stuff here. All we do is:
> > > > > (a) generate bits iso.
> > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > > > (c) collect and analyze logs.
> > > > >
> > > > > We are not inspecting guest memory or manipulating devices or pci
> > > > > buses. We do not need the power of C here. We need something that is
> > > > > simple to write, easy to maintain and understand and can deal with
> > > > > things like manipulating text files and configs easily. Python seems a
> > > > > better fit for the role.
> > > >
> > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > packages from the net during testing?
> > >
> > > We do that too. See requirements.txt in tests/
> > > Following two are downloaded:
> > > avocado-framework==88.1
> > > pycdlib==1.11.0
> > >
> > > Also see this line in Makefie.include:
> > >
> > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> >
> > Right but that's avocado since it pulls lots of stuff from
> > the net anyway.
> > Are the libraries in question not packaged on major distros?
> 
> Currently I only need this:
> https://github.com/python-tap/tappy
> which is the basic TAP processing library for python.
> 
> It seems its only installed through pip:
> https://tappy.readthedocs.io/en/latest/
> 
> I do not think this is packaged by default. It's such a basic library
> for parsing test output that maybe we can keep this somewhere within
> the python src tree? Not sure ...

It's pretty small for sure. Another submodule?

> >
> > > >
> > > > > >
> > > > > > > +
> > > > > > > +A brief description of the contents of this directory follows:
> > > > > > > +
> > > > > > > +├── acpi-bits-test.py
> > > > > > > +├── acpi-bits-test-venv.sh
> > > > > > > +├── bits-config
> > > > > > > +│   ├── bits-cfg.txt
> > > > > > > +│   └── meson.build
> > > > > > > +├── bits-tests
> > > > > > > +│   ├── meson.build
> > > > > > > +│   ├── smbios.py
> > > > > > > +│   ├── smilatency.py
> > > > > > > +│   ├── testacpi.py
> > > > > > > +│   └── testcpuid.py
> > > > > > > +├── meson.build
> > > > > > > +├── prebuilt
> > > > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > > > +│   ├── bits-2095.zip
> > > > > > > +│   └── meson.build
> > > > > > > +├── README
> > > > > > > +└── requirements.txt
> > > > > > > +
> > > > > > > +acpi-bits:
> > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > > > +   found within python/ directory in QEMU source.
> > > > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > > > +   from within that environment.
> > > > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > > > +   this file.
> > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > > > +   their tests.
> > > > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > > > +   it when setting up the python virtual environment using pip.
> > > > > > > + - README: This text file.
> > > > > > > +
> > > > > > > +acpi-bits/bits-config:
> > > > > > > +   This location contains biosbits config files that determine how the software
> > > > > > > +   runs the tests.
> > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > > > +   or actions are performed by bits. The description of the config options are
> > > > > > > +   provided in the file itself.
> > > > > > > +
> > > > > > > +acpi-bits/prebuilt:
> > > > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > > > +   the bits iso file for testing.
> > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > > > +    contains the full source of the bits software including the full text of
> > > > > > > +    the license agreement listed here:
> > > > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > > > +    The source tarball can be found in this location in the zip file:
> > > > > > > +    boot/src/bits-2095.tar.gz
> > > > > > > +    The additional changes beyond those that are present in the official
> > > > > > > +    biosbits github repository can be found here:
> > > > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > > > +
> > > > > > > +    Basically these changes can be categorized into the following:
> > > > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > > > +    (gcc 9.4).
> > > > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > > > +    debugcon can be used to collect the logs.
> > > > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > > > +    executed.
> > > > > > > +
> > > > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > > > +    number.
> > > > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > > > +    a new config option or change actions of existing options or change the
> > > > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > > > +    be upadated here first before introducing the new config option through
> > > > > > > +    a later commit.
> > > > > >
> > > > > > I don't think playing with source tarballs is a reasonable work environment.
> > > > >
> > > > > I agree. However, we do not do much with the source tarball. It is
> > > > > there as part of bits to satisfy the license requirement. If we need
> > > > > to manipulate any test script that is in the source file, we would
> > > > > copy it over and keep it in the bits-test directory and change it
> > > > > there.
> > > > >
> > > > > > Let's use submodules just like e.g. firmware does?
> > > > >
> > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > > > out a place to store the bits source if we think my github is not the
> > > > > best place. However, we need the source only if/when we need to
> > > > > rebuild bits. I suspect it won't be too often if at all.
> > > >
> > > > Why not on git.qemu.org ?
> > >
> > > Sounds good to me. Do I get push access? :-)
> > >
> > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > > > +    can run tests.
> > > > > > > +    In order to generate this tar archive, please put the following two
> > > > > > > +    directories that can be found in the bits build directory in a single
> > > > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > > > +    bits that generated the archive:
> > > > > > > +
> > > > > > > +    grub-inst-x86_64-efi
> > > > > > > +    grub-inst
> > > > > > > +
> > > > > > > +    This version should be the same as the version number of bits that generated
> > > > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > > > +    work if they come from different versions but mixing version numbers are
> > > > > > > +    not recommended.
> > > > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > > > +
> > > > > > > +acpi-bits/bits-tests:
> > > > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > > > +   into this directory.
> > > > > > > +
> > > > > > > +   step (a): copy unmodified test script to this directory.
> > > > > > > +   step (b): update meson.build and add this file to the list.
> > > > > > > +   Commit (a) and (b) together in the same commit.
> > > > > > > +
> > > > > > > +   step (c): perform modifications to the test.
> > > > > > > +   Commit (c) in a separate commit.
> > > > > > > +
> > > > > > > +   The test framework will then use your modified test script to run the test.
> > > > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > > > +   appropriate changes have taken effect.
> > > > > > > +
> > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > > > +has been built only for this architecture.
> > > > > > > +
> > > > > > > +
> > > > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > > > +
> > > > > > > +References:
> > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > > > --
> > > > > > > 2.25.1
> > > > > >
> > > >
> >

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > > > The README file is added describing the directory structure and the purpose
> > > > > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > > > > to existing tests or bits config files or regenerate the bits software.
> > > > > > > >
> > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > > > > ---
> > > > > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > > > > >  1 file changed, 168 insertions(+)
> > > > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > > > >
> > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000000..97b15f1665
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > > > @@ -0,0 +1,168 @@
> > > > > > > > +=============================================================================
> > > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > > > +=============================================================================
> > > > > > > > +
> > > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > > > > +operating system getting involved in between.
> > > > > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > > > > +life uses can be found in (a) and (b).
> > > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > > > > +bios components using biosbits and reports test failures.
> > > > > > > > +
> > > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > > > > >
> > > > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > > > what's wrong with using distro provided packages?
> > > > > >
> > > > > > There are two things:
> > > > > > (a) We are already using pip and venv for our avocado based
> > > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > > > tests.
> > > > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > > > our rich python library that handles QEMU based machines. There are
> > > > > > python qtest libraries as well. These are well tested and used with
> > > > > > integration tests and I wanted to keep the test part of the code
> > > > > > simple by simply reusing them. however, in order to use them, we need
> > > > > > a venv environment within which these qemu python libraries are
> > > > > > installed. Integration tests does the same thing.
> > > > > >
> > > > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > > > on this. We do not do a lot of stuff here. All we do is:
> > > > > > (a) generate bits iso.
> > > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > > > > (c) collect and analyze logs.
> > > > > >
> > > > > > We are not inspecting guest memory or manipulating devices or pci
> > > > > > buses. We do not need the power of C here. We need something that is
> > > > > > simple to write, easy to maintain and understand and can deal with
> > > > > > things like manipulating text files and configs easily. Python seems a
> > > > > > better fit for the role.
> > > > >
> > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > packages from the net during testing?
> > > >
> > > > We do that too. See requirements.txt in tests/
> > > > Following two are downloaded:
> > > > avocado-framework==88.1
> > > > pycdlib==1.11.0
> > > >
> > > > Also see this line in Makefie.include:
> > > >
> > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > >
> > > Right but that's avocado since it pulls lots of stuff from
> > > the net anyway.
> > > Are the libraries in question not packaged on major distros?
> >
> > Currently I only need this:
> > https://github.com/python-tap/tappy
> > which is the basic TAP processing library for python.
> >
> > It seems its only installed through pip:
> > https://tappy.readthedocs.io/en/latest/
> >
> > I do not think this is packaged by default. It's such a basic library
> > for parsing test output that maybe we can keep this somewhere within
> > the python src tree? Not sure ...
>
> It's pretty small for sure. Another submodule?

Unlike BITS, this one is likely going to be maintained for a while and
will receive new releases through
https://pypi.org/project/tap.py/
so forking is OK but someone has to keep this updated.

I am open to anything. Whatever feels right is fine to me.

>
> > >
> > > > >
> > > > > > >
> > > > > > > > +
> > > > > > > > +A brief description of the contents of this directory follows:
> > > > > > > > +
> > > > > > > > +├── acpi-bits-test.py
> > > > > > > > +├── acpi-bits-test-venv.sh
> > > > > > > > +├── bits-config
> > > > > > > > +│   ├── bits-cfg.txt
> > > > > > > > +│   └── meson.build
> > > > > > > > +├── bits-tests
> > > > > > > > +│   ├── meson.build
> > > > > > > > +│   ├── smbios.py
> > > > > > > > +│   ├── smilatency.py
> > > > > > > > +│   ├── testacpi.py
> > > > > > > > +│   └── testcpuid.py
> > > > > > > > +├── meson.build
> > > > > > > > +├── prebuilt
> > > > > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > > > > +│   ├── bits-2095.zip
> > > > > > > > +│   └── meson.build
> > > > > > > > +├── README
> > > > > > > > +└── requirements.txt
> > > > > > > > +
> > > > > > > > +acpi-bits:
> > > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > > > > +   found within python/ directory in QEMU source.
> > > > > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > > > > +   from within that environment.
> > > > > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > > > > +   this file.
> > > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > > > > +   their tests.
> > > > > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > > > > +   it when setting up the python virtual environment using pip.
> > > > > > > > + - README: This text file.
> > > > > > > > +
> > > > > > > > +acpi-bits/bits-config:
> > > > > > > > +   This location contains biosbits config files that determine how the software
> > > > > > > > +   runs the tests.
> > > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > > > > +   or actions are performed by bits. The description of the config options are
> > > > > > > > +   provided in the file itself.
> > > > > > > > +
> > > > > > > > +acpi-bits/prebuilt:
> > > > > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > > > > +   the bits iso file for testing.
> > > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > > > > +    contains the full source of the bits software including the full text of
> > > > > > > > +    the license agreement listed here:
> > > > > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > > > > +    The source tarball can be found in this location in the zip file:
> > > > > > > > +    boot/src/bits-2095.tar.gz
> > > > > > > > +    The additional changes beyond those that are present in the official
> > > > > > > > +    biosbits github repository can be found here:
> > > > > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > > > > +
> > > > > > > > +    Basically these changes can be categorized into the following:
> > > > > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > > > > +    (gcc 9.4).
> > > > > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > > > > +    debugcon can be used to collect the logs.
> > > > > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > > > > +    executed.
> > > > > > > > +
> > > > > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > > > > +    number.
> > > > > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > > > > +    a new config option or change actions of existing options or change the
> > > > > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > > > > +    be upadated here first before introducing the new config option through
> > > > > > > > +    a later commit.
> > > > > > >
> > > > > > > I don't think playing with source tarballs is a reasonable work environment.
> > > > > >
> > > > > > I agree. However, we do not do much with the source tarball. It is
> > > > > > there as part of bits to satisfy the license requirement. If we need
> > > > > > to manipulate any test script that is in the source file, we would
> > > > > > copy it over and keep it in the bits-test directory and change it
> > > > > > there.
> > > > > >
> > > > > > > Let's use submodules just like e.g. firmware does?
> > > > > >
> > > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > > > > out a place to store the bits source if we think my github is not the
> > > > > > best place. However, we need the source only if/when we need to
> > > > > > rebuild bits. I suspect it won't be too often if at all.
> > > > >
> > > > > Why not on git.qemu.org ?
> > > >
> > > > Sounds good to me. Do I get push access? :-)
> > > >
> > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > > > > +    can run tests.
> > > > > > > > +    In order to generate this tar archive, please put the following two
> > > > > > > > +    directories that can be found in the bits build directory in a single
> > > > > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > > > > +    bits that generated the archive:
> > > > > > > > +
> > > > > > > > +    grub-inst-x86_64-efi
> > > > > > > > +    grub-inst
> > > > > > > > +
> > > > > > > > +    This version should be the same as the version number of bits that generated
> > > > > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > > > > +    work if they come from different versions but mixing version numbers are
> > > > > > > > +    not recommended.
> > > > > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > > > > +
> > > > > > > > +acpi-bits/bits-tests:
> > > > > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > > > > +   into this directory.
> > > > > > > > +
> > > > > > > > +   step (a): copy unmodified test script to this directory.
> > > > > > > > +   step (b): update meson.build and add this file to the list.
> > > > > > > > +   Commit (a) and (b) together in the same commit.
> > > > > > > > +
> > > > > > > > +   step (c): perform modifications to the test.
> > > > > > > > +   Commit (c) in a separate commit.
> > > > > > > > +
> > > > > > > > +   The test framework will then use your modified test script to run the test.
> > > > > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > > > > +   appropriate changes have taken effect.
> > > > > > > > +
> > > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > > > > +has been built only for this architecture.
> > > > > > > > +
> > > > > > > > +
> > > > > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > > > > +
> > > > > > > > +References:
> > > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > > > > --
> > > > > > > > 2.25.1
> > > > > > >
> > > > >
> > >
>

venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 08.57, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
>>> On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>
>>>> On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
>>>>> On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>>>
>>>>>> On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
>>>>>>> On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>>>>>
>>>>>>>> On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
>>>>>>>>> The README file is added describing the directory structure and the purpose
>>>>>>>>> of every file it contains. It also describes how to add new tests, make changes
>>>>>>>>> to existing tests or bits config files or regenerate the bits software.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Ani Sinha <ani@anisinha.ca>
>>>>>>>>> ---
>>>>>>>>>   tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
>>>>>>>>>   1 file changed, 168 insertions(+)
>>>>>>>>>   create mode 100644 tests/qtest/acpi-bits/README
>>>>>>>>>
>>>>>>>>> diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
>>>>>>>>> new file mode 100644
>>>>>>>>> index 0000000000..97b15f1665
>>>>>>>>> --- /dev/null
>>>>>>>>> +++ b/tests/qtest/acpi-bits/README
>>>>>>>>> @@ -0,0 +1,168 @@
>>>>>>>>> +=============================================================================
>>>>>>>>> +ACPI/SMBIOS QTESTS USING BIOSBITS
>>>>>>>>> +=============================================================================
>>>>>>>>> +
>>>>>>>>> +Biosbits is a software written by Josh Triplett that can be downloaded by
>>>>>>>>> +visiting https://biosbits.org/. The github codebase can be found here:
>>>>>>>>> +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
>>>>>>>>> +the bios components such as acpi and smbios tables directly through acpica
>>>>>>>>> +bios interpreter (a freely available C based library written by Intel,
>>>>>>>>> +downloadable from https://acpica.org/ and is included with biosbits) without an
>>>>>>>>> +operating system getting involved in between.
>>>>>>>>> +There are several advantages to directly testing the bios in a real physical
>>>>>>>>> +machine or VM as opposed to indirectly discovering bios issues through the
>>>>>>>>> +operating system. For one thing, the OSes tend to hide bios problems from the
>>>>>>>>> +end user. The other is that we have more control of what we wanted to test
>>>>>>>>> +and how by directly using acpica interpreter on top of the bios on a running
>>>>>>>>> +system. More details on the inspiration for developing biosbits and its real
>>>>>>>>> +life uses can be found in (a) and (b).
>>>>>>>>> +This directory contains QEMU qtests written in python that exercizes the QEMU
>>>>>>>>> +bios components using biosbits and reports test failures.
>>>>>>>>> +
>>>>>>>>> +These tests use python virtual environment. In debian/ubuntu system, the tests
>>>>>>>>> +would require python3.8-venv and python3-pip packages to be installed.
>>>>>>>>
>>>>>>>> Why do we mess with venv and pip? Certainly possible but
>>>>>>>> what's wrong with using distro provided packages?
>>>>>>>
>>>>>>> There are two things:
>>>>>>> (a) We are already using pip and venv for our avocado based
>>>>>>> integration tests. Look for TESTS_VENV_DIR in Makefile.include under
>>>>>>> tests.
>>>>>>> (b) the venv is primarily needed because I wanted to take advantage of
>>>>>>> our rich python library that handles QEMU based machines. There are
>>>>>>> python qtest libraries as well. These are well tested and used with
>>>>>>> integration tests and I wanted to keep the test part of the code
>>>>>>> simple by simply reusing them. however, in order to use them, we need
>>>>>>> a venv environment within which these qemu python libraries are
>>>>>>> installed. Integration tests does the same thing.
>>>>>>>
>>>>>>> A note about my language of choice - python. I gave a lot of thoughts
>>>>>>> on this. We do not do a lot of stuff here. All we do is:
>>>>>>> (a) generate bits iso.
>>>>>>> (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
>>>>>>> (c) collect and analyze logs.
>>>>>>>
>>>>>>> We are not inspecting guest memory or manipulating devices or pci
>>>>>>> buses. We do not need the power of C here. We need something that is
>>>>>>> simple to write, easy to maintain and understand and can deal with
>>>>>>> things like manipulating text files and configs easily. Python seems a
>>>>>>> better fit for the role.
>>>>>>
>>>>>> No problem with that. So that's venv. But do we need pip and pulling
>>>>>> packages from the net during testing?
>>>>>
>>>>> We do that too. See requirements.txt in tests/
>>>>> Following two are downloaded:
>>>>> avocado-framework==88.1
>>>>> pycdlib==1.11.0
>>>>>
>>>>> Also see this line in Makefie.include:
>>>>>
>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>>>>
>>>> Right but that's avocado since it pulls lots of stuff from
>>>> the net anyway.
>>>> Are the libraries in question not packaged on major distros?
>>>
>>> Currently I only need this:
>>> https://github.com/python-tap/tappy
>>> which is the basic TAP processing library for python.
>>>
>>> It seems its only installed through pip:
>>> https://tappy.readthedocs.io/en/latest/
>>>
>>> I do not think this is packaged by default. It's such a basic library
>>> for parsing test output that maybe we can keep this somewhere within
>>> the python src tree? Not sure ...
>>
>> It's pretty small for sure. Another submodule?
> 
> Unlike BITS, this one is likely going to be maintained for a while and
> will receive new releases through
> https://pypi.org/project/tap.py/
> so forking is OK but someone has to keep this updated.
> 
> I am open to anything. Whatever feels right is fine to me.

John Snow is currently working on the "Pythonification" of various QEMU 
bits, I think you should loop him into this discussion, too.

  Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > packages from the net during testing?
> > > > > > 
> > > > > > We do that too. See requirements.txt in tests/
> > > > > > Following two are downloaded:
> > > > > > avocado-framework==88.1
> > > > > > pycdlib==1.11.0
> > > > > > 
> > > > > > Also see this line in Makefie.include:
> > > > > > 
> > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > 
> > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > the net anyway.
> > > > > Are the libraries in question not packaged on major distros?
> > > > 
> > > > Currently I only need this:
> > > > https://github.com/python-tap/tappy
> > > > which is the basic TAP processing library for python.
> > > > 
> > > > It seems its only installed through pip:
> > > > https://tappy.readthedocs.io/en/latest/
> > > > 
> > > > I do not think this is packaged by default. It's such a basic library
> > > > for parsing test output that maybe we can keep this somewhere within
> > > > the python src tree? Not sure ...
> > > 
> > > It's pretty small for sure. Another submodule?
> > 
> > Unlike BITS, this one is likely going to be maintained for a while and
> > will receive new releases through
> > https://pypi.org/project/tap.py/
> > so forking is OK but someone has to keep this updated.
> > 
> > I am open to anything. Whatever feels right is fine to me.
> 
> John Snow is currently working on the "Pythonification" of various QEMU
> bits, I think you should loop him into this discussion, too.
> 
>  Thomas

submodule does not mean we fork necessarily. We could have
all options: check for the module and use it if there, if not
use one from system if not there install with pip ..
But yea, I'm not sure what's best either.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
>>>>>>>> packages from the net during testing?
>>>>>>>
>>>>>>> We do that too. See requirements.txt in tests/
>>>>>>> Following two are downloaded:
>>>>>>> avocado-framework==88.1
>>>>>>> pycdlib==1.11.0
>>>>>>>
>>>>>>> Also see this line in Makefie.include:
>>>>>>>
>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>>>>>>
>>>>>> Right but that's avocado since it pulls lots of stuff from
>>>>>> the net anyway.
>>>>>> Are the libraries in question not packaged on major distros?
>>>>>
>>>>> Currently I only need this:
>>>>> https://github.com/python-tap/tappy
>>>>> which is the basic TAP processing library for python.
>>>>>
>>>>> It seems its only installed through pip:
>>>>> https://tappy.readthedocs.io/en/latest/
>>>>>
>>>>> I do not think this is packaged by default. It's such a basic library
>>>>> for parsing test output that maybe we can keep this somewhere within
>>>>> the python src tree? Not sure ...
>>>>
>>>> It's pretty small for sure. Another submodule?
>>>
>>> Unlike BITS, this one is likely going to be maintained for a while and
>>> will receive new releases through
>>> https://pypi.org/project/tap.py/
>>> so forking is OK but someone has to keep this updated.
>>>
>>> I am open to anything. Whatever feels right is fine to me.
>>
>> John Snow is currently working on the "Pythonification" of various QEMU
>> bits, I think you should loop him into this discussion, too.
>>
>>   Thomas
> 
> submodule does not mean we fork necessarily. We could have
> all options: check for the module and use it if there, if not
> use one from system if not there install with pip ..
> But yea, I'm not sure what's best either.

submodules create a dependency on an internet connection, too. So before you 
add yet another submodule (which have a couple of other disadvantages), I 
think you could also directly use the venv here.

  Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > packages from the net during testing?
> > > > > > > > 
> > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > Following two are downloaded:
> > > > > > > > avocado-framework==88.1
> > > > > > > > pycdlib==1.11.0
> > > > > > > > 
> > > > > > > > Also see this line in Makefie.include:
> > > > > > > > 
> > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > 
> > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > the net anyway.
> > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > 
> > > > > > Currently I only need this:
> > > > > > https://github.com/python-tap/tappy
> > > > > > which is the basic TAP processing library for python.
> > > > > > 
> > > > > > It seems its only installed through pip:
> > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > 
> > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > the python src tree? Not sure ...
> > > > > 
> > > > > It's pretty small for sure. Another submodule?
> > > > 
> > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > will receive new releases through
> > > > https://pypi.org/project/tap.py/
> > > > so forking is OK but someone has to keep this updated.
> > > > 
> > > > I am open to anything. Whatever feels right is fine to me.
> > > 
> > > John Snow is currently working on the "Pythonification" of various QEMU
> > > bits, I think you should loop him into this discussion, too.
> > > 
> > >   Thomas
> > 
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
> 
> submodules create a dependency on an internet connection, too. So before you
> add yet another submodule (which have a couple of other disadvantages), I
> think you could also directly use the venv here.

Definitely not submodules.

We need to get out of the mindset that submodules are needed for every new
dependancy we add. Submodules are only appropriate if the external project
is designed to be used as a copylib (eg the keycodemapdb tool), or if we
need to bundle in order to prevent a regression for previously deployed
QEMU installs where the dependancy is known not to exist on all our
supported platforms.

This does not apply in this case, because the proposed use of tappy is
merely for a test case. Meson just needs to check if tappy exists and if
it does, then use it, otherwise skip the tests that need it. The user can
arrange to install tappy, as they do with the majority of other deps.

If John's venv stuff is relevant, then we don't even need the meson checks,
just delegate to the venv setup.

Regardless, no submodules are needed or desirable.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > packages from the net during testing?
> > > > > > > > >
> > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > Following two are downloaded:
> > > > > > > > > avocado-framework==88.1
> > > > > > > > > pycdlib==1.11.0
> > > > > > > > >
> > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > >
> > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > >
> > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > the net anyway.
> > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > >
> > > > > > > Currently I only need this:
> > > > > > > https://github.com/python-tap/tappy
> > > > > > > which is the basic TAP processing library for python.
> > > > > > >
> > > > > > > It seems its only installed through pip:
> > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > >
> > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > the python src tree? Not sure ...
> > > > > >
> > > > > > It's pretty small for sure. Another submodule?
> > > > >
> > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > will receive new releases through
> > > > > https://pypi.org/project/tap.py/
> > > > > so forking is OK but someone has to keep this updated.
> > > > >
> > > > > I am open to anything. Whatever feels right is fine to me.
> > > >
> > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > bits, I think you should loop him into this discussion, too.
> > > >
> > > >   Thomas
> > >
> > > submodule does not mean we fork necessarily. We could have
> > > all options: check for the module and use it if there, if not
> > > use one from system if not there install with pip ..
> > > But yea, I'm not sure what's best either.
> >
> > submodules create a dependency on an internet connection, too. So before you
> > add yet another submodule (which have a couple of other disadvantages), I
> > think you could also directly use the venv here.
>
> Definitely not submodules.
>
> We need to get out of the mindset that submodules are needed for every new
> dependancy we add. Submodules are only appropriate if the external project
> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> need to bundle in order to prevent a regression for previously deployed
> QEMU installs where the dependancy is known not to exist on all our
> supported platforms.
>
> This does not apply in this case, because the proposed use of tappy is
> merely for a test case. Meson just needs to check if tappy exists and if
> it does, then use it, otherwise skip the tests that need it. The user can
> arrange to install tappy, as they do with the majority of other deps.
>
> If John's venv stuff is relevant, then we don't even need the meson checks,
> just delegate to the venv setup.
>
> Regardless, no submodules are needed or desirable.

What about keeping biosbits stuff? Source or pre-built.


>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > > packages from the net during testing?
> > > > > > > > > >
> > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > Following two are downloaded:
> > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > >
> > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > >
> > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > >
> > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > the net anyway.
> > > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > >
> > > > > > > > Currently I only need this:
> > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > which is the basic TAP processing library for python.
> > > > > > > >
> > > > > > > > It seems its only installed through pip:
> > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > >
> > > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > > the python src tree? Not sure ...
> > > > > > >
> > > > > > > It's pretty small for sure. Another submodule?
> > > > > >
> > > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > > will receive new releases through
> > > > > > https://pypi.org/project/tap.py/
> > > > > > so forking is OK but someone has to keep this updated.
> > > > > >
> > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > >
> > > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > > bits, I think you should loop him into this discussion, too.
> > > > >
> > > > >   Thomas
> > > >
> > > > submodule does not mean we fork necessarily. We could have
> > > > all options: check for the module and use it if there, if not
> > > > use one from system if not there install with pip ..
> > > > But yea, I'm not sure what's best either.
> > >
> > > submodules create a dependency on an internet connection, too. So before you
> > > add yet another submodule (which have a couple of other disadvantages), I
> > > think you could also directly use the venv here.
> >
> > Definitely not submodules.
> >
> > We need to get out of the mindset that submodules are needed for every new
> > dependancy we add. Submodules are only appropriate if the external project
> > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > need to bundle in order to prevent a regression for previously deployed
> > QEMU installs where the dependancy is known not to exist on all our
> > supported platforms.
> >
> > This does not apply in this case, because the proposed use of tappy is
> > merely for a test case. Meson just needs to check if tappy exists and if
> > it does, then use it, otherwise skip the tests that need it. The user can
> > arrange to install tappy, as they do with the majority of other deps.
> >
> > If John's venv stuff is relevant, then we don't even need the meson checks,
> > just delegate to the venv setup.
> >
> > Regardless, no submodules are needed or desirable.
> 
> What about keeping biosbits stuff? Source or pre-built.

Shipping them as pre-built binaries in QEMU is not a viable option
IMHO, especially for grub as a GPL'd project we need to be extremely
clear about the exact corresponding source and build process for any
binary.

For this kind of thing I would generally expect the distro to provide
packages that we consume. Looking at biosbits I see it is itself
bundling a bunch more 3rd party projects, libffi, grub2, and including
even an ancient version of python as a submodule.

So bundling a pre-built biosbits in QEMU appears to mean that we're in
turn going to unexpectedly bundle a bunch of other 3rd party projects
too, all with dubious license compliance. I don't think this looks like
something we should have in qemu.git or qemu tarballs. It will also
make it challenging for the distro to take biosbits at all, unless
those 3rd party bundles can be eliminated in favour of using existing
builds their have packaged for grub, python, libffi, etc.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
>> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>
>>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
>>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
>>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
>>>>>>>>>>>> packages from the net during testing?
>>>>>>>>>>>
>>>>>>>>>>> We do that too. See requirements.txt in tests/
>>>>>>>>>>> Following two are downloaded:
>>>>>>>>>>> avocado-framework==88.1
>>>>>>>>>>> pycdlib==1.11.0
>>>>>>>>>>>
>>>>>>>>>>> Also see this line in Makefie.include:
>>>>>>>>>>>
>>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>>>>>>>>>>
>>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
>>>>>>>>>> the net anyway.
>>>>>>>>>> Are the libraries in question not packaged on major distros?
>>>>>>>>>
>>>>>>>>> Currently I only need this:
>>>>>>>>> https://github.com/python-tap/tappy
>>>>>>>>> which is the basic TAP processing library for python.
>>>>>>>>>
>>>>>>>>> It seems its only installed through pip:
>>>>>>>>> https://tappy.readthedocs.io/en/latest/
>>>>>>>>>
>>>>>>>>> I do not think this is packaged by default. It's such a basic library
>>>>>>>>> for parsing test output that maybe we can keep this somewhere within
>>>>>>>>> the python src tree? Not sure ...
>>>>>>>>
>>>>>>>> It's pretty small for sure. Another submodule?
>>>>>>>
>>>>>>> Unlike BITS, this one is likely going to be maintained for a while and
>>>>>>> will receive new releases through
>>>>>>> https://pypi.org/project/tap.py/
>>>>>>> so forking is OK but someone has to keep this updated.
>>>>>>>
>>>>>>> I am open to anything. Whatever feels right is fine to me.
>>>>>>
>>>>>> John Snow is currently working on the "Pythonification" of various QEMU
>>>>>> bits, I think you should loop him into this discussion, too.
>>>>>>
>>>>>>    Thomas
>>>>>
>>>>> submodule does not mean we fork necessarily. We could have
>>>>> all options: check for the module and use it if there, if not
>>>>> use one from system if not there install with pip ..
>>>>> But yea, I'm not sure what's best either.
>>>>
>>>> submodules create a dependency on an internet connection, too. So before you
>>>> add yet another submodule (which have a couple of other disadvantages), I
>>>> think you could also directly use the venv here.
>>>
>>> Definitely not submodules.
>>>
>>> We need to get out of the mindset that submodules are needed for every new
>>> dependancy we add. Submodules are only appropriate if the external project
>>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
>>> need to bundle in order to prevent a regression for previously deployed
>>> QEMU installs where the dependancy is known not to exist on all our
>>> supported platforms.
>>>
>>> This does not apply in this case, because the proposed use of tappy is
>>> merely for a test case. Meson just needs to check if tappy exists and if
>>> it does, then use it, otherwise skip the tests that need it. The user can
>>> arrange to install tappy, as they do with the majority of other deps.
>>>
>>> If John's venv stuff is relevant, then we don't even need the meson checks,
>>> just delegate to the venv setup.
>>>
>>> Regardless, no submodules are needed or desirable.
>>
>> What about keeping biosbits stuff? Source or pre-built.
> 
> Shipping them as pre-built binaries in QEMU is not a viable option
> IMHO, especially for grub as a GPL'd project we need to be extremely
> clear about the exact corresponding source and build process for any
> binary.
> 
> For this kind of thing I would generally expect the distro to provide
> packages that we consume. Looking at biosbits I see it is itself
> bundling a bunch more 3rd party projects, libffi, grub2, and including
> even an ancient version of python as a submodule.
> 
> So bundling a pre-built biosbits in QEMU appears to mean that we're in
> turn going to unexpectedly bundle a bunch of other 3rd party projects
> too, all with dubious license compliance. I don't think this looks like
> something we should have in qemu.git or qemu tarballs. It will also
> make it challenging for the distro to take biosbits at all, unless
> those 3rd party bundles can be eliminated in favour of using existing
> builds their have packaged for grub, python, libffi, etc.

So if this depends on some third party binary bits, I think this is pretty 
similar to the tests in the avocado directory ... there we download third 
party binaries, too... Wouldn't it make sense to adapt your tests to that 
framework?

  Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >>>
> >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
> >>>>>>>>>>>> packages from the net during testing?
> >>>>>>>>>>>
> >>>>>>>>>>> We do that too. See requirements.txt in tests/
> >>>>>>>>>>> Following two are downloaded:
> >>>>>>>>>>> avocado-framework==88.1
> >>>>>>>>>>> pycdlib==1.11.0
> >>>>>>>>>>>
> >>>>>>>>>>> Also see this line in Makefie.include:
> >>>>>>>>>>>
> >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> >>>>>>>>>>
> >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
> >>>>>>>>>> the net anyway.
> >>>>>>>>>> Are the libraries in question not packaged on major distros?
> >>>>>>>>>
> >>>>>>>>> Currently I only need this:
> >>>>>>>>> https://github.com/python-tap/tappy
> >>>>>>>>> which is the basic TAP processing library for python.
> >>>>>>>>>
> >>>>>>>>> It seems its only installed through pip:
> >>>>>>>>> https://tappy.readthedocs.io/en/latest/
> >>>>>>>>>
> >>>>>>>>> I do not think this is packaged by default. It's such a basic library
> >>>>>>>>> for parsing test output that maybe we can keep this somewhere within
> >>>>>>>>> the python src tree? Not sure ...
> >>>>>>>>
> >>>>>>>> It's pretty small for sure. Another submodule?
> >>>>>>>
> >>>>>>> Unlike BITS, this one is likely going to be maintained for a while and
> >>>>>>> will receive new releases through
> >>>>>>> https://pypi.org/project/tap.py/
> >>>>>>> so forking is OK but someone has to keep this updated.
> >>>>>>>
> >>>>>>> I am open to anything. Whatever feels right is fine to me.
> >>>>>>
> >>>>>> John Snow is currently working on the "Pythonification" of various QEMU
> >>>>>> bits, I think you should loop him into this discussion, too.
> >>>>>>
> >>>>>>    Thomas
> >>>>>
> >>>>> submodule does not mean we fork necessarily. We could have
> >>>>> all options: check for the module and use it if there, if not
> >>>>> use one from system if not there install with pip ..
> >>>>> But yea, I'm not sure what's best either.
> >>>>
> >>>> submodules create a dependency on an internet connection, too. So before you
> >>>> add yet another submodule (which have a couple of other disadvantages), I
> >>>> think you could also directly use the venv here.
> >>>
> >>> Definitely not submodules.
> >>>
> >>> We need to get out of the mindset that submodules are needed for every new
> >>> dependancy we add. Submodules are only appropriate if the external project
> >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> >>> need to bundle in order to prevent a regression for previously deployed
> >>> QEMU installs where the dependancy is known not to exist on all our
> >>> supported platforms.
> >>>
> >>> This does not apply in this case, because the proposed use of tappy is
> >>> merely for a test case. Meson just needs to check if tappy exists and if
> >>> it does, then use it, otherwise skip the tests that need it. The user can
> >>> arrange to install tappy, as they do with the majority of other deps.
> >>>
> >>> If John's venv stuff is relevant, then we don't even need the meson checks,
> >>> just delegate to the venv setup.
> >>>
> >>> Regardless, no submodules are needed or desirable.
> >>
> >> What about keeping biosbits stuff? Source or pre-built.
> >
> > Shipping them as pre-built binaries in QEMU is not a viable option
> > IMHO, especially for grub as a GPL'd project we need to be extremely
> > clear about the exact corresponding source and build process for any
> > binary.
> >
> > For this kind of thing I would generally expect the distro to provide
> > packages that we consume. Looking at biosbits I see it is itself
> > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > even an ancient version of python as a submodule.
> >
> > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > too, all with dubious license compliance. I don't think this looks like
> > something we should have in qemu.git or qemu tarballs. It will also
> > make it challenging for the distro to take biosbits at all, unless
> > those 3rd party bundles can be eliminated in favour of using existing
> > builds their have packaged for grub, python, libffi, etc.
>
> So if this depends on some third party binary bits, I think this is pretty
> similar to the tests in the avocado directory ... there we download third
> party binaries, too... Wouldn't it make sense to adapt your tests to that
> framework?

I do not want to bring in the whole avocado framework because it would
unnecessarily make things complicated. I just need the qemu machine
python library and that is enough. For downloading third party stuff,
we can simply wget things from somewhere.

>
>   Thomas
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

[-- Attachment #1: Type: text/plain, Size: 5825 bytes --]

On Tue, Jun 28, 2022 at 14:05 Ani Sinha <ani@anisinha.ca> wrote:

> On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <
> berrange@redhat.com> wrote:
> > >>>
> > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip
> and pulling
> > >>>>>>>>>>>> packages from the net during testing?
> > >>>>>>>>>>>
> > >>>>>>>>>>> We do that too. See requirements.txt in tests/
> > >>>>>>>>>>> Following two are downloaded:
> > >>>>>>>>>>> avocado-framework==88.1
> > >>>>>>>>>>> pycdlib==1.11.0
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also see this line in Makefie.include:
> > >>>>>>>>>>>
> > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > >>>>>>>>>>
> > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
> > >>>>>>>>>> the net anyway.
> > >>>>>>>>>> Are the libraries in question not packaged on major distros?
> > >>>>>>>>>
> > >>>>>>>>> Currently I only need this:
> > >>>>>>>>> https://github.com/python-tap/tappy
> > >>>>>>>>> which is the basic TAP processing library for python.
> > >>>>>>>>>
> > >>>>>>>>> It seems its only installed through pip:
> > >>>>>>>>> https://tappy.readthedocs.io/en/latest/
> > >>>>>>>>>
> > >>>>>>>>> I do not think this is packaged by default. It's such a basic
> library
> > >>>>>>>>> for parsing test output that maybe we can keep this somewhere
> within
> > >>>>>>>>> the python src tree? Not sure ...
> > >>>>>>>>
> > >>>>>>>> It's pretty small for sure. Another submodule?
> > >>>>>>>
> > >>>>>>> Unlike BITS, this one is likely going to be maintained for a
> while and
> > >>>>>>> will receive new releases through
> > >>>>>>> https://pypi.org/project/tap.py/
> > >>>>>>> so forking is OK but someone has to keep this updated.
> > >>>>>>>
> > >>>>>>> I am open to anything. Whatever feels right is fine to me.
> > >>>>>>
> > >>>>>> John Snow is currently working on the "Pythonification" of
> various QEMU
> > >>>>>> bits, I think you should loop him into this discussion, too.
> > >>>>>>
> > >>>>>>    Thomas
> > >>>>>
> > >>>>> submodule does not mean we fork necessarily. We could have
> > >>>>> all options: check for the module and use it if there, if not
> > >>>>> use one from system if not there install with pip ..
> > >>>>> But yea, I'm not sure what's best either.
> > >>>>
> > >>>> submodules create a dependency on an internet connection, too. So
> before you
> > >>>> add yet another submodule (which have a couple of other
> disadvantages), I
> > >>>> think you could also directly use the venv here.
> > >>>
> > >>> Definitely not submodules.
> > >>>
> > >>> We need to get out of the mindset that submodules are needed for
> every new
> > >>> dependancy we add. Submodules are only appropriate if the external
> project
> > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or
> if we
> > >>> need to bundle in order to prevent a regression for previously
> deployed
> > >>> QEMU installs where the dependancy is known not to exist on all our
> > >>> supported platforms.
> > >>>
> > >>> This does not apply in this case, because the proposed use of tappy
> is
> > >>> merely for a test case. Meson just needs to check if tappy exists
> and if
> > >>> it does, then use it, otherwise skip the tests that need it. The
> user can
> > >>> arrange to install tappy, as they do with the majority of other deps.
> > >>>
> > >>> If John's venv stuff is relevant, then we don't even need the meson
> checks,
> > >>> just delegate to the venv setup.
> > >>>
> > >>> Regardless, no submodules are needed or desirable.
> > >>
> > >> What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > even an ancient version of python as a submodule.
> > >
> > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > too, all with dubious license compliance. I don't think this looks like
> > > something we should have in qemu.git or qemu tarballs. It will also
> > > make it challenging for the distro to take biosbits at all, unless
> > > those 3rd party bundles can be eliminated in favour of using existing
> > > builds their have packaged for grub, python, libffi, etc.
> >
> > So if this depends on some third party binary bits, I think this is
> pretty
> > similar to the tests in the avocado directory ... there we download third
> > party binaries, too... Wouldn't it make sense to adapt your tests to that
> > framework?
>
> I do not want to bring in the whole avocado framework because it would
> unnecessarily make things complicated. I just need the qemu machine
> python library and that is enough. For downloading third party stuff,

we can simply wget things from somewhere.


https://pypi.org/project/wget/

That get_asset() call is an overkill for downloading two archives.


>
> >
> >   Thomas
> >
>

[-- Attachment #2: Type: text/html, Size: 8879 bytes --]

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 02:19:41PM +0530, Ani Sinha wrote:
> 
> 
> On Tue, Jun 28, 2022 at 14:05 Ani Sinha <ani@anisinha.ca> wrote:
> 
>     On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote:
>     >
>     > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
>     > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
>     > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <
>     berrange@redhat.com> wrote:
>     > >>>
>     > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
>     > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
>     > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>     > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and
>     pulling
>     > >>>>>>>>>>>> packages from the net during testing?
>     > >>>>>>>>>>>
>     > >>>>>>>>>>> We do that too. See requirements.txt in tests/
>     > >>>>>>>>>>> Following two are downloaded:
>     > >>>>>>>>>>> avocado-framework==88.1
>     > >>>>>>>>>>> pycdlib==1.11.0
>     > >>>>>>>>>>>
>     > >>>>>>>>>>> Also see this line in Makefie.include:
>     > >>>>>>>>>>>
>     > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>     > >>>>>>>>>>
>     > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
>     > >>>>>>>>>> the net anyway.
>     > >>>>>>>>>> Are the libraries in question not packaged on major distros?
>     > >>>>>>>>>
>     > >>>>>>>>> Currently I only need this:
>     > >>>>>>>>> https://github.com/python-tap/tappy
>     > >>>>>>>>> which is the basic TAP processing library for python.
>     > >>>>>>>>>
>     > >>>>>>>>> It seems its only installed through pip:
>     > >>>>>>>>> https://tappy.readthedocs.io/en/latest/
>     > >>>>>>>>>
>     > >>>>>>>>> I do not think this is packaged by default. It's such a basic
>     library
>     > >>>>>>>>> for parsing test output that maybe we can keep this somewhere
>     within
>     > >>>>>>>>> the python src tree? Not sure ...
>     > >>>>>>>>
>     > >>>>>>>> It's pretty small for sure. Another submodule?
>     > >>>>>>>
>     > >>>>>>> Unlike BITS, this one is likely going to be maintained for a
>     while and
>     > >>>>>>> will receive new releases through
>     > >>>>>>> https://pypi.org/project/tap.py/
>     > >>>>>>> so forking is OK but someone has to keep this updated.
>     > >>>>>>>
>     > >>>>>>> I am open to anything. Whatever feels right is fine to me.
>     > >>>>>>
>     > >>>>>> John Snow is currently working on the "Pythonification" of various
>     QEMU
>     > >>>>>> bits, I think you should loop him into this discussion, too.
>     > >>>>>>
>     > >>>>>>    Thomas
>     > >>>>>
>     > >>>>> submodule does not mean we fork necessarily. We could have
>     > >>>>> all options: check for the module and use it if there, if not
>     > >>>>> use one from system if not there install with pip ..
>     > >>>>> But yea, I'm not sure what's best either.
>     > >>>>
>     > >>>> submodules create a dependency on an internet connection, too. So
>     before you
>     > >>>> add yet another submodule (which have a couple of other
>     disadvantages), I
>     > >>>> think you could also directly use the venv here.
>     > >>>
>     > >>> Definitely not submodules.
>     > >>>
>     > >>> We need to get out of the mindset that submodules are needed for
>     every new
>     > >>> dependancy we add. Submodules are only appropriate if the external
>     project
>     > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if
>     we
>     > >>> need to bundle in order to prevent a regression for previously
>     deployed
>     > >>> QEMU installs where the dependancy is known not to exist on all our
>     > >>> supported platforms.
>     > >>>
>     > >>> This does not apply in this case, because the proposed use of tappy
>     is
>     > >>> merely for a test case. Meson just needs to check if tappy exists and
>     if
>     > >>> it does, then use it, otherwise skip the tests that need it. The user
>     can
>     > >>> arrange to install tappy, as they do with the majority of other deps.
>     > >>>
>     > >>> If John's venv stuff is relevant, then we don't even need the meson
>     checks,
>     > >>> just delegate to the venv setup.
>     > >>>
>     > >>> Regardless, no submodules are needed or desirable.
>     > >>
>     > >> What about keeping biosbits stuff? Source or pre-built.
>     > >
>     > > Shipping them as pre-built binaries in QEMU is not a viable option
>     > > IMHO, especially for grub as a GPL'd project we need to be extremely
>     > > clear about the exact corresponding source and build process for any
>     > > binary.
>     > >
>     > > For this kind of thing I would generally expect the distro to provide
>     > > packages that we consume. Looking at biosbits I see it is itself
>     > > bundling a bunch more 3rd party projects, libffi, grub2, and including
>     > > even an ancient version of python as a submodule.
>     > >
>     > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
>     > > turn going to unexpectedly bundle a bunch of other 3rd party projects
>     > > too, all with dubious license compliance. I don't think this looks like
>     > > something we should have in qemu.git or qemu tarballs. It will also
>     > > make it challenging for the distro to take biosbits at all, unless
>     > > those 3rd party bundles can be eliminated in favour of using existing
>     > > builds their have packaged for grub, python, libffi, etc.
>     >
>     > So if this depends on some third party binary bits, I think this is
>     pretty
>     > similar to the tests in the avocado directory ... there we download third
>     > party binaries, too... Wouldn't it make sense to adapt your tests to that
>     > framework?
> 
>     I do not want to bring in the whole avocado framework because it would
>     unnecessarily make things complicated. I just need the qemu machine
>     python library and that is enough. For downloading third party stuff,
> 
>     we can simply wget things from somewhere.
> 
> 
> https://pypi.org/project/wget/
> 
> That get_asset() call is an overkill for downloading two archives. 
> 

For biosbits if we are going this route then I feel a submodule is much
better.  It records which version exactly each qemu version wants.


> 
> 
>     >
>     >   Thomas
>     >
> 

Why we should avoid new submodules if possible

[Thomas Huth]

On 28/06/2022 12.03, Michael S. Tsirkin wrote:
[...]
> For biosbits if we are going this route then I feel a submodule is much
> better.  It records which version exactly each qemu version wants.

As far as I know, you can also specify the version when using pip, can't 
you? So that's not really an advantage here.

On the contrary, submodules have a couple of disadvantages that I really 
dislike:

- submodules do not get updated automatically when doing a "git checkout", 
we have to update them via a script instead. This causes e.g. trouble if you 
rsync your source tree to a machine that has no access to the internet and 
you forgot to update the submodule before the sync

- the content of submodules is not added to the tarballs that get created on 
the git forges automatically. There were lots of requests from users in the 
past that tried to download a tarball from github and then wondered why they 
couldn't compile QEMU.

- we include the submodule content in our release tarballs, so people get 
the impression that hte submodule content is part of the QEMU sources. This 
has two disadvantages:
  * We already got bug reports for the code in the submodule,
    where people did not understand that they should report that
    rather to the original project instead (i.e. you ship it - you
    own it)
  * People get the impression that QEMU is a huge monster
    application if they count the number of code lines, run
    their code scanner tools on the tarball contents, etc.
    Remember "nemu", for example, where one of the main complaints
    was that QEMU has too many lines of code?

- If programs includes code via submodules, this gets a higher
   burder for distro maintainers, since they have to patch each
   and every package when there is a bug, instead of being able to
   fix it in one central place.

So in my opinion we should avoid new submodules if there is an alternative.

  Thomas

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> [...]
> > For biosbits if we are going this route then I feel a submodule is much
> > better.  It records which version exactly each qemu version wants.
> 
> As far as I know, you can also specify the version when using pip, can't
> you? So that's not really an advantage here.

But of course if you do you do not get updates ;) You do
however rely on a 3rd party to faithfully provide you
correct code based on the version, and host it forever.

> On the contrary, submodules have a couple of disadvantages that I really
> dislike:
> 
> - submodules do not get updated automatically when doing a "git checkout",
> we have to update them via a script instead. This causes e.g. trouble if you
> rsync your source tree to a machine that has no access to the internet and
> you forgot to update the submodule before the sync

how is pip better?

> - the content of submodules is not added to the tarballs that get created on
> the git forges automatically. There were lots of requests from users in the
> past that tried to download a tarball from github and then wondered why they
> couldn't compile QEMU.

how is pip better here?

> - we include the submodule content in our release tarballs, so people get
> the impression that hte submodule content is part of the QEMU sources. This
> has two disadvantages:
>  * We already got bug reports for the code in the submodule,
>    where people did not understand that they should report that
>    rather to the original project instead (i.e. you ship it - you
>    own it)
>  * People get the impression that QEMU is a huge monster
>    application if they count the number of code lines, run
>    their code scanner tools on the tarball contents, etc.
>    Remember "nemu", for example, where one of the main complaints
>    was that QEMU has too many lines of code?

I think we can skip the checkout in the tarball if we like.
If people want to run the test they can checkout then.


> - If programs includes code via submodules, this gets a higher
>   burder for distro maintainers, since they have to patch each
>   and every package when there is a bug, instead of being able to
>   fix it in one central place.

Come on, this is just a test. We *really* don't care if an ISO
we use to test ACPI is using an exploitable version of grub.

> So in my opinion we should avoid new submodules if there is an alternative.
> 
>  Thomas

Interesting take generally, but I don't see how the logic applies in this
case. Would appreciate hearing your answers to the above.

-- 
MST

Re: Why we should avoid new submodules if possible

[Peter Maydell]

On Tue, 28 Jun 2022 at 11:38, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > - we include the submodule content in our release tarballs, so people get
> > the impression that hte submodule content is part of the QEMU sources. This
> > has two disadvantages:
> >  * We already got bug reports for the code in the submodule,
> >    where people did not understand that they should report that
> >    rather to the original project instead (i.e. you ship it - you
> >    own it)
> >  * People get the impression that QEMU is a huge monster
> >    application if they count the number of code lines, run
> >    their code scanner tools on the tarball contents, etc.
> >    Remember "nemu", for example, where one of the main complaints
> >    was that QEMU has too many lines of code?
>
> I think we can skip the checkout in the tarball if we like.
> If people want to run the test they can checkout then.

For tarballs and submodules, we want to provide the code in the
cases where we're providing binary blobs, and for where it's
required to build QEMU proper.

Overall I think that the approach we use today for providing
guest binaries (submodules with the code, pre-built blobs checked
into git) is creaking at the seams and often awkward for downstream
distros (who want to rebuild the binaries anyway).

Plus submodules in general in git work really badly and awkwardly,
and I'd rather we didn't add them unless we really must.

We already have an approach for "tests that use binaries" --
the avocado test suites. Is that something we could use in this
case ?

thanks
-- PMM

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:43:58AM +0100, Peter Maydell wrote:
> On Tue, 28 Jun 2022 at 11:38, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > - we include the submodule content in our release tarballs, so people get
> > > the impression that hte submodule content is part of the QEMU sources. This
> > > has two disadvantages:
> > >  * We already got bug reports for the code in the submodule,
> > >    where people did not understand that they should report that
> > >    rather to the original project instead (i.e. you ship it - you
> > >    own it)
> > >  * People get the impression that QEMU is a huge monster
> > >    application if they count the number of code lines, run
> > >    their code scanner tools on the tarball contents, etc.
> > >    Remember "nemu", for example, where one of the main complaints
> > >    was that QEMU has too many lines of code?
> >
> > I think we can skip the checkout in the tarball if we like.
> > If people want to run the test they can checkout then.
> 
> For tarballs and submodules, we want to provide the code in the
> cases where we're providing binary blobs, and for where it's
> required to build QEMU proper.
> 
> Overall I think that the approach we use today for providing
> guest binaries (submodules with the code, pre-built blobs checked
> into git) is creaking at the seams and often awkward for downstream
> distros (who want to rebuild the binaries anyway).
> 
> Plus submodules in general in git work really badly and awkwardly,
> and I'd rather we didn't add them unless we really must.
> 
> We already have an approach for "tests that use binaries" --
> the avocado test suites. Is that something we could use in this
> case ?
> 
> thanks
> -- PMM

From experience avocado is basically rolling its own reimplementation of
git fetch. Result is weird failures when some corner case or another
isn't handled perfectly. Kind of unavoidable since you can't have
ISOs for a ton of guest OSes reasonably in a git repo. Well maybe
with sparse checkouts you can. Haven't really looked at it.

These specific tests are weird, kind of special in that they are developer
oriented.  I *would* like anyone making changes to ACPI to run them, and
for CI to run them on patches, but I do not believe e.g. distros
necessarily need them.

git submodules are awkward basically because they are an automated wget.
I don't think an explicit wget is much better ... but
looks like I'm alone in this. Oh well.
So it will be a weird dance of wget a tarball, unpack, generate
ISO and run. God help you if you need to patch the test - it's
wget all the way down.


-- 
MST

Re: Why we should avoid new submodules if possible

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 596 bytes --]

On Tue, Jun 28, 2022 at 5:10 AM Michael S. Tsirkin <mst@redhat.com> wrote:

> git submodules are awkward basically because they are an automated wget.
> I don't think an explicit wget is much better ... but
> looks like I'm alone in this. Oh well.
>

submodules are 90% of the hassles I have  in upstreaming and updating the
bsd-user stuff. They play terribly with branches, rebases, and any number of
other things and are a pain to reset at times. They are easily the most
brittle,
error-prone and difficult to use feature of git.

I hate them with a burning passion. You are not alone.

Warner

[-- Attachment #2: Type: text/html, Size: 1013 bytes --]

Re: Why we should avoid new submodules if possible

[Michal Suchánek]

Hello,

On Tue, Jun 28, 2022 at 07:00:59AM -0400, Michael S. Tsirkin wrote:

> 
> git submodules are awkward basically because they are an automated wget.
> I don't think an explicit wget is much better ... but
> looks like I'm alone in this. Oh well.
> So it will be a weird dance of wget a tarball, unpack, generate
> ISO and run. God help you if you need to patch the test - it's
> wget all the way down.

That's the problem - the submodules are not automated. They are
half-automated, and the rules for when the automation works and for when
the automation falls apart are not intellibible for the general Joe
Developer.

You might spend a few days studying how they behave exactly, and then you
will know. But unless you will use them every day you will forget again,
because the rules do not lend themselves to some abstraction easily
understood by humans.

Thanks

Michal

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Wed, Sep 28, 2022 at 10:48:03PM +0200, Michal Suchánek wrote:
> Hello,
> 
> On Tue, Jun 28, 2022 at 07:00:59AM -0400, Michael S. Tsirkin wrote:
> 
> > 
> > git submodules are awkward basically because they are an automated wget.
> > I don't think an explicit wget is much better ... but
> > looks like I'm alone in this. Oh well.
> > So it will be a weird dance of wget a tarball, unpack, generate
> > ISO and run. God help you if you need to patch the test - it's
> > wget all the way down.
> 
> That's the problem - the submodules are not automated. They are
> half-automated, and the rules for when the automation works and for when
> the automation falls apart are not intellibible for the general Joe
> Developer.
> 
> You might spend a few days studying how they behave exactly, and then you
> will know. But unless you will use them every day you will forget again,
> because the rules do not lend themselves to some abstraction easily
> understood by humans.
> 
> Thanks
> 
> Michal

But why would learning a different tool be easier?

-- 
MST

Re: Why we should avoid new submodules if possible

[Michal Suchánek]

On Wed, Sep 28, 2022 at 05:07:48PM -0400, Michael S. Tsirkin wrote:
> On Wed, Sep 28, 2022 at 10:48:03PM +0200, Michal Suchánek wrote:
> > Hello,
> > 
> > On Tue, Jun 28, 2022 at 07:00:59AM -0400, Michael S. Tsirkin wrote:
> > 
> > > 
> > > git submodules are awkward basically because they are an automated wget.
> > > I don't think an explicit wget is much better ... but
> > > looks like I'm alone in this. Oh well.
> > > So it will be a weird dance of wget a tarball, unpack, generate
> > > ISO and run. God help you if you need to patch the test - it's
> > > wget all the way down.
> > 
> > That's the problem - the submodules are not automated. They are
> > half-automated, and the rules for when the automation works and for when
> > the automation falls apart are not intellibible for the general Joe
> > Developer.
> > 
> > You might spend a few days studying how they behave exactly, and then you
> > will know. But unless you will use them every day you will forget again,
> > because the rules do not lend themselves to some abstraction easily
> > understood by humans.
> > 
> > Thanks
> > 
> > Michal
> 
> But why would learning a different tool be easier?

a) it's working lends itself to explaining in human-intelligible
concepts

b) it's used elsewhere as well

Thanks

Michal

Re: Why we should avoid new submodules if possible

[Thomas Huth]

On 28/06/2022 12.30, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
>> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
>> [...]
>>> For biosbits if we are going this route then I feel a submodule is much
>>> better.  It records which version exactly each qemu version wants.
>>
>> As far as I know, you can also specify the version when using pip, can't
>> you? So that's not really an advantage here.
> 
> But of course if you do you do not get updates ;) You do
> however rely on a 3rd party to faithfully provide you
> correct code based on the version, and host it forever.
> 
>> On the contrary, submodules have a couple of disadvantages that I really
>> dislike:
>>
>> - submodules do not get updated automatically when doing a "git checkout",
>> we have to update them via a script instead. This causes e.g. trouble if you
>> rsync your source tree to a machine that has no access to the internet and
>> you forgot to update the submodule before the sync
> 
> how is pip better?

You don't end up with an inconsistent source tree in that case (which 
happens with submodules quite easily, at least for me it happened a couple 
of times already). Either the machine has an internet connection, so that 
pip can install the missing bits, or it does not and the test has to be 
skipped. But if I copy the wrong state of a submodule around, things get 
messed up quite easily in my experience. Ok, you could say that this is just 
my special setup with rsync, but already given the fact that "git checkout" 
creates an inconsistent state of your source tree until you run the script 
for updating the submodules the next time is an indication that submodules 
are rather a shaky thing (e.g. if you'd create a tarball for somebody else 
from your source tree right after doing a "git checkout").

>> - the content of submodules is not added to the tarballs that get created on
>> the git forges automatically. There were lots of requests from users in the
>> past that tried to download a tarball from github and then wondered why they
>> couldn't compile QEMU.
> 
> how is pip better here?

You don't get incomplete/non-working tarballs in that case.

>> - we include the submodule content in our release tarballs, so people get
>> the impression that hte submodule content is part of the QEMU sources. This
>> has two disadvantages:
>>   * We already got bug reports for the code in the submodule,
>>     where people did not understand that they should report that
>>     rather to the original project instead (i.e. you ship it - you
>>     own it)
>>   * People get the impression that QEMU is a huge monster
>>     application if they count the number of code lines, run
>>     their code scanner tools on the tarball contents, etc.
>>     Remember "nemu", for example, where one of the main complaints
>>     was that QEMU has too many lines of code?
> 
> I think we can skip the checkout in the tarball if we like.
> If people want to run the test they can checkout then.

Release tarballs don't include the ".git" folder infrastructur, so everybody 
who downloads a tarball will simply never be able to run the test.

> 
>> - If programs includes code via submodules, this gets a higher
>>    burder for distro maintainers, since they have to patch each
>>    and every package when there is a bug, instead of being able to
>>    fix it in one central place.
> 
> Come on, this is just a test. We *really* don't care if an ISO
> we use to test ACPI is using an exploitable version of grub.

Wait, I thought we were only talking about tappy here? The ISO binaries 
should certainly *not* be bundled in the QEMU tarballs (they are too big 
already anyway, we should rather think of moving the firmware binaries out 
of the tarball instead).

  Thomas

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
> On 28/06/2022 12.30, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > [...]
> > > > For biosbits if we are going this route then I feel a submodule is much
> > > > better.  It records which version exactly each qemu version wants.
> > > 
> > > As far as I know, you can also specify the version when using pip, can't
> > > you? So that's not really an advantage here.
> > 
> > But of course if you do you do not get updates ;) You do
> > however rely on a 3rd party to faithfully provide you
> > correct code based on the version, and host it forever.
> > 
> > > On the contrary, submodules have a couple of disadvantages that I really
> > > dislike:
> > > 
> > > - submodules do not get updated automatically when doing a "git checkout",
> > > we have to update them via a script instead. This causes e.g. trouble if you
> > > rsync your source tree to a machine that has no access to the internet and
> > > you forgot to update the submodule before the sync
> > 
> > how is pip better?
> 
> You don't end up with an inconsistent source tree in that case (which
> happens with submodules quite easily, at least for me it happened a couple
> of times already). Either the machine has an internet connection, so that
> pip can install the missing bits, or it does not and the test has to be
> skipped.

skipped tests are too easy to ignore ...

> But if I copy the wrong state of a submodule around, things get
> messed up quite easily in my experience. Ok, you could say that this is just
> my special setup with rsync, but already given the fact that "git checkout"
> creates an inconsistent state of your source tree until you run the script
> for updating the submodules the next time is an indication that submodules
> are rather a shaky thing (e.g. if you'd create a tarball for somebody else
> from your source tree right after doing a "git checkout").

yea one has to remember to set submodule.recurse = true in .gitconfig

I agree it's annoying, I guess they don't change it for compat reasons.


> > > - the content of submodules is not added to the tarballs that get created on
> > > the git forges automatically. There were lots of requests from users in the
> > > past that tried to download a tarball from github and then wondered why they
> > > couldn't compile QEMU.
> > 
> > how is pip better here?
> 
> You don't get incomplete/non-working tarballs in that case.

So skip the test ;)

> > > - we include the submodule content in our release tarballs, so people get
> > > the impression that hte submodule content is part of the QEMU sources. This
> > > has two disadvantages:
> > >   * We already got bug reports for the code in the submodule,
> > >     where people did not understand that they should report that
> > >     rather to the original project instead (i.e. you ship it - you
> > >     own it)
> > >   * People get the impression that QEMU is a huge monster
> > >     application if they count the number of code lines, run
> > >     their code scanner tools on the tarball contents, etc.
> > >     Remember "nemu", for example, where one of the main complaints
> > >     was that QEMU has too many lines of code?
> > 
> > I think we can skip the checkout in the tarball if we like.
> > If people want to run the test they can checkout then.
> 
> Release tarballs don't include the ".git" folder infrastructur, so everybody
> who downloads a tarball will simply never be able to run the test.

I actually think I'm fine with that for this specific case.

> > 
> > > - If programs includes code via submodules, this gets a higher
> > >    burder for distro maintainers, since they have to patch each
> > >    and every package when there is a bug, instead of being able to
> > >    fix it in one central place.
> > 
> > Come on, this is just a test. We *really* don't care if an ISO
> > we use to test ACPI is using an exploitable version of grub.
> 
> Wait, I thought we were only talking about tappy here? The ISO binaries
> should certainly *not* be bundled in the QEMU tarballs (they are too big
> already anyway, we should rather think of moving the firmware binaries out
> of the tarball instead).
> 
>  Thomas

IIUC there are three things we are discussing
- biosbits source
- biosbits image
- tappy


-- 
MST

Re: Why we should avoid new submodules if possible

[Thomas Huth]

On 28/06/2022 13.14, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
[...]
>>> Come on, this is just a test. We *really* don't care if an ISO
>>> we use to test ACPI is using an exploitable version of grub.
>>
>> Wait, I thought we were only talking about tappy here? The ISO binaries
>> should certainly *not* be bundled in the QEMU tarballs (they are too big
>> already anyway, we should rather think of moving the firmware binaries out
>> of the tarball instead).
>>
>>   Thomas
> 
> IIUC there are three things we are discussing
> - biosbits source
> - biosbits image
> - tappy

Oh well, I missed that part of the discussion so far since the corresponding 
patches did not make it to the mailing list ¯\_(ツ)_/¯

Anyway, that's just another indication that it might not be the right fit 
for inclusion into the QEMU source tree. So either download it similar to 
(or included in) the avocado tests, or maybe another solution would be to 
have a separate "qemu-ci" or "qemu-testing" repository for stuff like this ... ?

  Thomas

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 02:39:31PM +0200, Thomas Huth wrote:
> On 28/06/2022 13.14, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
> [...]
> > > > Come on, this is just a test. We *really* don't care if an ISO
> > > > we use to test ACPI is using an exploitable version of grub.
> > > 
> > > Wait, I thought we were only talking about tappy here? The ISO binaries
> > > should certainly *not* be bundled in the QEMU tarballs (they are too big
> > > already anyway, we should rather think of moving the firmware binaries out
> > > of the tarball instead).
> > > 
> > >   Thomas
> > 
> > IIUC there are three things we are discussing
> > - biosbits source
> > - biosbits image
> > - tappy
> 
> Oh well, I missed that part of the discussion so far since the corresponding
> patches did not make it to the mailing list ¯\_(ツ)_/¯
> 
> Anyway, that's just another indication that it might not be the right fit
> for inclusion into the QEMU source tree. So either download it similar to
> (or included in) the avocado tests, or maybe another solution would be to
> have a separate "qemu-ci" or "qemu-testing" repository for stuff like this
> ... ?
> 
>  Thomas

I don't think anyone suggested sticking all those blobs in qemu.git: I
proposed a submodule, others proposed the web (fetched with avocado or
just wget). If it's a submodule we'll want to skip it in the tarball,
and skip the test if not checked out.  Whoever is trying to do
development out of a tarball should just stop :).

-- 
MST

Re: Why we should avoid new submodules if possible

[Ani Sinha]

On Tue, Jun 28, 2022 at 6:09 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 28/06/2022 13.14, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
> [...]
> >>> Come on, this is just a test. We *really* don't care if an ISO
> >>> we use to test ACPI is using an exploitable version of grub.
> >>
> >> Wait, I thought we were only talking about tappy here? The ISO binaries
> >> should certainly *not* be bundled in the QEMU tarballs (they are too big
> >> already anyway, we should rather think of moving the firmware binaries out
> >> of the tarball instead).
> >>
> >>   Thomas
> >
> > IIUC there are three things we are discussing
> > - biosbits source
> > - biosbits image
> > - tappy
>
> Oh well, I missed that part of the discussion so far since the corresponding
> patches did not make it to the mailing list ¯\_(ツ)_/¯
>
> Anyway, that's just another indication that it might not be the right fit
> for inclusion into the QEMU source tree. So either download it similar to
> (or included in) the avocado tests, or maybe another solution would be to
> have a separate "qemu-ci" or "qemu-testing" repository for stuff like this ... ?

Yes, I think we should have a separate repo for all testing related
stuff like blobs, guest images etc. Curently, some of them are in
personal github accounts which is not idea IMHo.

Bits stuff can reside in the same place.

Re: Why we should avoid new submodules if possible

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 09:24:34PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 6:09 PM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 28/06/2022 13.14, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 12:50:06PM +0200, Thomas Huth wrote:
> > [...]
> > >>> Come on, this is just a test. We *really* don't care if an ISO
> > >>> we use to test ACPI is using an exploitable version of grub.
> > >>
> > >> Wait, I thought we were only talking about tappy here? The ISO binaries
> > >> should certainly *not* be bundled in the QEMU tarballs (they are too big
> > >> already anyway, we should rather think of moving the firmware binaries out
> > >> of the tarball instead).
> > >>
> > >>   Thomas
> > >
> > > IIUC there are three things we are discussing
> > > - biosbits source
> > > - biosbits image
> > > - tappy
> >
> > Oh well, I missed that part of the discussion so far since the corresponding
> > patches did not make it to the mailing list ¯\_(ツ)_/¯
> >
> > Anyway, that's just another indication that it might not be the right fit
> > for inclusion into the QEMU source tree. So either download it similar to
> > (or included in) the avocado tests, or maybe another solution would be to
> > have a separate "qemu-ci" or "qemu-testing" repository for stuff like this ... ?
> 
> Yes, I think we should have a separate repo for all testing related
> stuff like blobs, guest images etc. Curently, some of them are in
> personal github accounts which is not idea IMHo.
> 
> Bits stuff can reside in the same place.

FYI, the reason much of this is intentionally NOT under the /qemu-project
gitlab namespace is that we did not want to be responsible for distributing
arbitrary binary blobs/images. That in turn makes the QEMU project responsible
for license compliance, which is non-trivial todo correctly for much of this
stuff. As such it is highly desirable to delegate both the hosting the
binaries and source to the third party who builds it.

I agree the use of personal github accounts is not nice, but it was the
least worst solution identified.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote:
> FYI, the reason much of this is intentionally NOT under the /qemu-project
> gitlab namespace is that we did not want to be responsible for distributing
> arbitrary binary blobs/images. That in turn makes the QEMU project responsible
> for license compliance, which is non-trivial todo correctly for much of this
> stuff. As such it is highly desirable to delegate both the hosting the
> binaries and source to the third party who builds it.

This might be understadable for random guest OS images which include tons of stuff
and are thus hard to audit.  But not to biosbits which has its own
license (more or less bsd) + gpl for grub:
https://github.com/biosbits/bits/blob/master/COPYING

> I agree the use of personal github accounts is not nice, but it was the
> least worst solution identified.


-- 
MST

Re: Why we should avoid new submodules if possible

[Ani Sinha]

On Tue, Jun 28, 2022 at 11:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote:
> > FYI, the reason much of this is intentionally NOT under the /qemu-project
> > gitlab namespace is that we did not want to be responsible for distributing
> > arbitrary binary blobs/images. That in turn makes the QEMU project responsible
> > for license compliance, which is non-trivial todo correctly for much of this
> > stuff. As such it is highly desirable to delegate both the hosting the
> > binaries and source to the third party who builds it.
>
> This might be understadable for random guest OS images which include tons of stuff
> and are thus hard to audit.  But not to biosbits which has its own
> license (more or less bsd) + gpl for grub:
> https://github.com/biosbits/bits/blob/master/COPYING

These are all the dependencies:
https://github.com/biosbits/bits/tree/master/deps

We can go through the licenses for each and make a determination. The
audit would be lost easier because there is a bounded number of
dependencies for bits.

>
> > I agree the use of personal github accounts is not nice, but it was the
> > least worst solution identified.
>
>
> --
> MST
>

Re: Why we should avoid new submodules if possible

[Thomas Huth]

On 29/06/2022 08.28, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 11:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote:
>>> FYI, the reason much of this is intentionally NOT under the /qemu-project
>>> gitlab namespace is that we did not want to be responsible for distributing
>>> arbitrary binary blobs/images. That in turn makes the QEMU project responsible
>>> for license compliance, which is non-trivial todo correctly for much of this
>>> stuff. As such it is highly desirable to delegate both the hosting the
>>> binaries and source to the third party who builds it.
>>
>> This might be understadable for random guest OS images which include tons of stuff
>> and are thus hard to audit.  But not to biosbits which has its own
>> license (more or less bsd) + gpl for grub:
>> https://github.com/biosbits/bits/blob/master/COPYING
> 
> These are all the dependencies:
> https://github.com/biosbits/bits/tree/master/deps
> 
> We can go through the licenses for each and make a determination. The
> audit would be lost easier because there is a bounded number of
> dependencies for bits.

That's quite a bit of dependencies already ... I don't think that we should 
put the burden of keeping up with the licenses of those projects to the QEMU 
project. So just make sure that the binaries are available somewhere and 
then include your test into the avocado framework or download via curl/wget 
as suggested in other mails.

  Thomas

Re: Why we should avoid new submodules if possible

[Philippe Mathieu-Daudé via]

On Fri, Jul 1, 2022 at 5:37 AM Thomas Huth <thuth@redhat.com> wrote:
> On 29/06/2022 08.28, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 11:30 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >> On Tue, Jun 28, 2022 at 05:15:05PM +0100, Daniel P. Berrangé wrote:
> >>> FYI, the reason much of this is intentionally NOT under the /qemu-project
> >>> gitlab namespace is that we did not want to be responsible for distributing
> >>> arbitrary binary blobs/images. That in turn makes the QEMU project responsible
> >>> for license compliance, which is non-trivial todo correctly for much of this
> >>> stuff. As such it is highly desirable to delegate both the hosting the
> >>> binaries and source to the third party who builds it.
> >>
> >> This might be understadable for random guest OS images which include tons of stuff
> >> and are thus hard to audit.  But not to biosbits which has its own
> >> license (more or less bsd) + gpl for grub:
> >> https://github.com/biosbits/bits/blob/master/COPYING
> >
> > These are all the dependencies:
> > https://github.com/biosbits/bits/tree/master/deps
> >
> > We can go through the licenses for each and make a determination. The
> > audit would be lost easier because there is a bounded number of
> > dependencies for bits.
>
> That's quite a bit of dependencies already ... I don't think that we should
> put the burden of keeping up with the licenses of those projects to the QEMU
> project. So just make sure that the binaries are available somewhere and
> then include your test into the avocado framework or download via curl/wget
> as suggested in other mails.

"available somewhere" until they get moved or deleted...

This happened a few times with archives/artifacts Avocado tests fetch.

We ended up committing one or a set of URLs pointing to a file, its
usual name and its hash.
If you downloaded the file in time and don't flush the Avocado cache,
you can keep testing.
If it is too late (the file disappeared) and you have time to dig for
such a file online, and are lucky because you found it and its hash
matches, then you can run some tests. As a maintainer I find this
feature useful, and find it worse to lose the ability to test
something that used to work.

My 2¢...

Phil.

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> [...]
> > For biosbits if we are going this route then I feel a submodule is much
> > better.  It records which version exactly each qemu version wants.
> 
> As far as I know, you can also specify the version when using pip, can't
> you? So that's not really an advantage here.
> 
> On the contrary, submodules have a couple of disadvantages that I really
> dislike:
> 
> - submodules do not get updated automatically when doing a "git checkout",
> we have to update them via a script instead. This causes e.g. trouble if you
> rsync your source tree to a machine that has no access to the internet and
> you forgot to update the submodule before the sync
> 
> - the content of submodules is not added to the tarballs that get created on
> the git forges automatically. There were lots of requests from users in the
> past that tried to download a tarball from github and then wondered why they
> couldn't compile QEMU.
> 
> - we include the submodule content in our release tarballs, so people get
> the impression that hte submodule content is part of the QEMU sources. This
> has two disadvantages:
>  * We already got bug reports for the code in the submodule,
>    where people did not understand that they should report that
>    rather to the original project instead (i.e. you ship it - you
>    own it)
>  * People get the impression that QEMU is a huge monster
>    application if they count the number of code lines, run
>    their code scanner tools on the tarball contents, etc.
>    Remember "nemu", for example, where one of the main complaints
>    was that QEMU has too many lines of code?
> 
> - If programs includes code via submodules, this gets a higher
>   burder for distro maintainers, since they have to patch each
>   and every package when there is a bug, instead of being able to
>   fix it in one central place.
> 
> So in my opinion we should avoid new submodules if there is an alternative.
> 
>  Thomas

So looking at the latest proposals downloading files from CI,
checksumming them etc etc. No auto checkout, not added automatically
either, right?

This seems to be the only difference:
- we include the submodule content in our release tarballs

How about we just fix that? Thomas would that address your
concern at least wrt tests?


-- 
MST

Re: Why we should avoid new submodules if possible

[Thomas Huth]

On 28/09/2022 11.26, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
>> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
>> [...]
>>> For biosbits if we are going this route then I feel a submodule is much
>>> better.  It records which version exactly each qemu version wants.
>>
>> As far as I know, you can also specify the version when using pip, can't
>> you? So that's not really an advantage here.
>>
>> On the contrary, submodules have a couple of disadvantages that I really
>> dislike:
>>
>> - submodules do not get updated automatically when doing a "git checkout",
>> we have to update them via a script instead. This causes e.g. trouble if you
>> rsync your source tree to a machine that has no access to the internet and
>> you forgot to update the submodule before the sync
>>
>> - the content of submodules is not added to the tarballs that get created on
>> the git forges automatically. There were lots of requests from users in the
>> past that tried to download a tarball from github and then wondered why they
>> couldn't compile QEMU.
>>
>> - we include the submodule content in our release tarballs, so people get
>> the impression that hte submodule content is part of the QEMU sources. This
>> has two disadvantages:
>>   * We already got bug reports for the code in the submodule,
>>     where people did not understand that they should report that
>>     rather to the original project instead (i.e. you ship it - you
>>     own it)
>>   * People get the impression that QEMU is a huge monster
>>     application if they count the number of code lines, run
>>     their code scanner tools on the tarball contents, etc.
>>     Remember "nemu", for example, where one of the main complaints
>>     was that QEMU has too many lines of code?
>>
>> - If programs includes code via submodules, this gets a higher
>>    burder for distro maintainers, since they have to patch each
>>    and every package when there is a bug, instead of being able to
>>    fix it in one central place.
>>
>> So in my opinion we should avoid new submodules if there is an alternative.
>>
>>   Thomas
> 
> So looking at the latest proposals downloading files from CI,
> checksumming them etc etc. No auto checkout, not added automatically
> either, right?
> 
> This seems to be the only difference:
> - we include the submodule content in our release tarballs
> 
> How about we just fix that? Thomas would that address your
> concern at least wrt tests?

If I'm not forced to checkout that submodule, and if we don't add it to the 
release tarball, I guess there's not much left I can complain about ;-)

  Thomas

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Wed, Sep 28, 2022 at 11:33:52AM +0200, Thomas Huth wrote:
> On 28/09/2022 11.26, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > [...]
> > > > For biosbits if we are going this route then I feel a submodule is much
> > > > better.  It records which version exactly each qemu version wants.
> > > 
> > > As far as I know, you can also specify the version when using pip, can't
> > > you? So that's not really an advantage here.
> > > 
> > > On the contrary, submodules have a couple of disadvantages that I really
> > > dislike:
> > > 
> > > - submodules do not get updated automatically when doing a "git checkout",
> > > we have to update them via a script instead. This causes e.g. trouble if you
> > > rsync your source tree to a machine that has no access to the internet and
> > > you forgot to update the submodule before the sync
> > > 
> > > - the content of submodules is not added to the tarballs that get created on
> > > the git forges automatically. There were lots of requests from users in the
> > > past that tried to download a tarball from github and then wondered why they
> > > couldn't compile QEMU.
> > > 
> > > - we include the submodule content in our release tarballs, so people get
> > > the impression that hte submodule content is part of the QEMU sources. This
> > > has two disadvantages:
> > >   * We already got bug reports for the code in the submodule,
> > >     where people did not understand that they should report that
> > >     rather to the original project instead (i.e. you ship it - you
> > >     own it)
> > >   * People get the impression that QEMU is a huge monster
> > >     application if they count the number of code lines, run
> > >     their code scanner tools on the tarball contents, etc.
> > >     Remember "nemu", for example, where one of the main complaints
> > >     was that QEMU has too many lines of code?
> > > 
> > > - If programs includes code via submodules, this gets a higher
> > >    burder for distro maintainers, since they have to patch each
> > >    and every package when there is a bug, instead of being able to
> > >    fix it in one central place.
> > > 
> > > So in my opinion we should avoid new submodules if there is an alternative.
> > > 
> > >   Thomas
> > 
> > So looking at the latest proposals downloading files from CI,
> > checksumming them etc etc. No auto checkout, not added automatically
> > either, right?
> > 
> > This seems to be the only difference:
> > - we include the submodule content in our release tarballs
> > 
> > How about we just fix that? Thomas would that address your
> > concern at least wrt tests?
> 
> If I'm not forced to checkout that submodule,

I think the make check script can do that?

> and if we don't add it to the
> release tarball, I guess there's not much left I can complain about ;-)
> 
>  Thomas

Re: Why we should avoid new submodules if possible

[Thomas Huth]

On 28/09/2022 11.47, Michael S. Tsirkin wrote:
> On Wed, Sep 28, 2022 at 11:33:52AM +0200, Thomas Huth wrote:
>> On 28/09/2022 11.26, Michael S. Tsirkin wrote:
>>> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
>>>> On 28/06/2022 12.03, Michael S. Tsirkin wrote:
>>>> [...]
>>>>> For biosbits if we are going this route then I feel a submodule is much
>>>>> better.  It records which version exactly each qemu version wants.
>>>>
>>>> As far as I know, you can also specify the version when using pip, can't
>>>> you? So that's not really an advantage here.
>>>>
>>>> On the contrary, submodules have a couple of disadvantages that I really
>>>> dislike:
>>>>
>>>> - submodules do not get updated automatically when doing a "git checkout",
>>>> we have to update them via a script instead. This causes e.g. trouble if you
>>>> rsync your source tree to a machine that has no access to the internet and
>>>> you forgot to update the submodule before the sync
>>>>
>>>> - the content of submodules is not added to the tarballs that get created on
>>>> the git forges automatically. There were lots of requests from users in the
>>>> past that tried to download a tarball from github and then wondered why they
>>>> couldn't compile QEMU.
>>>>
>>>> - we include the submodule content in our release tarballs, so people get
>>>> the impression that hte submodule content is part of the QEMU sources. This
>>>> has two disadvantages:
>>>>    * We already got bug reports for the code in the submodule,
>>>>      where people did not understand that they should report that
>>>>      rather to the original project instead (i.e. you ship it - you
>>>>      own it)
>>>>    * People get the impression that QEMU is a huge monster
>>>>      application if they count the number of code lines, run
>>>>      their code scanner tools on the tarball contents, etc.
>>>>      Remember "nemu", for example, where one of the main complaints
>>>>      was that QEMU has too many lines of code?
>>>>
>>>> - If programs includes code via submodules, this gets a higher
>>>>     burder for distro maintainers, since they have to patch each
>>>>     and every package when there is a bug, instead of being able to
>>>>     fix it in one central place.
>>>>
>>>> So in my opinion we should avoid new submodules if there is an alternative.
>>>>
>>>>    Thomas
>>>
>>> So looking at the latest proposals downloading files from CI,
>>> checksumming them etc etc. No auto checkout, not added automatically
>>> either, right?
>>>
>>> This seems to be the only difference:
>>> - we include the submodule content in our release tarballs
>>>
>>> How about we just fix that? Thomas would that address your
>>> concern at least wrt tests?
>>
>> If I'm not forced to checkout that submodule,
> 
> I think the make check script can do that?

Yes, currently there are only some submodules that are always checked out: 
dtc, meson, tests/fp/berkeley-softfloat-3, tests/fp/berkeley-testfloat-3 and 
ui/keycodemapdb.
And for example the ones in the roms/ directory are not checked out by default.

  Thomas

Re: Why we should avoid new submodules if possible

[Daniel P. Berrangé]

On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > [...]
> > > For biosbits if we are going this route then I feel a submodule is much
> > > better.  It records which version exactly each qemu version wants.
> > 
> > As far as I know, you can also specify the version when using pip, can't
> > you? So that's not really an advantage here.
> > 
> > On the contrary, submodules have a couple of disadvantages that I really
> > dislike:
> > 
> > - submodules do not get updated automatically when doing a "git checkout",
> > we have to update them via a script instead. This causes e.g. trouble if you
> > rsync your source tree to a machine that has no access to the internet and
> > you forgot to update the submodule before the sync
> > 
> > - the content of submodules is not added to the tarballs that get created on
> > the git forges automatically. There were lots of requests from users in the
> > past that tried to download a tarball from github and then wondered why they
> > couldn't compile QEMU.
> > 
> > - we include the submodule content in our release tarballs, so people get
> > the impression that hte submodule content is part of the QEMU sources. This
> > has two disadvantages:
> >  * We already got bug reports for the code in the submodule,
> >    where people did not understand that they should report that
> >    rather to the original project instead (i.e. you ship it - you
> >    own it)
> >  * People get the impression that QEMU is a huge monster
> >    application if they count the number of code lines, run
> >    their code scanner tools on the tarball contents, etc.
> >    Remember "nemu", for example, where one of the main complaints
> >    was that QEMU has too many lines of code?
> > 
> > - If programs includes code via submodules, this gets a higher
> >   burder for distro maintainers, since they have to patch each
> >   and every package when there is a bug, instead of being able to
> >   fix it in one central place.
> > 
> > So in my opinion we should avoid new submodules if there is an alternative.
> > 
> >  Thomas
> 
> So looking at the latest proposals downloading files from CI,
> checksumming them etc etc. No auto checkout, not added automatically
> either, right?
> 
> This seems to be the only difference:
> - we include the submodule content in our release tarballs

That's just one of the issues with submodules. Working with them in general
is not a pleasant experiance. Thomas pointed out some of the issues, such
as 'git checkout' ignoring submodules, requiring extra steps to sync them.
The flipside on tarballs is that the auto-generated tarballs from gitlab
do not include submodules, so the tests will be missing the files they
wanted. There's also the perenial problem that developers frequently send
patches that mistakenly include submodule changes, which is related to the
way that 'git checkout' doesn't sync submodule state when switching branches.
I'd really like to see us doing more to eliminate as much use of submodules
as is possible over time. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > [...]
> > > > For biosbits if we are going this route then I feel a submodule is much
> > > > better.  It records which version exactly each qemu version wants.
> > > 
> > > As far as I know, you can also specify the version when using pip, can't
> > > you? So that's not really an advantage here.
> > > 
> > > On the contrary, submodules have a couple of disadvantages that I really
> > > dislike:
> > > 
> > > - submodules do not get updated automatically when doing a "git checkout",
> > > we have to update them via a script instead. This causes e.g. trouble if you
> > > rsync your source tree to a machine that has no access to the internet and
> > > you forgot to update the submodule before the sync
> > > 
> > > - the content of submodules is not added to the tarballs that get created on
> > > the git forges automatically. There were lots of requests from users in the
> > > past that tried to download a tarball from github and then wondered why they
> > > couldn't compile QEMU.
> > > 
> > > - we include the submodule content in our release tarballs, so people get
> > > the impression that hte submodule content is part of the QEMU sources. This
> > > has two disadvantages:
> > >  * We already got bug reports for the code in the submodule,
> > >    where people did not understand that they should report that
> > >    rather to the original project instead (i.e. you ship it - you
> > >    own it)
> > >  * People get the impression that QEMU is a huge monster
> > >    application if they count the number of code lines, run
> > >    their code scanner tools on the tarball contents, etc.
> > >    Remember "nemu", for example, where one of the main complaints
> > >    was that QEMU has too many lines of code?
> > > 
> > > - If programs includes code via submodules, this gets a higher
> > >   burder for distro maintainers, since they have to patch each
> > >   and every package when there is a bug, instead of being able to
> > >   fix it in one central place.
> > > 
> > > So in my opinion we should avoid new submodules if there is an alternative.
> > > 
> > >  Thomas
> > 
> > So looking at the latest proposals downloading files from CI,
> > checksumming them etc etc. No auto checkout, not added automatically
> > either, right?
> > 
> > This seems to be the only difference:
> > - we include the submodule content in our release tarballs
> 
> That's just one of the issues with submodules. Working with them in general
> is not a pleasant experiance.

This is what I asked about at the maintainers summit.
I'd like to map the issues and see if there's anything
we can do to solve them. In particular we will likely
keep using submodules where we historically did
so it's time well spent.

I agree generally but the big question is what to replace these with.
Below I assume the replacement is a script such as avocado or pytest
with its own hashing, calling wget internally etc etc.


> Thomas pointed out some of the issues, such
> as 'git checkout' ignoring submodules, requiring extra steps to sync them.


Not different from a home grown SCM as part of test script, right?

> The flipside on tarballs is that the auto-generated tarballs from gitlab
> do not include submodules, so the tests will be missing the files they
> wanted.

Not different from a home grown SCM as part of test script, right?

> There's also the perenial problem that developers frequently send
> patches that mistakenly include submodule changes,

OK, so the thing to do would be to look for ways to exclude submodule changes
from git commits.

> which is related to the
> way that 'git checkout' doesn't sync submodule state when switching branches.

Not different from a home grown SCM as part of test script, right?

> I'd really like to see us doing more to eliminate as much use of submodules
> as is possible over time. 

Or try to fix the problems, right?


> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why we should avoid new submodules if possible

[Daniel P. Berrangé]

On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote:
> On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > > [...]
> > > > > For biosbits if we are going this route then I feel a submodule is much
> > > > > better.  It records which version exactly each qemu version wants.
> > > > 
> > > > As far as I know, you can also specify the version when using pip, can't
> > > > you? So that's not really an advantage here.
> > > > 
> > > > On the contrary, submodules have a couple of disadvantages that I really
> > > > dislike:
> > > > 
> > > > - submodules do not get updated automatically when doing a "git checkout",
> > > > we have to update them via a script instead. This causes e.g. trouble if you
> > > > rsync your source tree to a machine that has no access to the internet and
> > > > you forgot to update the submodule before the sync
> > > > 
> > > > - the content of submodules is not added to the tarballs that get created on
> > > > the git forges automatically. There were lots of requests from users in the
> > > > past that tried to download a tarball from github and then wondered why they
> > > > couldn't compile QEMU.
> > > > 
> > > > - we include the submodule content in our release tarballs, so people get
> > > > the impression that hte submodule content is part of the QEMU sources. This
> > > > has two disadvantages:
> > > >  * We already got bug reports for the code in the submodule,
> > > >    where people did not understand that they should report that
> > > >    rather to the original project instead (i.e. you ship it - you
> > > >    own it)
> > > >  * People get the impression that QEMU is a huge monster
> > > >    application if they count the number of code lines, run
> > > >    their code scanner tools on the tarball contents, etc.
> > > >    Remember "nemu", for example, where one of the main complaints
> > > >    was that QEMU has too many lines of code?
> > > > 
> > > > - If programs includes code via submodules, this gets a higher
> > > >   burder for distro maintainers, since they have to patch each
> > > >   and every package when there is a bug, instead of being able to
> > > >   fix it in one central place.
> > > > 
> > > > So in my opinion we should avoid new submodules if there is an alternative.
> > > > 
> > > >  Thomas
> > > 
> > > So looking at the latest proposals downloading files from CI,
> > > checksumming them etc etc. No auto checkout, not added automatically
> > > either, right?
> > > 
> > > This seems to be the only difference:
> > > - we include the submodule content in our release tarballs
> > 
> > That's just one of the issues with submodules. Working with them in general
> > is not a pleasant experiance.
> 
> This is what I asked about at the maintainers summit.
> I'd like to map the issues and see if there's anything
> we can do to solve them. In particular we will likely
> keep using submodules where we historically did
> so it's time well spent.
> 
> I agree generally but the big question is what to replace these with.
> Below I assume the replacement is a script such as avocado or pytest
> with its own hashing, calling wget internally etc etc.
> 
> 
> > Thomas pointed out some of the issues, such
> > as 'git checkout' ignoring submodules, requiring extra steps to sync them.
> 
> 
> Not different from a home grown SCM as part of test script, right?

We're not building a home grown SCM as part of a test script, so
this answer is irrelevant.

> > There's also the perenial problem that developers frequently send
> > patches that mistakenly include submodule changes,
> 
> OK, so the thing to do would be to look for ways to exclude submodule changes
> from git commits.

If someone wants to make git suck less with submodules great, but needs
someone to actually do the work.

> > I'd really like to see us doing more to eliminate as much use of submodules
> > as is possible over time. 
> 
> Or try to fix the problems, right?

Again needs someone to actually make it happen.

Meanwhile  QEMU already has an integrated test harness in the form
of Avocado that does everything needed. If Avocado had just been
used for this biosbits test in the first place, the test would
likely have already been merged to QEMU, instead of us having this
never ending debate on how to re-invent an alternative to what
already avocado does.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Wed, Sep 28, 2022 at 10:57:47AM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote:
> > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > > > [...]
> > > > > > For biosbits if we are going this route then I feel a submodule is much
> > > > > > better.  It records which version exactly each qemu version wants.
> > > > > 
> > > > > As far as I know, you can also specify the version when using pip, can't
> > > > > you? So that's not really an advantage here.
> > > > > 
> > > > > On the contrary, submodules have a couple of disadvantages that I really
> > > > > dislike:
> > > > > 
> > > > > - submodules do not get updated automatically when doing a "git checkout",
> > > > > we have to update them via a script instead. This causes e.g. trouble if you
> > > > > rsync your source tree to a machine that has no access to the internet and
> > > > > you forgot to update the submodule before the sync
> > > > > 
> > > > > - the content of submodules is not added to the tarballs that get created on
> > > > > the git forges automatically. There were lots of requests from users in the
> > > > > past that tried to download a tarball from github and then wondered why they
> > > > > couldn't compile QEMU.
> > > > > 
> > > > > - we include the submodule content in our release tarballs, so people get
> > > > > the impression that hte submodule content is part of the QEMU sources. This
> > > > > has two disadvantages:
> > > > >  * We already got bug reports for the code in the submodule,
> > > > >    where people did not understand that they should report that
> > > > >    rather to the original project instead (i.e. you ship it - you
> > > > >    own it)
> > > > >  * People get the impression that QEMU is a huge monster
> > > > >    application if they count the number of code lines, run
> > > > >    their code scanner tools on the tarball contents, etc.
> > > > >    Remember "nemu", for example, where one of the main complaints
> > > > >    was that QEMU has too many lines of code?
> > > > > 
> > > > > - If programs includes code via submodules, this gets a higher
> > > > >   burder for distro maintainers, since they have to patch each
> > > > >   and every package when there is a bug, instead of being able to
> > > > >   fix it in one central place.
> > > > > 
> > > > > So in my opinion we should avoid new submodules if there is an alternative.
> > > > > 
> > > > >  Thomas
> > > > 
> > > > So looking at the latest proposals downloading files from CI,
> > > > checksumming them etc etc. No auto checkout, not added automatically
> > > > either, right?
> > > > 
> > > > This seems to be the only difference:
> > > > - we include the submodule content in our release tarballs
> > > 
> > > That's just one of the issues with submodules. Working with them in general
> > > is not a pleasant experiance.
> > 
> > This is what I asked about at the maintainers summit.
> > I'd like to map the issues and see if there's anything
> > we can do to solve them. In particular we will likely
> > keep using submodules where we historically did
> > so it's time well spent.
> > 
> > I agree generally but the big question is what to replace these with.
> > Below I assume the replacement is a script such as avocado or pytest
> > with its own hashing, calling wget internally etc etc.
> > 
> > 
> > > Thomas pointed out some of the issues, such
> > > as 'git checkout' ignoring submodules, requiring extra steps to sync them.
> > 
> > 
> > Not different from a home grown SCM as part of test script, right?
> 
> We're not building a home grown SCM as part of a test script, so
> this answer is irrelevant.

If you are managing contents of files using code then you have
a content managing system :)

> > > There's also the perenial problem that developers frequently send
> > > patches that mistakenly include submodule changes,
> > 
> > OK, so the thing to do would be to look for ways to exclude submodule changes
> > from git commits.
> 
> If someone wants to make git suck less with submodules great, but needs
> someone to actually do the work.

I have some experience with git, might be able to if I can figure out
what the issues are. Or maybe there's already a flag to do what we want
git is pretty configurable.

> > > I'd really like to see us doing more to eliminate as much use of submodules
> > > as is possible over time. 
> > 
> > Or try to fix the problems, right?
> 
> Again needs someone to actually make it happen.
> 
> Meanwhile  QEMU already has an integrated test harness in the form
> of Avocado that does everything needed. If Avocado had just been
> used for this biosbits test in the first place, the test would
> likely have already been merged to QEMU, instead of us having this
> never ending debate on how to re-invent an alternative to what
> already avocado does.
> 
> With regards,
> Daniel

I just think there's a ton of difference between linux guest images
which we want to keep running indefinitely and a unit test firmware
which we want to version with qemu.


> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why we should avoid new submodules if possible

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 6086 bytes --]

On Wed, Sep 28, 2022 at 7:09 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote:
> > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
> > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
> > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
> > > > > [...]
> > > > > > For biosbits if we are going this route then I feel a submodule
> is much
> > > > > > better.  It records which version exactly each qemu version
> wants.
> > > > >
> > > > > As far as I know, you can also specify the version when using pip,
> can't
> > > > > you? So that's not really an advantage here.
> > > > >
> > > > > On the contrary, submodules have a couple of disadvantages that I
> really
> > > > > dislike:
> > > > >
> > > > > - submodules do not get updated automatically when doing a "git
> checkout",
> > > > > we have to update them via a script instead. This causes e.g.
> trouble if you
> > > > > rsync your source tree to a machine that has no access to the
> internet and
> > > > > you forgot to update the submodule before the sync
> > > > >
> > > > > - the content of submodules is not added to the tarballs that get
> created on
> > > > > the git forges automatically. There were lots of requests from
> users in the
> > > > > past that tried to download a tarball from github and then
> wondered why they
> > > > > couldn't compile QEMU.
> > > > >
> > > > > - we include the submodule content in our release tarballs, so
> people get
> > > > > the impression that hte submodule content is part of the QEMU
> sources. This
> > > > > has two disadvantages:
> > > > >  * We already got bug reports for the code in the submodule,
> > > > >    where people did not understand that they should report that
> > > > >    rather to the original project instead (i.e. you ship it - you
> > > > >    own it)
> > > > >  * People get the impression that QEMU is a huge monster
> > > > >    application if they count the number of code lines, run
> > > > >    their code scanner tools on the tarball contents, etc.
> > > > >    Remember "nemu", for example, where one of the main complaints
> > > > >    was that QEMU has too many lines of code?
> > > > >
> > > > > - If programs includes code via submodules, this gets a higher
> > > > >   burder for distro maintainers, since they have to patch each
> > > > >   and every package when there is a bug, instead of being able to
> > > > >   fix it in one central place.
> > > > >
> > > > > So in my opinion we should avoid new submodules if there is an
> alternative.
> > > > >
> > > > >  Thomas
> > > >
> > > > So looking at the latest proposals downloading files from CI,
> > > > checksumming them etc etc. No auto checkout, not added automatically
> > > > either, right?
> > > >
> > > > This seems to be the only difference:
> > > > - we include the submodule content in our release tarballs
> > >
> > > That's just one of the issues with submodules. Working with them in
> general
> > > is not a pleasant experiance.
> >
> > This is what I asked about at the maintainers summit.
> > I'd like to map the issues and see if there's anything
> > we can do to solve them. In particular we will likely
> > keep using submodules where we historically did
> > so it's time well spent.
> >
> > I agree generally but the big question is what to replace these with.
> > Below I assume the replacement is a script such as avocado or pytest
> > with its own hashing, calling wget internally etc etc.
> >
> >
> > > Thomas pointed out some of the issues, such
> > > as 'git checkout' ignoring submodules, requiring extra steps to sync
> them.
> >
> >
> > Not different from a home grown SCM as part of test script, right?
>
> We're not building a home grown SCM as part of a test script, so
> this answer is irrelevant.
>
> > > There's also the perenial problem that developers frequently send
> > > patches that mistakenly include submodule changes,
> >
> > OK, so the thing to do would be to look for ways to exclude submodule
> changes
> > from git commits.
>
> If someone wants to make git suck less with submodules great, but needs
> someone to actually do the work.
>

A big part of the problem is knowing which of the following commands I have
to
do to undo the uncommitted changes, the committed changes, the staged
changes,
etc:

git submodule update --init --recursive
git submodule update --init
git submodule foreach --recursive git reset --hard
git submodule foreach --recursive git clean -xfd
git submodule foreach --recursive git clean -xfg

(all of these are in my history, I honestly don't know the difference
between the last two).
And each 'oops' takes time away from upstreaming bsd-user I don't really
have that
much of. I've wasted hours on this over the past year between all the
different ways
it can screw up.

To be fair, it is a relatively small fraction of the time, but as you can
tell from the
animation in my email it inspires much passion.

Warner


> > > I'd really like to see us doing more to eliminate as much use of
> submodules
> > > as is possible over time.p
> >
> > Or try to fix the problems, right?
>
> Again needs someone to actually make it happen.
>
> Meanwhile  QEMU already has an integrated test harness in the form
> of Avocado that does everything needed. If Avocado had just been
> used for this biosbits test in the first place, the test would
> likely have already been merged to QEMU, instead of us having this
> never ending debate on how to re-invent an alternative to what
> already avocado does.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>
>

[-- Attachment #2: Type: text/html, Size: 8233 bytes --]

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Wed, Sep 28, 2022 at 07:15:53AM -0600, Warner Losh wrote:
> 
> 
> On Wed, Sep 28, 2022 at 7:09 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
>     On Wed, Sep 28, 2022 at 05:53:17AM -0400, Michael S. Tsirkin wrote:
>     > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
>     > > On Wed, Sep 28, 2022 at 05:26:42AM -0400, Michael S. Tsirkin wrote:
>     > > > On Tue, Jun 28, 2022 at 12:21:39PM +0200, Thomas Huth wrote:
>     > > > > On 28/06/2022 12.03, Michael S. Tsirkin wrote:
>     > > > > [...]
>     > > > > > For biosbits if we are going this route then I feel a submodule
>     is much
>     > > > > > better.  It records which version exactly each qemu version
>     wants.
>     > > > >
>     > > > > As far as I know, you can also specify the version when using pip,
>     can't
>     > > > > you? So that's not really an advantage here.
>     > > > >
>     > > > > On the contrary, submodules have a couple of disadvantages that I
>     really
>     > > > > dislike:
>     > > > >
>     > > > > - submodules do not get updated automatically when doing a "git
>     checkout",
>     > > > > we have to update them via a script instead. This causes e.g.
>     trouble if you
>     > > > > rsync your source tree to a machine that has no access to the
>     internet and
>     > > > > you forgot to update the submodule before the sync
>     > > > >
>     > > > > - the content of submodules is not added to the tarballs that get
>     created on
>     > > > > the git forges automatically. There were lots of requests from
>     users in the
>     > > > > past that tried to download a tarball from github and then wondered
>     why they
>     > > > > couldn't compile QEMU.
>     > > > >
>     > > > > - we include the submodule content in our release tarballs, so
>     people get
>     > > > > the impression that hte submodule content is part of the QEMU
>     sources. This
>     > > > > has two disadvantages:
>     > > > >  * We already got bug reports for the code in the submodule,
>     > > > >    where people did not understand that they should report that
>     > > > >    rather to the original project instead (i.e. you ship it - you
>     > > > >    own it)
>     > > > >  * People get the impression that QEMU is a huge monster
>     > > > >    application if they count the number of code lines, run
>     > > > >    their code scanner tools on the tarball contents, etc.
>     > > > >    Remember "nemu", for example, where one of the main complaints
>     > > > >    was that QEMU has too many lines of code?
>     > > > >
>     > > > > - If programs includes code via submodules, this gets a higher
>     > > > >   burder for distro maintainers, since they have to patch each
>     > > > >   and every package when there is a bug, instead of being able to
>     > > > >   fix it in one central place.
>     > > > >
>     > > > > So in my opinion we should avoid new submodules if there is an
>     alternative.
>     > > > >
>     > > > >  Thomas
>     > > >
>     > > > So looking at the latest proposals downloading files from CI,
>     > > > checksumming them etc etc. No auto checkout, not added automatically
>     > > > either, right?
>     > > >
>     > > > This seems to be the only difference:
>     > > > - we include the submodule content in our release tarballs
>     > >
>     > > That's just one of the issues with submodules. Working with them in
>     general
>     > > is not a pleasant experiance.
>     >
>     > This is what I asked about at the maintainers summit.
>     > I'd like to map the issues and see if there's anything
>     > we can do to solve them. In particular we will likely
>     > keep using submodules where we historically did
>     > so it's time well spent.
>     >
>     > I agree generally but the big question is what to replace these with.
>     > Below I assume the replacement is a script such as avocado or pytest
>     > with its own hashing, calling wget internally etc etc.
>     >
>     >
>     > > Thomas pointed out some of the issues, such
>     > > as 'git checkout' ignoring submodules, requiring extra steps to sync
>     them.
>     >
>     >
>     > Not different from a home grown SCM as part of test script, right?
> 
>     We're not building a home grown SCM as part of a test script, so
>     this answer is irrelevant.
> 
>     > > There's also the perenial problem that developers frequently send
>     > > patches that mistakenly include submodule changes,
>     >
>     > OK, so the thing to do would be to look for ways to exclude submodule
>     changes
>     > from git commits.
> 
>     If someone wants to make git suck less with submodules great, but needs
>     someone to actually do the work.
> 
> 
> A big part of the problem is knowing which of the following commands I have to
> do to undo the uncommitted changes, the committed changes, the staged changes,
> etc:
> 
> git submodule update --init --recursive
> git submodule update --init
> git submodule foreach --recursive git reset --hard
> git submodule foreach --recursive git clean -xfd
> git submodule foreach --recursive git clean -xfg
> 
> (all of these are in my history, I honestly don't know the difference between
> the last two).
> And each 'oops' takes time away from upstreaming bsd-user I don't really have
> that
> much of. I've wasted hours on this over the past year between all the different
> ways
> it can screw up.
> 
> To be fair, it is a relatively small fraction of the time, but as you can tell
> from the
> animation in my email it inspires much passion.
> 
> Warner
>  

OK this is understandable.

I think for what we do in qemu (never change upstream really)
	git submodule update --checkout testsub
	git update-index --skip-worktree testsub
is more or less what most people want.

This assumes you have a script that basically does
	git submodule deinit XYZ
	git submodule update --init XYZ
	git update-index --skip-worktree XYZ
each time, and do not need to work on the submodule proper.




>     > > I'd really like to see us doing more to eliminate as much use of
>     submodules
>     > > as is possible over time.p
>     >
>     > Or try to fix the problems, right?
> 
>     Again needs someone to actually make it happen.
> 
>     Meanwhile  QEMU already has an integrated test harness in the form
>     of Avocado that does everything needed. If Avocado had just been
>     used for this biosbits test in the first place, the test would
>     likely have already been merged to QEMU, instead of us having this
>     never ending debate on how to re-invent an alternative to what
>     already avocado does.
> 
>     With regards,
>     Daniel
>     --
>     |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange
>     :|
>     |: https://libvirt.org         -o-            https://fstop138.berrange.com
>     :|
>     |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange
>     :|
> 
> 
> 

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> There's also the perenial problem that developers frequently send
> patches that mistakenly include submodule changes, which is related to the
> way that 'git checkout' doesn't sync submodule state when switching branches.

Do you happen to know how exactly that happens?

-- 
MST

Re: Why we should avoid new submodules if possible

[Daniel P. Berrangé]

On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote:
> On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > There's also the perenial problem that developers frequently send
> > patches that mistakenly include submodule changes, which is related to the
> > way that 'git checkout' doesn't sync submodule state when switching branches.
> 
> Do you happen to know how exactly that happens?

For any given branch the submodule is synced to a given git commit hash.
If the submodule checkout is not synced to the same commit hash it will
show as dirty, and if you git add this pending change, it'll record that
new submodule commit hash.  Seeing dirty state is common when you switch
between branches, either side of a git master change that updated a
submodule.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Wed, Sep 28, 2022 at 11:18:18AM +0100, Daniel P. Berrangé wrote:
> On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote:
> > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > There's also the perenial problem that developers frequently send
> > > patches that mistakenly include submodule changes, which is related to the
> > > way that 'git checkout' doesn't sync submodule state when switching branches.
> > 
> > Do you happen to know how exactly that happens?
> 
> For any given branch the submodule is synced to a given git commit hash.
> If the submodule checkout is not synced to the same commit hash it will
> show as dirty, and if you git add this pending change, it'll record that
> new submodule commit hash.  Seeing dirty state is common when you switch
> between branches, either side of a git master change that updated a
> submodule.
> 
> With regards,
> Daniel

I see. It is interesting.

So apparently what you want is ignore submodule
changes, right? If yes this is close to what we want:

git submodule update --checkout testsub
git update-index --skip-worktree testsub

A script checking out the submodule can just run this
command.



> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why we should avoid new submodules if possible

[Peter Maydell]

On Wed, 28 Sept 2022 at 15:29, Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Sep 28, 2022 at 11:18:18AM +0100, Daniel P. Berrangé wrote:
> > On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote:
> > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > > There's also the perenial problem that developers frequently send
> > > > patches that mistakenly include submodule changes, which is related to the
> > > > way that 'git checkout' doesn't sync submodule state when switching branches.
> > >
> > > Do you happen to know how exactly that happens?
> >
> > For any given branch the submodule is synced to a given git commit hash.
> > If the submodule checkout is not synced to the same commit hash it will
> > show as dirty, and if you git add this pending change, it'll record that
> > new submodule commit hash.  Seeing dirty state is common when you switch
> > between branches, either side of a git master change that updated a
> > submodule.
> I see. It is interesting.
>
> So apparently what you want is ignore submodule
> changes, right? If yes this is close to what we want:
>
> git submodule update --checkout testsub
> git update-index --skip-worktree testsub
>
> A script checking out the submodule can just run this
> command.

The problem happens not when you check out the submodule, but when
you do basic not-submodule-related git operations like
"git checkout my-working-branch". The fix would be if git itself
automatically kept submodule state in sync with the main repo's
branch. But it doesn't, so the UI is a massive beartrap.

-- PMM

Re: Why we should avoid new submodules if possible

[Michael S. Tsirkin]

On Wed, Sep 28, 2022 at 04:07:40PM +0100, Peter Maydell wrote:
> On Wed, 28 Sept 2022 at 15:29, Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Sep 28, 2022 at 11:18:18AM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Sep 28, 2022 at 06:13:45AM -0400, Michael S. Tsirkin wrote:
> > > > On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > > > > There's also the perenial problem that developers frequently send
> > > > > patches that mistakenly include submodule changes, which is related to the
> > > > > way that 'git checkout' doesn't sync submodule state when switching branches.
> > > >
> > > > Do you happen to know how exactly that happens?
> > >
> > > For any given branch the submodule is synced to a given git commit hash.
> > > If the submodule checkout is not synced to the same commit hash it will
> > > show as dirty, and if you git add this pending change, it'll record that
> > > new submodule commit hash.  Seeing dirty state is common when you switch
> > > between branches, either side of a git master change that updated a
> > > submodule.
> > I see. It is interesting.
> >
> > So apparently what you want is ignore submodule
> > changes, right? If yes this is close to what we want:
> >
> > git submodule update --checkout testsub
> > git update-index --skip-worktree testsub
> >
> > A script checking out the submodule can just run this
> > command.
> 
> The problem happens not when you check out the submodule, but when
> you do basic not-submodule-related git operations like
> "git checkout my-working-branch". The fix would be if git itself
> automatically kept submodule state in sync with the main repo's
> branch. But it doesn't, so the UI is a massive beartrap.
> 
> -- PMM

If this is what you want, you can make it do that too:

$ cat .gitmodules 
[submodule "testsub"]
        path = testsub
        recurse = true
        url = /tmp/testsub/


git checkout (and a bunch of other commands) will now recurse into this
submodule.


-- 
MST

Re: Why we should avoid new submodules if possible

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1604 bytes --]

On Wed, Sep 28, 2022 at 6:44 AM Michael S. Tsirkin <mst@redhat.com> wrote:

> On Wed, Sep 28, 2022 at 10:37:14AM +0100, Daniel P. Berrangé wrote:
> > There's also the perenial problem that developers frequently send
> > patches that mistakenly include submodule changes, which is related to
> the
> > way that 'git checkout' doesn't sync submodule state when switching
> branches.
>
> Do you happen to know how exactly that happens?
>

Here's how it happens to me.

% git checkout master
% git checkout -b bsd-user-2022q4
# bring changes in, doesn't matter
# Build a few times
# time passes, master upstream evolves, submodules move
% git checkout master
% git pull # master fast forwards
% git rebase -i master bsd-user-2022q4

at this point I have to do unnatural git submodule recursive things to make
it stop
complaining about how all the submodules are now 'modified' because they
have
moved on the master branch.

It is real, lasting, lingering pain and a large part of the reason that I
refuse to allow FreeBSD
to use submodules[*]. It's just too much extra hassle for no gain at all
(at least for me, I
see only the pain and no benefits to me or anybody else). I've sent at
least three review
requests that have this pollution in it since it's hard to avoid if you
press forward after the
rebase not to have something swept up into an innocuous change.

Warner

[*] They are also generally a poor match for FreeBSD since we rarely import
things
verbatim, unchanged from upstreams that have well-oiled CI to give us
assurance
that things are good...

[-- Attachment #2: Type: text/html, Size: 2232 bytes --]

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 02:05:19PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 1:58 PM Thomas Huth <thuth@redhat.com> wrote:
> >
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > >> On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >>>
> > >>> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > >>>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > >>>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > >>>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
> > >>>>>>>>>>>> packages from the net during testing?
> > >>>>>>>>>>>
> > >>>>>>>>>>> We do that too. See requirements.txt in tests/
> > >>>>>>>>>>> Following two are downloaded:
> > >>>>>>>>>>> avocado-framework==88.1
> > >>>>>>>>>>> pycdlib==1.11.0
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also see this line in Makefie.include:
> > >>>>>>>>>>>
> > >>>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > >>>>>>>>>>
> > >>>>>>>>>> Right but that's avocado since it pulls lots of stuff from
> > >>>>>>>>>> the net anyway.
> > >>>>>>>>>> Are the libraries in question not packaged on major distros?
> > >>>>>>>>>
> > >>>>>>>>> Currently I only need this:
> > >>>>>>>>> https://github.com/python-tap/tappy
> > >>>>>>>>> which is the basic TAP processing library for python.
> > >>>>>>>>>
> > >>>>>>>>> It seems its only installed through pip:
> > >>>>>>>>> https://tappy.readthedocs.io/en/latest/
> > >>>>>>>>>
> > >>>>>>>>> I do not think this is packaged by default. It's such a basic library
> > >>>>>>>>> for parsing test output that maybe we can keep this somewhere within
> > >>>>>>>>> the python src tree? Not sure ...
> > >>>>>>>>
> > >>>>>>>> It's pretty small for sure. Another submodule?
> > >>>>>>>
> > >>>>>>> Unlike BITS, this one is likely going to be maintained for a while and
> > >>>>>>> will receive new releases through
> > >>>>>>> https://pypi.org/project/tap.py/
> > >>>>>>> so forking is OK but someone has to keep this updated.
> > >>>>>>>
> > >>>>>>> I am open to anything. Whatever feels right is fine to me.
> > >>>>>>
> > >>>>>> John Snow is currently working on the "Pythonification" of various QEMU
> > >>>>>> bits, I think you should loop him into this discussion, too.
> > >>>>>>
> > >>>>>>    Thomas
> > >>>>>
> > >>>>> submodule does not mean we fork necessarily. We could have
> > >>>>> all options: check for the module and use it if there, if not
> > >>>>> use one from system if not there install with pip ..
> > >>>>> But yea, I'm not sure what's best either.
> > >>>>
> > >>>> submodules create a dependency on an internet connection, too. So before you
> > >>>> add yet another submodule (which have a couple of other disadvantages), I
> > >>>> think you could also directly use the venv here.
> > >>>
> > >>> Definitely not submodules.
> > >>>
> > >>> We need to get out of the mindset that submodules are needed for every new
> > >>> dependancy we add. Submodules are only appropriate if the external project
> > >>> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > >>> need to bundle in order to prevent a regression for previously deployed
> > >>> QEMU installs where the dependancy is known not to exist on all our
> > >>> supported platforms.
> > >>>
> > >>> This does not apply in this case, because the proposed use of tappy is
> > >>> merely for a test case. Meson just needs to check if tappy exists and if
> > >>> it does, then use it, otherwise skip the tests that need it. The user can
> > >>> arrange to install tappy, as they do with the majority of other deps.
> > >>>
> > >>> If John's venv stuff is relevant, then we don't even need the meson checks,
> > >>> just delegate to the venv setup.
> > >>>
> > >>> Regardless, no submodules are needed or desirable.
> > >>
> > >> What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > even an ancient version of python as a submodule.
> > >
> > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > too, all with dubious license compliance. I don't think this looks like
> > > something we should have in qemu.git or qemu tarballs. It will also
> > > make it challenging for the distro to take biosbits at all, unless
> > > those 3rd party bundles can be eliminated in favour of using existing
> > > builds their have packaged for grub, python, libffi, etc.
> >
> > So if this depends on some third party binary bits, I think this is pretty
> > similar to the tests in the avocado directory ... there we download third
> > party binaries, too... Wouldn't it make sense to adapt your tests to that
> > framework?
> 
> I do not want to bring in the whole avocado framework because it would
> unnecessarily make things complicated. I just need the qemu machine
> python library and that is enough. For downloading third party stuff,
> we can simply wget things from somewhere.

Well avocado is already a part of QEMU's testing framework, so it is
not bringing in a whole new framework. Arguably the introduction of
python usage in qtest is what's introducing a new maint burden here.

If it is actually booting a real guest image (from biosbits) and interacting
with it, then it does feel like the scope of this testing is more appropriate
to QEMU's avocado framework than qtest, especially given the desire to use
python for it all.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> If it is actually booting a real guest image (from biosbits) and interacting
> with it, then it does feel like the scope of this testing is more appropriate
> to QEMU's avocado framework than qtest, especially given the desire to use
> python for it all.
> 
> With regards,
> Daniel

I feel avocado is directed towards booting full fledged guest OS.
It makes it much easier to figure out guest issues but it also
prone to false positives and is harder to debug as a result.
Booting a minimal image like this shouldn't require that.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > If it is actually booting a real guest image (from biosbits) and interacting
> > with it, then it does feel like the scope of this testing is more appropriate
> > to QEMU's avocado framework than qtest, especially given the desire to use
> > python for it all.
> > 
> > With regards,
> > Daniel
> 
> I feel avocado is directed towards booting full fledged guest OS.

That's essentially what this is doing - its a custom guest OS rather
than a common distro IIUC

> It makes it much easier to figure out guest issues but it also
> prone to false positives and is harder to debug as a result.
> Booting a minimal image like this shouldn't require that.

Well avocado is as reliable as the tests you write for it. The problems
are largely around the images being used in avocado. If the biosbits
testing system is reliable, then avocado will be too, and if they not
reliable, then it will affect qtest too.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:18:14AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > If it is actually booting a real guest image (from biosbits) and interacting
> > > with it, then it does feel like the scope of this testing is more appropriate
> > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > python for it all.
> > > 
> > > With regards,
> > > Daniel
> > 
> > I feel avocado is directed towards booting full fledged guest OS.
> 
> That's essentially what this is doing - its a custom guest OS rather
> than a common distro IIUC

yes but then so is bios tables test - we generate the disk on the fly.

> > It makes it much easier to figure out guest issues but it also
> > prone to false positives and is harder to debug as a result.
> > Booting a minimal image like this shouldn't require that.
> 
> Well avocado is as reliable as the tests you write for it. The problems
> are largely around the images being used in avocado.

The idea of downloading large images as part of the test as opposed to
part of setup is a large part of what makes it flaky - due to
running into unpredictable latency and errors on both the internet and
local IO side of things.

> If the biosbits
> testing system is reliable, then avocado will be too, and if they not
> reliable, then it will affect qtest too.

A larger system has more potential for bugs ;) I'm with Ani here. Using
a system that is overkill will just make debugging more painful than it
needs to be.

> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 3:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 06:07:13AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > If it is actually booting a real guest image (from biosbits) and interacting
> > > with it, then it does feel like the scope of this testing is more appropriate
> > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > python for it all.
> > >
> > > With regards,
> > > Daniel
> >
> > I feel avocado is directed towards booting full fledged guest OS.
>
> That's essentially what this is doing - its a custom guest OS rather

Daniel, there is no OS :-) Like there is no kernel image, no
persistent real file system (there is a fake one called pyfs), no
packages, nothing. Its very minimalistic image narrowly targeted for
one thing alone - to test the bios.

> than a common distro IIUC
>
> > It makes it much easier to figure out guest issues but it also
> > prone to false positives and is harder to debug as a result.
> > Booting a minimal image like this shouldn't require that.
>
> Well avocado is as reliable as the tests you write for it. The problems
> are largely around the images being used in avocado. If the biosbits
> testing system is reliable, then avocado will be too, and if they not
> reliable, then it will affect qtest too.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > If it is actually booting a real guest image (from biosbits) and interacting
> > with it, then it does feel like the scope of this testing is more appropriate
> > to QEMU's avocado framework than qtest, especially given the desire to use
> > python for it all.
> >
> > With regards,
> > Daniel
>
> I feel avocado is directed towards booting full fledged guest OS.
> It makes it much easier to figure out guest issues but it also
> prone to false positives and is harder to debug as a result.
> Booting a minimal image like this shouldn't require that.

Yes 100% agree with Michael on this. Biobits is *not* booting any OS
image. It runs off grub, that is, directly from bootloader stage. The
interraction with the VM is minimal.

>
> --
> MST
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > If it is actually booting a real guest image (from biosbits) and interacting
> > > with it, then it does feel like the scope of this testing is more appropriate
> > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > python for it all.
> > >
> > > With regards,
> > > Daniel
> >
> > I feel avocado is directed towards booting full fledged guest OS.
> > It makes it much easier to figure out guest issues but it also
> > prone to false positives and is harder to debug as a result.
> > Booting a minimal image like this shouldn't require that.
> 
> Yes 100% agree with Michael on this. Biobits is *not* booting any OS
> image. It runs off grub, that is, directly from bootloader stage. The
> interraction with the VM is minimal.

Just because it doesn't run a whole Linux kernel, doesn't make it
not a guest OS image. It is merely unsual in that it can do everything
it needs from grub stage, because it is just poking low level BIOS
stuff and doesn't need a full OS like Linux on top. This is still
functional integration testing IMHO and relevant to avocado in QEMU.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:42:37AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > > If it is actually booting a real guest image (from biosbits) and interacting
> > > > with it, then it does feel like the scope of this testing is more appropriate
> > > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > > python for it all.
> > > >
> > > > With regards,
> > > > Daniel
> > >
> > > I feel avocado is directed towards booting full fledged guest OS.
> > > It makes it much easier to figure out guest issues but it also
> > > prone to false positives and is harder to debug as a result.
> > > Booting a minimal image like this shouldn't require that.
> > 
> > Yes 100% agree with Michael on this. Biobits is *not* booting any OS
> > image. It runs off grub, that is, directly from bootloader stage. The
> > interraction with the VM is minimal.
> 
> Just because it doesn't run a whole Linux kernel, doesn't make it
> not a guest OS image. It is merely unsual in that it can do everything
> it needs from grub stage, because it is just poking low level BIOS
> stuff and doesn't need a full OS like Linux on top. This is still
> functional integration testing IMHO and relevant to avocado in QEMU.
> 
> With regards,
> Daniel

I wonder why it needs grub BTW. Let's just load all of disk into ram
and be done with it?


> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:42:37AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 03:58:21PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 3:37 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 28, 2022 at 11:04:30AM +0100, Daniel P. Berrangé wrote:
> > > > If it is actually booting a real guest image (from biosbits) and interacting
> > > > with it, then it does feel like the scope of this testing is more appropriate
> > > > to QEMU's avocado framework than qtest, especially given the desire to use
> > > > python for it all.
> > > >
> > > > With regards,
> > > > Daniel
> > >
> > > I feel avocado is directed towards booting full fledged guest OS.
> > > It makes it much easier to figure out guest issues but it also
> > > prone to false positives and is harder to debug as a result.
> > > Booting a minimal image like this shouldn't require that.
> > 
> > Yes 100% agree with Michael on this. Biobits is *not* booting any OS
> > image. It runs off grub, that is, directly from bootloader stage. The
> > interraction with the VM is minimal.
> 
> Just because it doesn't run a whole Linux kernel, doesn't make it
> not a guest OS image. It is merely unsual in that it can do everything
> it needs from grub stage, because it is just poking low level BIOS
> stuff and doesn't need a full OS like Linux on top. This is still
> functional integration testing IMHO and relevant to avocado in QEMU.
> 
> With regards,
> Daniel

I think the main difference is not even in how it works, it's
in what it does. Which is check that ACPI tables are sane.
Who cares about that? Well developers do when they change the
tables. Users really don't because for users we have the expected
tables in tree and we check against these.


-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Peter Maydell]

On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> I think the main difference is not even in how it works, it's
> in what it does. Which is check that ACPI tables are sane.
> Who cares about that? Well developers do when they change the
> tables. Users really don't because for users we have the expected
> tables in tree and we check against these.

It wants to build and run a big guest binary blob -- that to me is
the main difference. Users don't much care about any of our tests,
whether they're under 'make check' or 'make check-avocado' or the
iotests framework. The reason to pick one framework or another
is mostly I think whether the properties of the test are such
that one framework works better. Avocado is (for better or worse)
the one we have for dealing with "actually run a guest machine
with a big lump of guest code in it".

-- PMM

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> > I think the main difference is not even in how it works, it's
> > in what it does. Which is check that ACPI tables are sane.
> > Who cares about that? Well developers do when they change the
> > tables. Users really don't because for users we have the expected
> > tables in tree and we check against these.
>
> It wants to build and run a big guest binary blob -- that to me is
> the main difference. Users don't much care about any of our tests,

perhaps but we do enforce patch submitters to make sure make check
passes before submitting patches. make check-avocado is not run as
part of make check, requires considerable disk space to download all
guest images and hence generally not run by patch submitters. Making
bits parts of avocado tests almost defeats the purpose of having this
test at all.

> whether they're under 'make check' or 'make check-avocado' or the
> iotests framework. The reason to pick one framework or another
> is mostly I think whether the properties of the test are such
> that one framework works better. Avocado is (for better or worse)
> the one we have for dealing with "actually run a guest machine
> with a big lump of guest code in it".
>
> -- PMM

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 14.36, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
>>> I think the main difference is not even in how it works, it's
>>> in what it does. Which is check that ACPI tables are sane.
>>> Who cares about that? Well developers do when they change the
>>> tables. Users really don't because for users we have the expected
>>> tables in tree and we check against these.
>>
>> It wants to build and run a big guest binary blob -- that to me is
>> the main difference. Users don't much care about any of our tests,
> 
> perhaps but we do enforce patch submitters to make sure make check
> passes before submitting patches. make check-avocado is not run as
> part of make check, requires considerable disk space to download all
> guest images and hence generally not run by patch submitters. Making
> bits parts of avocado tests almost defeats the purpose of having this
> test at all.
You can define sane subsets of the avocado tests by using tags. For example, 
as I'm one of the s390x maintainers, I'm also only running the tests that 
are tagged with tags=arch:s390x there. You could simply introduce an "acpi" 
tag for your tests, too.

  Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > I think the main difference is not even in how it works, it's
> > > in what it does. Which is check that ACPI tables are sane.
> > > Who cares about that? Well developers do when they change the
> > > tables. Users really don't because for users we have the expected
> > > tables in tree and we check against these.
> >
> > It wants to build and run a big guest binary blob -- that to me is
> > the main difference. Users don't much care about any of our tests,
> 
> perhaps but we do enforce patch submitters to make sure make check
> passes before submitting patches. make check-avocado is not run as
> part of make check, requires considerable disk space to download all
> guest images and hence generally not run by patch submitters. Making
> bits parts of avocado tests almost defeats the purpose of having this
> test at all.

This proposed biosbits test also involves a considerable download.
The test is said to be irrelevant for anyone except those working
on a fairly narrow set of QEMU firmware related bits. So by the same
rationale we shouldn't impose that burden on everyone working on
QEMU by having it in qtest. Making it entirely optional, only
downloaded by avocado on demand, for the people who need to run
the tests is best.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > I think the main difference is not even in how it works, it's
> > > > in what it does. Which is check that ACPI tables are sane.
> > > > Who cares about that? Well developers do when they change the
> > > > tables. Users really don't because for users we have the expected
> > > > tables in tree and we check against these.
> > >
> > > It wants to build and run a big guest binary blob -- that to me is
> > > the main difference. Users don't much care about any of our tests,
> >
> > perhaps but we do enforce patch submitters to make sure make check
> > passes before submitting patches. make check-avocado is not run as
> > part of make check, requires considerable disk space to download all
> > guest images and hence generally not run by patch submitters. Making
> > bits parts of avocado tests almost defeats the purpose of having this
> > test at all.
>
> This proposed biosbits test also involves a considerable download.

I do not think 50 MB is "considerable" . Last time I tried to run
avocado tests, my laptop ran out of disk space!

> The test is said to be irrelevant for anyone except those working
> on a fairly narrow set of QEMU firmware related bits.

Well ok that is just a bad argument. You can say the same thing for
most qtests. In fact, that is why most qtetes can run directly simply
by passing QTEST_QEMU_BINARY in the environment. No need to go through
make check. Same with the bits test. It can be run directly.

So by the same
> rationale we shouldn't impose that burden on everyone working on
> QEMU by having it in qtest.

So why burden everyone by having bios-tables-test when it only affects
acpi/smbios developers?

Making it entirely optional, only
> downloaded by avocado on demand, for the people who need to run
> the tests is best.
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Peter Maydell]

On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca> wrote:
> On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > This proposed biosbits test also involves a considerable download.
>
> I do not think 50 MB is "considerable" . Last time I tried to run
> avocado tests, my laptop ran out of disk space!

I think 50MB is pretty big. It might be smaller than some other
avocado tests, but it's not exactly the "no binary involved"
that most qtests are.

> > The test is said to be irrelevant for anyone except those working
> > on a fairly narrow set of QEMU firmware related bits.
>
> Well ok that is just a bad argument. You can say the same thing for
> most qtests. In fact, that is why most qtetes can run directly simply
> by passing QTEST_QEMU_BINARY in the environment. No need to go through
> make check. Same with the bits test. It can be run directly.

'make check' is generally the small, fast, no-binary-blobs tests.
Very few 'make check' tests even run code in the guest.

> So by the same
> > rationale we shouldn't impose that burden on everyone working on
> > QEMU by having it in qtest.
>
> So why burden everyone by having bios-tables-test when it only affects
> acpi/smbios developers?

Because it's small and fast and doesn't have a binary blob to download.

There are definitely some awkwardnesses with 'check-avocado',
but we should work on fixing those, not use them as a reason
to refuse to put tests into the avocado tests if that's where
they fit best.

-- PMM

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

[-- Attachment #1: Type: text/plain, Size: 2311 bytes --]

On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca> wrote:
> > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> > > This proposed biosbits test also involves a considerable download.
> >
> > I do not think 50 MB is "considerable" . Last time I tried to run
> > avocado tests, my laptop ran out of disk space!
>
> I think 50MB is pretty big. It might be smaller than some other
> avocado tests, but it's not exactly the "no binary involved"
> that most qtests are.


Well bios-tables-test uses the binary blobs of the acpi tables. Only
difference is that in this case, we could maintain them within  the qemu
tree. In this case the blob in slightly larger and comes from a third
party. That is the difference.



>
> > > The test is said to be irrelevant for anyone except those working
> > > on a fairly narrow set of QEMU firmware related bits.
> >
> > Well ok that is just a bad argument. You can say the same thing for
> > most qtests. In fact, that is why most qtetes can run directly simply
> > by passing QTEST_QEMU_BINARY in the environment. No need to go through
> > make check. Same with the bits test. It can be run directly.
>
> 'make check' is generally the small, fast, no-binary-blobs tests.


See above.


> Very few 'make check' tests even run code in the guest.


So bits test is similar here. It runs code in the guest vm.


>
> > So by the same
> > > rationale we shouldn't impose that burden on everyone working on
> > > QEMU by having it in qtest.
> >
> > So why burden everyone by having bios-tables-test when it only affects
> > acpi/smbios developers?
>
> Because it's small and fast and doesn't have a binary blob to download.
>
> There are definitely some awkwardnesses with 'check-avocado',
> but we should work on fixing those, not use them as a reason
> to refuse to put tests into the avocado tests if that's where
> they fit best.


I think this test fits best in the qtrst not with the integration test
framework. Very few path developers will ever run it and wrote new tests
for it if we have it there. I would be terribly discouraged if that’s where
this test landed.


>
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 3691 bytes --]

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Peter Maydell]

On Tue, 28 Jun 2022 at 14:53, Ani Sinha <ani@anisinha.ca> wrote:
>
>
>
> On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca> wrote:
>> > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>> > > This proposed biosbits test also involves a considerable download.
>> >
>> > I do not think 50 MB is "considerable" . Last time I tried to run
>> > avocado tests, my laptop ran out of disk space!
>>
>> I think 50MB is pretty big. It might be smaller than some other
>> avocado tests, but it's not exactly the "no binary involved"
>> that most qtests are.
>
>
> Well bios-tables-test uses the binary blobs of the acpi tables. Only difference is that in this case, we could maintain them within  the qemu tree. In this case the blob in slightly larger and comes from a third party. That is the difference.

Yes. That is exactly the difference that means it should go in
the avocado tests.

-- PMM

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 15.53, Ani Sinha wrote:
> 
> 
> On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.maydell@linaro.org 
> <mailto:peter.maydell@linaro.org>> wrote:
> 
>     On Tue, 28 Jun 2022 at 14:23, Ani Sinha <ani@anisinha.ca
>     <mailto:ani@anisinha.ca>> wrote:
>      > On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé
>     <berrange@redhat.com <mailto:berrange@redhat.com>> wrote:
>      > > This proposed biosbits test also involves a considerable download.
>      >
>      > I do not think 50 MB is "considerable" . Last time I tried to run
>      > avocado tests, my laptop ran out of disk space!
> 
>     I think 50MB is pretty big. It might be smaller than some other
>     avocado tests, but it's not exactly the "no binary involved"
>     that most qtests are.
> 
> 
> Well bios-tables-test uses the binary blobs of the acpi tables. Only 
> difference is that in this case, we could maintain them within  the qemu 
> tree. In this case the blob in slightly larger and comes from a third party. 
> That is the difference.

"slightly larger" ... it apparently contains already a complete grub and 
python 2.7 interpreter ... I'd consider that as *much* larger than the ~ 2k 
loc bios-tables-test ;-)

>     Very few 'make check' tests even run code in the guest.
> 
> So bits test is similar here. It runs code in the guest vm.

The qtests that run some TCG code only use some few lines of code. The tests 
that run third party firmware images are the avocado tests.

>     There are definitely some awkwardnesses with 'check-avocado',
>     but we should work on fixing those, not use them as a reason
>     to refuse to put tests into the avocado tests if that's where
>     they fit best.
> 
> I think this test fits best in the qtrst not with the integration test 
> framework.

I disagree. Third party binary blob tests are certainly nothing for the 
qtest directory.

> Very few path developers will ever run it and wrote new tests for 
> it if we have it there. I would be terribly discouraged if that’s where this 
> test landed.

I see your point - I'm also hardly ever running whole "make check-avocado" 
since it's too slow/fat/broken. But using this as a reason to stick your 
test into the qtest framework also does not work/scale: Being one of the 
s390x maintainers - What if I now also want my s390x related tests to be 
executed by everybody? The kernel + initrd images from 
tests/avocado/machine_s390_ccw_virtio.py are also not that big, ca. 50 MiB 
each. Should I now move that out of the avocade directory, too, and force 
everybody to consume it via a submodule? Then who's next? QEMU has 21 target 
architectures, if every maintainers adds a 50 MiB test to the tree, we're at 
more than a gigabyte of data already that you'd have to download before you 
were able to run "make check". This simply does not scale.

So the avocado framework is currently still the best place that we have for 
such tests. You just have to get used to consume it via "tags" instead of 
running the whole "make check-avocado" suite. For example, being a s390x 
maintainer, I'm only running:

  make check-venv
  ./tests/venv/bin/avocado run -t arch:s390x tests/avocado/

and that finishes within some few minutes and only downloads some few 
hundreds of megabytes. You could do something similar with acpi. You could 
even wrap it in a "make check-avocado-acpi" target if you don't like to 
execute avocado directly.

I even wouldn't mind if you put your python stuff in a new directory like 
tests/pytests/ for example, as long as it downloads your binaries separately 
- as I wrote in another mail, the avocado framework rather looks like an 
oddball in our test framework nowadays since it uses a separate test runner 
and not the meson test harness, so having a new approach for python-based 
tests is maybe even a good idea. I just really want to avoid that this goes 
into tests/qtest (since it really does not belong there), and please don't 
add more external stuff via git submodules, that's really the wrong approach 
for this.

  Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> I even wouldn't mind if you put your python stuff in a new directory like
> tests/pytests/ for example, as long as it downloads your binaries separately
> - as I wrote in another mail, the avocado framework rather looks like an
> oddball in our test framework nowadays since it uses a separate test runner
> and not the meson test harness, so having a new approach for python-based
> tests is maybe even a good idea. I just really want to avoid that this goes
> into tests/qtest (since it really does not belong there), and please don't
> add more external stuff via git submodules, that's really the wrong approach
> for this.

I get it, people hate submodules with passion.  I think trying another
approach for testing that is neither avocado nor qtest is
not too bad. As long as this is not user visible, we can
allow ourselves space to experiment.

OK so, how about this:
- put it in a new directory: tests/roms?
- create repo for a fork of biosbits under git.qemu.org
- roll our own analog to git submodules: a script
  that clones the repo
- new target make check-roms, if the clone exists locally -
  run the test, if not - skip it
- as for using pre-generates ISOs as an optimization,
  I'm not sure how important that is, if yes -
  we can add another repo and another make target along the
  same lines



-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > I even wouldn't mind if you put your python stuff in a new directory like
> > tests/pytests/ for example, as long as it downloads your binaries separately
> > - as I wrote in another mail, the avocado framework rather looks like an
> > oddball in our test framework nowadays since it uses a separate test runner
> > and not the meson test harness, so having a new approach for python-based
> > tests is maybe even a good idea. I just really want to avoid that this goes
> > into tests/qtest (since it really does not belong there), and please don't
> > add more external stuff via git submodules, that's really the wrong approach
> > for this.
>
> I get it, people hate submodules with passion.  I think trying another
> approach for testing that is neither avocado nor qtest is
> not too bad. As long as this is not user visible, we can
> allow ourselves space to experiment.
>
> OK so, how about this:
> - put it in a new directory: tests/roms?
> - create repo for a fork of biosbits under git.qemu.org
> - roll our own analog to git submodules: a script
>   that clones the repo

No need to clone the whole repo. We can simply download the binaries
that the girlab CI job would generate from the bits sources in that
repo.
We need to clone if we are always building bits from source for every
test. That is not necessary IMHO since much of the bits package would
remain as is without modification.

> - new target make check-roms,

I think make pytest or some such is better and more generic if other
such tests in other areas follow suit.

if the clone exists locally -
>   run the test, if not - skip it

if download of the bits binaries fail, skip it.

> - as for using pre-generates ISOs as an optimization,
>   I'm not sure how important that is, if yes -
>   we can add another repo and another make target along the
>   same lines
>
>
>
> --
> MST
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > I even wouldn't mind if you put your python stuff in a new directory like
> > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > - as I wrote in another mail, the avocado framework rather looks like an
> > > oddball in our test framework nowadays since it uses a separate test runner
> > > and not the meson test harness, so having a new approach for python-based
> > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > into tests/qtest (since it really does not belong there), and please don't
> > > add more external stuff via git submodules, that's really the wrong approach
> > > for this.
> >
> > I get it, people hate submodules with passion.  I think trying another
> > approach for testing that is neither avocado nor qtest is
> > not too bad. As long as this is not user visible, we can
> > allow ourselves space to experiment.
> >
> > OK so, how about this:
> > - put it in a new directory: tests/roms?
> > - create repo for a fork of biosbits under git.qemu.org
> > - roll our own analog to git submodules: a script
> >   that clones the repo
> 
> No need to clone the whole repo. We can simply download the binaries
> that the girlab CI job would generate from the bits sources in that
> repo.
> We need to clone if we are always building bits from source for every
> test. That is not necessary IMHO since much of the bits package would
> remain as is without modification.

IMHO CI job idea isn't great since isn't versioned at all, is it?
Also as long as test passes, sure. But if it fails one will
need the sources to investigate.

Let's start with building things from source. Add an option
of prebuilt binaries as an optimization once things
stabilize.


> > - new target make check-roms,
> 
> I think make pytest or some such is better and more generic if other
> such tests in other areas follow suit.

The name is not critical in my mind, but I think we need to decide
what exactly differentiates it from other tests.


> 
> if the clone exists locally -
> >   run the test, if not - skip it
> 
> if download of the bits binaries fail, skip it.

You seem to be recreating either git or avocado or both here.

Personally I want something that works offline.



> > - as for using pre-generates ISOs as an optimization,
> >   I'm not sure how important that is, if yes -
> >   we can add another repo and another make target along the
> >   same lines
> >
> >
> >
> > --
> > MST
> >

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > I even wouldn't mind if you put your python stuff in a new directory like
> > > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > oddball in our test framework nowadays since it uses a separate test runner
> > > > and not the meson test harness, so having a new approach for python-based
> > > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > > into tests/qtest (since it really does not belong there), and please don't
> > > > add more external stuff via git submodules, that's really the wrong approach
> > > > for this.
> > >
> > > I get it, people hate submodules with passion.  I think trying another
> > > approach for testing that is neither avocado nor qtest is
> > > not too bad. As long as this is not user visible, we can
> > > allow ourselves space to experiment.
> > >
> > > OK so, how about this:
> > > - put it in a new directory: tests/roms?
> > > - create repo for a fork of biosbits under git.qemu.org
> > > - roll our own analog to git submodules: a script
> > >   that clones the repo
> >
> > No need to clone the whole repo. We can simply download the binaries
> > that the girlab CI job would generate from the bits sources in that
> > repo.
> > We need to clone if we are always building bits from source for every
> > test. That is not necessary IMHO since much of the bits package would
> > remain as is without modification.
>
> IMHO CI job idea isn't great since isn't versioned at all, is it?

bits is versioned yes, in a crude way. every time you make a commit in
the top level repo, the version would increment by one.

> Also as long as test passes, sure. But if it fails one will
> need the sources to investigate.

sources might also be needed to write the tests.

>
> Let's start with building things from source.

hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
build depending on parallelity and build host.

Add an option
> of prebuilt binaries as an optimization once things
> stabilize.
>
>
> > > - new target make check-roms,
> >
> > I think make pytest or some such is better and more generic if other
> > such tests in other areas follow suit.
>
> The name is not critical in my mind, but I think we need to decide
> what exactly differentiates it from other tests.
>
>
> >
> > if the clone exists locally -
> > >   run the test, if not - skip it
> >
> > if download of the bits binaries fail, skip it.
>
> You seem to be recreating either git or avocado or both here.
>
> Personally I want something that works offline.
>
>
>
> > > - as for using pre-generates ISOs as an optimization,
> > >   I'm not sure how important that is, if yes -
> > >   we can add another repo and another make target along the
> > >   same lines
> > >
> > >
> > >
> > > --
> > > MST
> > >
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Fri, Jul 01, 2022 at 01:20:30PM +0530, Ani Sinha wrote:
> On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > > I even wouldn't mind if you put your python stuff in a new directory like
> > > > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > > oddball in our test framework nowadays since it uses a separate test runner
> > > > > and not the meson test harness, so having a new approach for python-based
> > > > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > > > into tests/qtest (since it really does not belong there), and please don't
> > > > > add more external stuff via git submodules, that's really the wrong approach
> > > > > for this.
> > > >
> > > > I get it, people hate submodules with passion.  I think trying another
> > > > approach for testing that is neither avocado nor qtest is
> > > > not too bad. As long as this is not user visible, we can
> > > > allow ourselves space to experiment.
> > > >
> > > > OK so, how about this:
> > > > - put it in a new directory: tests/roms?
> > > > - create repo for a fork of biosbits under git.qemu.org
> > > > - roll our own analog to git submodules: a script
> > > >   that clones the repo
> > >
> > > No need to clone the whole repo. We can simply download the binaries
> > > that the girlab CI job would generate from the bits sources in that
> > > repo.
> > > We need to clone if we are always building bits from source for every
> > > test. That is not necessary IMHO since much of the bits package would
> > > remain as is without modification.
> >
> > IMHO CI job idea isn't great since isn't versioned at all, is it?
> 
> bits is versioned yes, in a crude way. every time you make a commit in
> the top level repo, the version would increment by one.

Is it easy to find out which source was this generated from?
And is there a promise to keep these around indefinitely?

> > Also as long as test passes, sure. But if it fails one will
> > need the sources to investigate.
> 
> sources might also be needed to write the tests.
> 
> >
> > Let's start with building things from source.
> 
> hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
> build depending on parallelity and build host.

Right. But whoever wants to use these just needs to do it once.


> Add an option
> > of prebuilt binaries as an optimization once things
> > stabilize.
> >
> >
> > > > - new target make check-roms,
> > >
> > > I think make pytest or some such is better and more generic if other
> > > such tests in other areas follow suit.
> >
> > The name is not critical in my mind, but I think we need to decide
> > what exactly differentiates it from other tests.
> >
> >
> > >
> > > if the clone exists locally -
> > > >   run the test, if not - skip it
> > >
> > > if download of the bits binaries fail, skip it.
> >
> > You seem to be recreating either git or avocado or both here.
> >
> > Personally I want something that works offline.
> >
> >
> >
> > > > - as for using pre-generates ISOs as an optimization,
> > > >   I'm not sure how important that is, if yes -
> > > >   we can add another repo and another make target along the
> > > >   same lines
> > > >
> > > >
> > > >
> > > > --
> > > > MST
> > > >
> >

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 3:12 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 01:20:30PM +0530, Ani Sinha wrote:
> > On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > > > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > > > I even wouldn't mind if you put your python stuff in a new directory like
> > > > > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > > > oddball in our test framework nowadays since it uses a separate test runner
> > > > > > and not the meson test harness, so having a new approach for python-based
> > > > > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > > > > into tests/qtest (since it really does not belong there), and please don't
> > > > > > add more external stuff via git submodules, that's really the wrong approach
> > > > > > for this.
> > > > >
> > > > > I get it, people hate submodules with passion.  I think trying another
> > > > > approach for testing that is neither avocado nor qtest is
> > > > > not too bad. As long as this is not user visible, we can
> > > > > allow ourselves space to experiment.
> > > > >
> > > > > OK so, how about this:
> > > > > - put it in a new directory: tests/roms?
> > > > > - create repo for a fork of biosbits under git.qemu.org
> > > > > - roll our own analog to git submodules: a script
> > > > >   that clones the repo
> > > >
> > > > No need to clone the whole repo. We can simply download the binaries
> > > > that the girlab CI job would generate from the bits sources in that
> > > > repo.
> > > > We need to clone if we are always building bits from source for every
> > > > test. That is not necessary IMHO since much of the bits package would
> > > > remain as is without modification.
> > >
> > > IMHO CI job idea isn't great since isn't versioned at all, is it?
> >
> > bits is versioned yes, in a crude way. every time you make a commit in
> > the top level repo, the version would increment by one.
>
> Is it easy to find out which source was this generated from?

yes because this is how the version number is generated:

expr 2000 + `git rev-list HEAD 2>/dev/null | wc -l`

That means we simply count the number of commits in the top level
directory. If we overwrote history by force pushing, then well yes all
bets are off.

> And is there a promise to keep these around indefinitely?

We do not need to maintain multiple versions of bits or increment the
version number every time we wrote a test. Typically new
commits/changes will be made if we wanted to fix a bug or upgrade
acpica or whatever. Test scripts are python based and can be pushed
into bits while generating a new iso for testing purposes.

>
> > > Also as long as test passes, sure. But if it fails one will
> > > need the sources to investigate.
> >
> > sources might also be needed to write the tests.
> >
> > >
> > > Let's start with building things from source.
> >
> > hmm. bitys uses old autotools, not ninja and takes about 10/15 mins to
> > build depending on parallelity and build host.
>
> Right. But whoever wants to use these just needs to do it once.

but I thought you were suggesting we built bits every time the test is run?

>
>
> > Add an option
> > > of prebuilt binaries as an optimization once things
> > > stabilize.
> > >
> > >
> > > > > - new target make check-roms,
> > > >
> > > > I think make pytest or some such is better and more generic if other
> > > > such tests in other areas follow suit.
> > >
> > > The name is not critical in my mind, but I think we need to decide
> > > what exactly differentiates it from other tests.
> > >
> > >
> > > >
> > > > if the clone exists locally -
> > > > >   run the test, if not - skip it
> > > >
> > > > if download of the bits binaries fail, skip it.
> > >
> > > You seem to be recreating either git or avocado or both here.
> > >
> > > Personally I want something that works offline.
> > >
> > >
> > >
> > > > > - as for using pre-generates ISOs as an optimization,
> > > > >   I'm not sure how important that is, if yes -
> > > > >   we can add another repo and another make target along the
> > > > >   same lines
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > MST
> > > > >
> > >
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> but I thought you were suggesting we built bits every time the test is run?

In my opinion 3 scenarios are worth supporting:
- people not touching ACPI, including users - simply don't run the tests,
	comparing tables with expected output should be enough
- people making changes indirectly affecting ACPI -
 	use tests to validate that tables are still well formed,
	using a pre built binary should be enough IMO
- people working on ACPI - use tests to analyse the tables,
	building from source might be necessary for debugging,
	sources change very rarely
- people developing the tests
	building from source is required

So I would suggest basically two github repos, one with binaries one with
sources. We'll keep relevant hashes to use in the script.
All in all not that different from submodules but I guess
people have submodules and that is that.

And I personally would probably not tie it to CI whoever owns the
repository can worry about the builds, and I think keeping
things distributed is important.

So
- people not touching ACPI - make check should see directory not found
	and skip the test
- people making changes indirectly affecting ACPI -
	check out binaries and use
- people working on ACPI -
	see that source directory is present, go there
	and run make. should not rebuild each time right?
- people developing the tests
	building from source is required


-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 6:25 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> > but I thought you were suggesting we built bits every time the test is run?
>
> In my opinion 3 scenarios are worth supporting:
> - people not touching ACPI, including users - simply don't run the tests,
>         comparing tables with expected output should be enough
> - people making changes indirectly affecting ACPI -
>         use tests to validate that tables are still well formed,
>         using a pre built binary should be enough IMO
> - people working on ACPI - use tests to analyse the tables,
>         building from source might be necessary for debugging,
>         sources change very rarely
> - people developing the tests
>         building from source is required
>
> So I would suggest basically two github repos, one with binaries one with
> sources. We'll keep relevant hashes to use in the script.
> All in all not that different from submodules but I guess
> people have submodules and that is that.
>
> And I personally would probably not tie it to CI whoever owns the
> repository can worry about the builds, and I think keeping
> things distributed is important.
>
> So
> - people not touching ACPI - make check should see directory not found
>         and skip the test
> - people making changes indirectly affecting ACPI -
>         check out binaries and use
> - people working on ACPI -
>         see that source directory is present, go there
>         and run make. should not rebuild each time right?
> - people developing the tests
>         building from source is required

Ok I have now committed a Dockerfile that has all the build time
dependencies and builds bits and generates the tarballs/zip file that
my test requires:
https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile

We just need to fork the repo and generate automated builds with this
Dockerfile and put the binaries somewhere. This should also help
developers requiring to rebuild bits when necessary.
Oh and btw, I also made bits compliant with the latest gcc 11 compiler
that comes with Ubuntu 22.04 LTS :-)

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Mon, Jul 4, 2022 at 7:02 PM Ani Sinha <ani@anisinha.ca> wrote:
>
> On Fri, Jul 1, 2022 at 6:25 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Fri, Jul 01, 2022 at 03:44:32PM +0530, Ani Sinha wrote:
> > > but I thought you were suggesting we built bits every time the test is run?
> >
> > In my opinion 3 scenarios are worth supporting:
> > - people not touching ACPI, including users - simply don't run the tests,
> >         comparing tables with expected output should be enough
> > - people making changes indirectly affecting ACPI -
> >         use tests to validate that tables are still well formed,
> >         using a pre built binary should be enough IMO
> > - people working on ACPI - use tests to analyse the tables,
> >         building from source might be necessary for debugging,
> >         sources change very rarely
> > - people developing the tests
> >         building from source is required
> >
> > So I would suggest basically two github repos, one with binaries one with
> > sources. We'll keep relevant hashes to use in the script.
> > All in all not that different from submodules but I guess
> > people have submodules and that is that.
> >
> > And I personally would probably not tie it to CI whoever owns the
> > repository can worry about the builds, and I think keeping
> > things distributed is important.
> >
> > So
> > - people not touching ACPI - make check should see directory not found
> >         and skip the test
> > - people making changes indirectly affecting ACPI -
> >         check out binaries and use
> > - people working on ACPI -
> >         see that source directory is present, go there
> >         and run make. should not rebuild each time right?
> > - people developing the tests
> >         building from source is required
>
> Ok I have now committed a Dockerfile that has all the build time
> dependencies and builds bits and generates the tarballs/zip file that
> my test requires:
> https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile
>
> We just need to fork the repo and generate automated builds with this
> Dockerfile and put the binaries somewhere. This should also help
> developers requiring to rebuild bits when necessary.
> Oh and btw, I also made bits compliant with the latest gcc 11 compiler
> that comes with Ubuntu 22.04 LTS :-)

Pushed a build script here:
https://github.com/ani-sinha/bits/commit/90b99ef05d55ead4b33b2fb19ad07dfb9682ec92

and the bios bits binaries are in this branch:
https://github.com/ani-sinha/bits/tree/bits-builds

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Fri, Jul 1, 2022 at 1:08 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jul 01, 2022 at 12:58:33PM +0530, Ani Sinha wrote:
> > On Fri, Jul 1, 2022 at 12:23 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Fri, Jul 01, 2022 at 06:12:14AM +0200, Thomas Huth wrote:
> > > > I even wouldn't mind if you put your python stuff in a new directory like
> > > > tests/pytests/ for example, as long as it downloads your binaries separately
> > > > - as I wrote in another mail, the avocado framework rather looks like an
> > > > oddball in our test framework nowadays since it uses a separate test runner
> > > > and not the meson test harness, so having a new approach for python-based
> > > > tests is maybe even a good idea. I just really want to avoid that this goes
> > > > into tests/qtest (since it really does not belong there), and please don't
> > > > add more external stuff via git submodules, that's really the wrong approach
> > > > for this.
> > >
> > > I get it, people hate submodules with passion.  I think trying another
> > > approach for testing that is neither avocado nor qtest is
> > > not too bad. As long as this is not user visible, we can
> > > allow ourselves space to experiment.
> > >
> > > OK so, how about this:
> > > - put it in a new directory: tests/roms?
> > > - create repo for a fork of biosbits under git.qemu.org
> > > - roll our own analog to git submodules: a script
> > >   that clones the repo
> >
> > No need to clone the whole repo. We can simply download the binaries
> > that the girlab CI job would generate from the bits sources in that
> > repo.
> > We need to clone if we are always building bits from source for every
> > test. That is not necessary IMHO since much of the bits package would
> > remain as is without modification.
>
> IMHO CI job idea isn't great since isn't versioned at all, is it?
> Also as long as test passes, sure. But if it fails one will
> need the sources to investigate.
>
> Let's start with building things from source. Add an option
> of prebuilt binaries as an optimization once things
> stabilize.
>
>
> > > - new target make check-roms,
> >
> > I think make pytest or some such is better and more generic if other
> > such tests in other areas follow suit.
>
> The name is not critical in my mind, but I think we need to decide
> what exactly differentiates it from other tests.
>
>
> >
> > if the clone exists locally -
> > >   run the test, if not - skip it
> >
> > if download of the bits binaries fail, skip it.
>
> You seem to be recreating either git or avocado or both here.
>
> Personally I want something that works offline.

Well we need to make a call as to how to make things work offline, if
that's what we wanted.  I have made changes in v2 to download the
binaries from my github repo. It works great. If download fails, it
skips the test. If those files are available before, it will not
attempt to download them.
https://pastebin.com/nrcdN8WS and
https://pastebin.com/m7Dx3PZk

The Dockerfile and build script I have in the repo will generate these
artifacts. So if you wanted to run the test offline, either build
those from source and place them where the test expects to find them
or build them from source using Dockerfile/build script and place them
there before running the test.

>
>
>
> > > - as for using pre-generates ISOs as an optimization,
> > >   I'm not sure how important that is, if yes -
> > >   we can add another repo and another make target along the
> > >   same lines
> > >
> > >
> > >
> > > --
> > > MST
> > >
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 06:52:55PM +0530, Ani Sinha wrote:
> So why burden everyone by having bios-tables-test when it only affects
> acpi/smbios developers?

No it doesn't. Any change in qemu can break ACPI - it's enough to
register an MR overlapping the ACPI registers.  So we have a basic test
making sure ACPI is in expected state and have everyone run it.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 01:55:48PM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 06:06:19PM +0530, Ani Sinha wrote:
> > On Tue, Jun 28, 2022 at 5:40 PM Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > On Tue, 28 Jun 2022 at 12:50, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > I think the main difference is not even in how it works, it's
> > > > in what it does. Which is check that ACPI tables are sane.
> > > > Who cares about that? Well developers do when they change the
> > > > tables. Users really don't because for users we have the expected
> > > > tables in tree and we check against these.
> > >
> > > It wants to build and run a big guest binary blob -- that to me is
> > > the main difference. Users don't much care about any of our tests,
> > 
> > perhaps but we do enforce patch submitters to make sure make check
> > passes before submitting patches. make check-avocado is not run as
> > part of make check, requires considerable disk space to download all
> > guest images and hence generally not run by patch submitters. Making
> > bits parts of avocado tests almost defeats the purpose of having this
> > test at all.
> 
> This proposed biosbits test also involves a considerable download.
> The test is said to be irrelevant for anyone except those working
> on a fairly narrow set of QEMU firmware related bits. So by the same
> rationale we shouldn't impose that burden on everyone working on
> QEMU by having it in qtest. Making it entirely optional, only
> downloaded by avocado on demand, for the people who need to run
> the tests is best.
> 
> With regards,
> Daniel

Optional is fine. I would like to make life easy for
developers working on ACPI though, the field is unforgiving
enough as it is.
How about we skip the test if the submodule is not checked out?

> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote:
> On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > > 
> > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > > > > packages from the net during testing?
> > > > > > > > > > > > 
> > > > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > > > Following two are downloaded:
> > > > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > > > > 
> > > > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > > > > 
> > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > > > > 
> > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > > > the net anyway.
> > > > > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > > > > 
> > > > > > > > > > Currently I only need this:
> > > > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > > > which is the basic TAP processing library for python.
> > > > > > > > > > 
> > > > > > > > > > It seems its only installed through pip:
> > > > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > > > > 
> > > > > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > > > > the python src tree? Not sure ...
> > > > > > > > > 
> > > > > > > > > It's pretty small for sure. Another submodule?
> > > > > > > > 
> > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > > > > will receive new releases through
> > > > > > > > https://pypi.org/project/tap.py/
> > > > > > > > so forking is OK but someone has to keep this updated.
> > > > > > > > 
> > > > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > > > > 
> > > > > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > > > > bits, I think you should loop him into this discussion, too.
> > > > > > > 
> > > > > > >    Thomas
> > > > > > 
> > > > > > submodule does not mean we fork necessarily. We could have
> > > > > > all options: check for the module and use it if there, if not
> > > > > > use one from system if not there install with pip ..
> > > > > > But yea, I'm not sure what's best either.
> > > > > 
> > > > > submodules create a dependency on an internet connection, too. So before you
> > > > > add yet another submodule (which have a couple of other disadvantages), I
> > > > > think you could also directly use the venv here.
> > > > 
> > > > Definitely not submodules.
> > > > 
> > > > We need to get out of the mindset that submodules are needed for every new
> > > > dependancy we add. Submodules are only appropriate if the external project
> > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > > > need to bundle in order to prevent a regression for previously deployed
> > > > QEMU installs where the dependancy is known not to exist on all our
> > > > supported platforms.
> > > > 
> > > > This does not apply in this case, because the proposed use of tappy is
> > > > merely for a test case. Meson just needs to check if tappy exists and if
> > > > it does, then use it, otherwise skip the tests that need it. The user can
> > > > arrange to install tappy, as they do with the majority of other deps.
> > > > 
> > > > If John's venv stuff is relevant, then we don't even need the meson checks,
> > > > just delegate to the venv setup.
> > > > 
> > > > Regardless, no submodules are needed or desirable.
> > > 
> > > What about keeping biosbits stuff? Source or pre-built.
> > 
> > Shipping them as pre-built binaries in QEMU is not a viable option
> > IMHO, especially for grub as a GPL'd project we need to be extremely
> > clear about the exact corresponding source and build process for any
> > binary.
> > 
> > For this kind of thing I would generally expect the distro to provide
> > packages that we consume. Looking at biosbits I see it is itself
> > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > even an ancient version of python as a submodule.
> > 
> > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > too, all with dubious license compliance. I don't think this looks like
> > something we should have in qemu.git or qemu tarballs. It will also
> > make it challenging for the distro to take biosbits at all, unless
> > those 3rd party bundles can be eliminated in favour of using existing
> > builds their have packaged for grub, python, libffi, etc.
> 
> So if this depends on some third party binary bits, I think this is pretty
> similar to the tests in the avocado directory ... there we download third
> party binaries, too... Wouldn't it make sense to adapt your tests to that
> framework?

Now that you mention it, avocado does feel like a more appropriate fit.
IIUC the biosbits project appears to be effectively providing a custom
guest OS ISO image. IOW this testing is quite biased towards being
integration testing which is the target of avocado, while qtest is much
more to the unit testing end of the spectrum. This would avoid all the
discussion and patches around introducing python to qtest

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:14:53AM +0100, Daniel P. Berrangé wrote:
> Now that you mention it, avocado does feel like a more appropriate fit.
> IIUC the biosbits project appears to be effectively providing a custom
> guest OS ISO image. IOW this testing is quite biased towards being
> integration testing which is the target of avocado, while qtest is much
> more to the unit testing end of the spectrum. This would avoid all the
> discussion and patches around introducing python to qtest
> 
> With regards,
> Daniel

Yes it's an ISO image but not a full OS. So IMHO not really a
integration test, more like  a unit test for ACPI - current
tests we have just compare ACPI to expected file which is
okay but very limited.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 12.21, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 11:14:53AM +0100, Daniel P. Berrangé wrote:
>> Now that you mention it, avocado does feel like a more appropriate fit.
>> IIUC the biosbits project appears to be effectively providing a custom
>> guest OS ISO image. IOW this testing is quite biased towards being
>> integration testing which is the target of avocado, while qtest is much
>> more to the unit testing end of the spectrum. This would avoid all the
>> discussion and patches around introducing python to qtest
>>
>> With regards,
>> Daniel
> 
> Yes it's an ISO image but not a full OS.

The avocado framework is not meant for full OS testing only. We have a 
couple of tests there that just run a small firmware image. For example:

- tests/avocado/machine_m68k_nextcube.py : Just runs the original firmware 
of the NexT Cube machine

- tests/avocado/ppc_405.py : Just runs an U-Boot firmware image

Some of the tests even do not download anything at all and just run a QEMU 
instance via python, e.g.:

- tests/avocado/virtio_check_params.py

- tests/avocado/x86_cpu_model_versions.py

So I fail to see why biosbits should be so much different here.

Of course we could argue whether the avocado framework is really such a 
great fit for the QEMU test suite (as I said in another mail, it rather 
feels like an oddball nowadays), but that's a different argument.

  Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 3:45 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote:
> > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > > >
> > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > > > > > packages from the net during testing?
> > > > > > > > > > > > >
> > > > > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > > > > Following two are downloaded:
> > > > > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > > > > >
> > > > > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > > > > >
> > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > > > > >
> > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > > > > the net anyway.
> > > > > > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > > > > >
> > > > > > > > > > > Currently I only need this:
> > > > > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > > > > which is the basic TAP processing library for python.
> > > > > > > > > > >
> > > > > > > > > > > It seems its only installed through pip:
> > > > > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > > > > >
> > > > > > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > > > > > the python src tree? Not sure ...
> > > > > > > > > >
> > > > > > > > > > It's pretty small for sure. Another submodule?
> > > > > > > > >
> > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > > > > > will receive new releases through
> > > > > > > > > https://pypi.org/project/tap.py/
> > > > > > > > > so forking is OK but someone has to keep this updated.
> > > > > > > > >
> > > > > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > > > > >
> > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > > > > > bits, I think you should loop him into this discussion, too.
> > > > > > > >
> > > > > > > >    Thomas
> > > > > > >
> > > > > > > submodule does not mean we fork necessarily. We could have
> > > > > > > all options: check for the module and use it if there, if not
> > > > > > > use one from system if not there install with pip ..
> > > > > > > But yea, I'm not sure what's best either.
> > > > > >
> > > > > > submodules create a dependency on an internet connection, too. So before you
> > > > > > add yet another submodule (which have a couple of other disadvantages), I
> > > > > > think you could also directly use the venv here.
> > > > >
> > > > > Definitely not submodules.
> > > > >
> > > > > We need to get out of the mindset that submodules are needed for every new
> > > > > dependancy we add. Submodules are only appropriate if the external project
> > > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > > > > need to bundle in order to prevent a regression for previously deployed
> > > > > QEMU installs where the dependancy is known not to exist on all our
> > > > > supported platforms.
> > > > >
> > > > > This does not apply in this case, because the proposed use of tappy is
> > > > > merely for a test case. Meson just needs to check if tappy exists and if
> > > > > it does, then use it, otherwise skip the tests that need it. The user can
> > > > > arrange to install tappy, as they do with the majority of other deps.
> > > > >
> > > > > If John's venv stuff is relevant, then we don't even need the meson checks,
> > > > > just delegate to the venv setup.
> > > > >
> > > > > Regardless, no submodules are needed or desirable.
> > > >
> > > > What about keeping biosbits stuff? Source or pre-built.
> > >
> > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > clear about the exact corresponding source and build process for any
> > > binary.
> > >
> > > For this kind of thing I would generally expect the distro to provide
> > > packages that we consume. Looking at biosbits I see it is itself
> > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > even an ancient version of python as a submodule.
> > >
> > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > too, all with dubious license compliance. I don't think this looks like
> > > something we should have in qemu.git or qemu tarballs. It will also
> > > make it challenging for the distro to take biosbits at all, unless
> > > those 3rd party bundles can be eliminated in favour of using existing
> > > builds their have packaged for grub, python, libffi, etc.
> >
> > So if this depends on some third party binary bits, I think this is pretty
> > similar to the tests in the avocado directory ... there we download third
> > party binaries, too... Wouldn't it make sense to adapt your tests to that
> > framework?
>
> Now that you mention it, avocado does feel like a more appropriate fit.
> IIUC the biosbits project appears to be effectively providing a custom
> guest OS ISO image. IOW this testing is quite biased towards being
> integration testing which is the target of avocado, while qtest is much
> more to the unit testing end of the spectrum.

This is more like unit testing than integration testing, now that you
mention it. It tests only the bios, very narrowly and does not involve
any OS at all.

This would avoid all the
> discussion and patches around introducing python to qtest
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 4:00 PM Ani Sinha <ani@anisinha.ca> wrote:
>
> On Tue, Jun 28, 2022 at 3:45 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 10:28:04AM +0200, Thomas Huth wrote:
> > > On 28/06/2022 10.23, Daniel P. Berrangé wrote:
> > > > On Tue, Jun 28, 2022 at 01:21:35PM +0530, Ani Sinha wrote:
> > > > > On Tue, Jun 28, 2022 at 1:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > > > > > > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > > > > > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > > > > > > packages from the net during testing?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > > > > > > Following two are downloaded:
> > > > > > > > > > > > > > avocado-framework==88.1
> > > > > > > > > > > > > > pycdlib==1.11.0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > > > > > >
> > > > > > > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > > > > > > the net anyway.
> > > > > > > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > > > > > >
> > > > > > > > > > > > Currently I only need this:
> > > > > > > > > > > > https://github.com/python-tap/tappy
> > > > > > > > > > > > which is the basic TAP processing library for python.
> > > > > > > > > > > >
> > > > > > > > > > > > It seems its only installed through pip:
> > > > > > > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > > > > > >
> > > > > > > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > > > > > > the python src tree? Not sure ...
> > > > > > > > > > >
> > > > > > > > > > > It's pretty small for sure. Another submodule?
> > > > > > > > > >
> > > > > > > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > > > > > > will receive new releases through
> > > > > > > > > > https://pypi.org/project/tap.py/
> > > > > > > > > > so forking is OK but someone has to keep this updated.
> > > > > > > > > >
> > > > > > > > > > I am open to anything. Whatever feels right is fine to me.
> > > > > > > > >
> > > > > > > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > > > > > > bits, I think you should loop him into this discussion, too.
> > > > > > > > >
> > > > > > > > >    Thomas
> > > > > > > >
> > > > > > > > submodule does not mean we fork necessarily. We could have
> > > > > > > > all options: check for the module and use it if there, if not
> > > > > > > > use one from system if not there install with pip ..
> > > > > > > > But yea, I'm not sure what's best either.
> > > > > > >
> > > > > > > submodules create a dependency on an internet connection, too. So before you
> > > > > > > add yet another submodule (which have a couple of other disadvantages), I
> > > > > > > think you could also directly use the venv here.
> > > > > >
> > > > > > Definitely not submodules.
> > > > > >
> > > > > > We need to get out of the mindset that submodules are needed for every new
> > > > > > dependancy we add. Submodules are only appropriate if the external project
> > > > > > is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> > > > > > need to bundle in order to prevent a regression for previously deployed
> > > > > > QEMU installs where the dependancy is known not to exist on all our
> > > > > > supported platforms.
> > > > > >
> > > > > > This does not apply in this case, because the proposed use of tappy is
> > > > > > merely for a test case. Meson just needs to check if tappy exists and if
> > > > > > it does, then use it, otherwise skip the tests that need it. The user can
> > > > > > arrange to install tappy, as they do with the majority of other deps.
> > > > > >
> > > > > > If John's venv stuff is relevant, then we don't even need the meson checks,
> > > > > > just delegate to the venv setup.
> > > > > >
> > > > > > Regardless, no submodules are needed or desirable.
> > > > >
> > > > > What about keeping biosbits stuff? Source or pre-built.
> > > >
> > > > Shipping them as pre-built binaries in QEMU is not a viable option
> > > > IMHO, especially for grub as a GPL'd project we need to be extremely
> > > > clear about the exact corresponding source and build process for any
> > > > binary.
> > > >
> > > > For this kind of thing I would generally expect the distro to provide
> > > > packages that we consume. Looking at biosbits I see it is itself
> > > > bundling a bunch more 3rd party projects, libffi, grub2, and including
> > > > even an ancient version of python as a submodule.
> > > >
> > > > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > > > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > > > too, all with dubious license compliance. I don't think this looks like
> > > > something we should have in qemu.git or qemu tarballs. It will also
> > > > make it challenging for the distro to take biosbits at all, unless
> > > > those 3rd party bundles can be eliminated in favour of using existing
> > > > builds their have packaged for grub, python, libffi, etc.
> > >
> > > So if this depends on some third party binary bits, I think this is pretty
> > > similar to the tests in the avocado directory ... there we download third
> > > party binaries, too... Wouldn't it make sense to adapt your tests to that
> > > framework?
> >
> > Now that you mention it, avocado does feel like a more appropriate fit.
> > IIUC the biosbits project appears to be effectively providing a custom
> > guest OS ISO image. IOW this testing is quite biased towards being
> > integration testing which is the target of avocado, while qtest is much
> > more to the unit testing end of the spectrum.
>
> This is more like unit testing than integration testing, now that you
> mention it. It tests only the bios, very narrowly and does not involve
> any OS at all.

Another thing to consider is that integration testing is further down
the line? Not for once when submitting patches on acpi have I run
them. However, every time I have run make check to make sure
bios-tables-test passes and I am not breaking anything. It's much more
useful to have this kind of thing part of make check before patch
submitters can quickly check for failures either in bios-tables-test
or in bits. Also its lot easier to add new acpi/smbios tests as a part
of this when bios-tables-test and this one are closer together.

>
> This would avoid all the
> > discussion and patches around introducing python to qtest
> >
> > With regards,
> > Daniel
> > --
> > |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> > |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> > |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> >

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 09:23:06AM +0100, Daniel P. Berrangé wrote:
> So bundling a pre-built biosbits in QEMU appears to mean that we're in
> turn going to unexpectedly bundle a bunch of other 3rd party projects
> too, all with dubious license compliance.

Well looks like classical mere aggregation to me ... license issues
need to be figured out if we are to distribute things but I think
this is basically what distros do anyway.

And I doubt we want to support arbitrary versions of grub etc,
they are very distro specific tools.
I don't see why we can't have the resulting ISOs in some submodule -
nothing requires us to distribute it in qemu tarballs.

-- 
MST

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 06:12:48AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 09:23:06AM +0100, Daniel P. Berrangé wrote:
> > So bundling a pre-built biosbits in QEMU appears to mean that we're in
> > turn going to unexpectedly bundle a bunch of other 3rd party projects
> > too, all with dubious license compliance.
> 
> Well looks like classical mere aggregation to me ... license issues
> need to be figured out if we are to distribute things but I think
> this is basically what distros do anyway.
> 
> And I doubt we want to support arbitrary versions of grub etc,
> they are very distro specific tools.
> I don't see why we can't have the resulting ISOs in some submodule -
> nothing requires us to distribute it in qemu tarballs.

If we don't distribute it in the tarballs then these tests won't be
usable for testing release builds of QEMU which feels wrong to me.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 08:48:42AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> > On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > > packages from the net during testing?
> > > > > > > > > 
> > > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > > Following two are downloaded:
> > > > > > > > > avocado-framework==88.1
> > > > > > > > > pycdlib==1.11.0
> > > > > > > > > 
> > > > > > > > > Also see this line in Makefie.include:
> > > > > > > > > 
> > > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > > 
> > > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > > the net anyway.
> > > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > > 
> > > > > > > Currently I only need this:
> > > > > > > https://github.com/python-tap/tappy
> > > > > > > which is the basic TAP processing library for python.
> > > > > > > 
> > > > > > > It seems its only installed through pip:
> > > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > > 
> > > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > > the python src tree? Not sure ...
> > > > > > 
> > > > > > It's pretty small for sure. Another submodule?
> > > > > 
> > > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > > will receive new releases through
> > > > > https://pypi.org/project/tap.py/
> > > > > so forking is OK but someone has to keep this updated.
> > > > > 
> > > > > I am open to anything. Whatever feels right is fine to me.
> > > > 
> > > > John Snow is currently working on the "Pythonification" of various QEMU
> > > > bits, I think you should loop him into this discussion, too.
> > > > 
> > > >   Thomas
> > > 
> > > submodule does not mean we fork necessarily. We could have
> > > all options: check for the module and use it if there, if not
> > > use one from system if not there install with pip ..
> > > But yea, I'm not sure what's best either.
> > 
> > submodules create a dependency on an internet connection, too. So before you
> > add yet another submodule (which have a couple of other disadvantages), I
> > think you could also directly use the venv here.
> 
> Definitely not submodules.
> 
> We need to get out of the mindset that submodules are needed for every new
> dependancy we add. Submodules are only appropriate if the external project
> is designed to be used as a copylib (eg the keycodemapdb tool), or if we
> need to bundle in order to prevent a regression for previously deployed
> QEMU installs where the dependancy is known not to exist on all our
> supported platforms.
> 
> This does not apply in this case, because the proposed use of tappy is
> merely for a test case. Meson just needs to check if tappy exists and if
> it does, then use it, otherwise skip the tests that need it. The user can
> arrange to install tappy, as they do with the majority of other deps.

I think that's reasonable, except I think we should
make configure do the checks as that is when users see the output
of what is available.

By the way, it might make sense
to have a script setting up developer dependencies for qemu
for major platforms.

> If John's venv stuff is relevant, then we don't even need the meson checks,
> just delegate to the venv setup.

IIUC venv does not install things itself. pip does.

> Regardless, no submodules are needed or desirable.
> 
> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Ani Sinha]

On Tue, Jun 28, 2022 at 12:55 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> >>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
> >>>>>>>> packages from the net during testing?
> >>>>>>>
> >>>>>>> We do that too. See requirements.txt in tests/
> >>>>>>> Following two are downloaded:
> >>>>>>> avocado-framework==88.1
> >>>>>>> pycdlib==1.11.0
> >>>>>>>
> >>>>>>> Also see this line in Makefie.include:
> >>>>>>>
> >>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> >>>>>>
> >>>>>> Right but that's avocado since it pulls lots of stuff from
> >>>>>> the net anyway.
> >>>>>> Are the libraries in question not packaged on major distros?
> >>>>>
> >>>>> Currently I only need this:
> >>>>> https://github.com/python-tap/tappy
> >>>>> which is the basic TAP processing library for python.
> >>>>>
> >>>>> It seems its only installed through pip:
> >>>>> https://tappy.readthedocs.io/en/latest/
> >>>>>
> >>>>> I do not think this is packaged by default. It's such a basic library
> >>>>> for parsing test output that maybe we can keep this somewhere within
> >>>>> the python src tree? Not sure ...
> >>>>
> >>>> It's pretty small for sure. Another submodule?
> >>>
> >>> Unlike BITS, this one is likely going to be maintained for a while and
> >>> will receive new releases through
> >>> https://pypi.org/project/tap.py/
> >>> so forking is OK but someone has to keep this updated.
> >>>
> >>> I am open to anything. Whatever feels right is fine to me.
> >>
> >> John Snow is currently working on the "Pythonification" of various QEMU
> >> bits, I think you should loop him into this discussion, too.
> >>
> >>   Thomas
> >
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
>
> submodules create a dependency on an internet connection, too. So before you
> add yet another submodule (which have a couple of other disadvantages), I
> think you could also directly use the venv here.

Not sure what you mean by directly using the venv.

>
>   Thomas
>

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Thomas Huth]

On 28/06/2022 09.49, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 12:55 PM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
>>> On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
>>>>>>>>>> No problem with that. So that's venv. But do we need pip and pulling
>>>>>>>>>> packages from the net during testing?
>>>>>>>>>
>>>>>>>>> We do that too. See requirements.txt in tests/
>>>>>>>>> Following two are downloaded:
>>>>>>>>> avocado-framework==88.1
>>>>>>>>> pycdlib==1.11.0
>>>>>>>>>
>>>>>>>>> Also see this line in Makefie.include:
>>>>>>>>>
>>>>>>>>> $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>>>>>>>>
>>>>>>>> Right but that's avocado since it pulls lots of stuff from
>>>>>>>> the net anyway.
>>>>>>>> Are the libraries in question not packaged on major distros?
>>>>>>>
>>>>>>> Currently I only need this:
>>>>>>> https://github.com/python-tap/tappy
>>>>>>> which is the basic TAP processing library for python.
>>>>>>>
>>>>>>> It seems its only installed through pip:
>>>>>>> https://tappy.readthedocs.io/en/latest/
>>>>>>>
>>>>>>> I do not think this is packaged by default. It's such a basic library
>>>>>>> for parsing test output that maybe we can keep this somewhere within
>>>>>>> the python src tree? Not sure ...
>>>>>>
>>>>>> It's pretty small for sure. Another submodule?
>>>>>
>>>>> Unlike BITS, this one is likely going to be maintained for a while and
>>>>> will receive new releases through
>>>>> https://pypi.org/project/tap.py/
>>>>> so forking is OK but someone has to keep this updated.
>>>>>
>>>>> I am open to anything. Whatever feels right is fine to me.
>>>>
>>>> John Snow is currently working on the "Pythonification" of various QEMU
>>>> bits, I think you should loop him into this discussion, too.
>>>>
>>>>    Thomas
>>>
>>> submodule does not mean we fork necessarily. We could have
>>> all options: check for the module and use it if there, if not
>>> use one from system if not there install with pip ..
>>> But yea, I'm not sure what's best either.
>>
>> submodules create a dependency on an internet connection, too. So before you
>> add yet another submodule (which have a couple of other disadvantages), I
>> think you could also directly use the venv here.
> 
> Not sure what you mean by directly using the venv.

I meant installing the dependency via pip in a venv, as you are currently 
doing it via the acpi-bits-test-venv.sh script.

  Thomas

Re: venv for python qtest bits? (was: Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests)

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 09:25:35AM +0200, Thomas Huth wrote:
> On 28/06/2022 09.10, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 09:03:33AM +0200, Thomas Huth wrote:
> > > > > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > > > > packages from the net during testing?
> > > > > > > > 
> > > > > > > > We do that too. See requirements.txt in tests/
> > > > > > > > Following two are downloaded:
> > > > > > > > avocado-framework==88.1
> > > > > > > > pycdlib==1.11.0
> > > > > > > > 
> > > > > > > > Also see this line in Makefie.include:
> > > > > > > > 
> > > > > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > > > > > 
> > > > > > > Right but that's avocado since it pulls lots of stuff from
> > > > > > > the net anyway.
> > > > > > > Are the libraries in question not packaged on major distros?
> > > > > > 
> > > > > > Currently I only need this:
> > > > > > https://github.com/python-tap/tappy
> > > > > > which is the basic TAP processing library for python.
> > > > > > 
> > > > > > It seems its only installed through pip:
> > > > > > https://tappy.readthedocs.io/en/latest/
> > > > > > 
> > > > > > I do not think this is packaged by default. It's such a basic library
> > > > > > for parsing test output that maybe we can keep this somewhere within
> > > > > > the python src tree? Not sure ...
> > > > > 
> > > > > It's pretty small for sure. Another submodule?
> > > > 
> > > > Unlike BITS, this one is likely going to be maintained for a while and
> > > > will receive new releases through
> > > > https://pypi.org/project/tap.py/
> > > > so forking is OK but someone has to keep this updated.
> > > > 
> > > > I am open to anything. Whatever feels right is fine to me.
> > > 
> > > John Snow is currently working on the "Pythonification" of various QEMU
> > > bits, I think you should loop him into this discussion, too.
> > > 
> > >   Thomas
> > 
> > submodule does not mean we fork necessarily. We could have
> > all options: check for the module and use it if there, if not
> > use one from system if not there install with pip ..
> > But yea, I'm not sure what's best either.
> 
> submodules create a dependency on an internet connection, too.

Yes but this dependency is explicit.

> So before you
> add yet another submodule (which have a couple of other disadvantages), I
> think you could also directly use the venv here.
> 
>  Thomas

Re: [PATCH 11/12] acpi/tests/bits: add README file for bits qtests

[Ani Sinha]

+danPb who pointed me to the TAP stuff. Maybe he has some good ideas on this.

On Tue, Jun 28, 2022 at 12:27 PM Ani Sinha <ani@anisinha.ca> wrote:
>
> On Tue, Jun 28, 2022 at 12:20 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 28, 2022 at 12:06:31PM +0530, Ani Sinha wrote:
> > > On Tue, Jun 28, 2022 at 11:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Jun 28, 2022 at 11:46:13AM +0530, Ani Sinha wrote:
> > > > > On Tue, Jun 28, 2022 at 11:36 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Jun 28, 2022 at 10:27:38AM +0530, Ani Sinha wrote:
> > > > > > > On Tue, Jun 28, 2022 at 3:56 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Mon, Jun 27, 2022 at 12:58:55PM +0530, Ani Sinha wrote:
> > > > > > > > > The README file is added describing the directory structure and the purpose
> > > > > > > > > of every file it contains. It also describes how to add new tests, make changes
> > > > > > > > > to existing tests or bits config files or regenerate the bits software.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca>
> > > > > > > > > ---
> > > > > > > > >  tests/qtest/acpi-bits/README | 168 +++++++++++++++++++++++++++++++++++
> > > > > > > > >  1 file changed, 168 insertions(+)
> > > > > > > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > > > > > > >
> > > > > > > > > diff --git a/tests/qtest/acpi-bits/README b/tests/qtest/acpi-bits/README
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000000..97b15f1665
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/tests/qtest/acpi-bits/README
> > > > > > > > > @@ -0,0 +1,168 @@
> > > > > > > > > +=============================================================================
> > > > > > > > > +ACPI/SMBIOS QTESTS USING BIOSBITS
> > > > > > > > > +=============================================================================
> > > > > > > > > +
> > > > > > > > > +Biosbits is a software written by Josh Triplett that can be downloaded by
> > > > > > > > > +visiting https://biosbits.org/. The github codebase can be found here:
> > > > > > > > > +https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > > > > > > > +the bios components such as acpi and smbios tables directly through acpica
> > > > > > > > > +bios interpreter (a freely available C based library written by Intel,
> > > > > > > > > +downloadable from https://acpica.org/ and is included with biosbits) without an
> > > > > > > > > +operating system getting involved in between.
> > > > > > > > > +There are several advantages to directly testing the bios in a real physical
> > > > > > > > > +machine or VM as opposed to indirectly discovering bios issues through the
> > > > > > > > > +operating system. For one thing, the OSes tend to hide bios problems from the
> > > > > > > > > +end user. The other is that we have more control of what we wanted to test
> > > > > > > > > +and how by directly using acpica interpreter on top of the bios on a running
> > > > > > > > > +system. More details on the inspiration for developing biosbits and its real
> > > > > > > > > +life uses can be found in (a) and (b).
> > > > > > > > > +This directory contains QEMU qtests written in python that exercizes the QEMU
> > > > > > > > > +bios components using biosbits and reports test failures.
> > > > > > > > > +
> > > > > > > > > +These tests use python virtual environment. In debian/ubuntu system, the tests
> > > > > > > > > +would require python3.8-venv and python3-pip packages to be installed.
> > > > > > > >
> > > > > > > > Why do we mess with venv and pip? Certainly possible but
> > > > > > > > what's wrong with using distro provided packages?
> > > > > > >
> > > > > > > There are two things:
> > > > > > > (a) We are already using pip and venv for our avocado based
> > > > > > > integration tests. Look for TESTS_VENV_DIR in Makefile.include under
> > > > > > > tests.
> > > > > > > (b) the venv is primarily needed because I wanted to take advantage of
> > > > > > > our rich python library that handles QEMU based machines. There are
> > > > > > > python qtest libraries as well. These are well tested and used with
> > > > > > > integration tests and I wanted to keep the test part of the code
> > > > > > > simple by simply reusing them. however, in order to use them, we need
> > > > > > > a venv environment within which these qemu python libraries are
> > > > > > > installed. Integration tests does the same thing.
> > > > > > >
> > > > > > > A note about my language of choice - python. I gave a lot of thoughts
> > > > > > > on this. We do not do a lot of stuff here. All we do is:
> > > > > > > (a) generate bits iso.
> > > > > > > (b) spawn a QEMU vm with the iso which then runs a bunch of tests within the vm.
> > > > > > > (c) collect and analyze logs.
> > > > > > >
> > > > > > > We are not inspecting guest memory or manipulating devices or pci
> > > > > > > buses. We do not need the power of C here. We need something that is
> > > > > > > simple to write, easy to maintain and understand and can deal with
> > > > > > > things like manipulating text files and configs easily. Python seems a
> > > > > > > better fit for the role.
> > > > > >
> > > > > > No problem with that. So that's venv. But do we need pip and pulling
> > > > > > packages from the net during testing?
> > > > >
> > > > > We do that too. See requirements.txt in tests/
> > > > > Following two are downloaded:
> > > > > avocado-framework==88.1
> > > > > pycdlib==1.11.0
> > > > >
> > > > > Also see this line in Makefie.include:
> > > > >
> > > > > $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
> > > >
> > > > Right but that's avocado since it pulls lots of stuff from
> > > > the net anyway.
> > > > Are the libraries in question not packaged on major distros?
> > >
> > > Currently I only need this:
> > > https://github.com/python-tap/tappy
> > > which is the basic TAP processing library for python.
> > >
> > > It seems its only installed through pip:
> > > https://tappy.readthedocs.io/en/latest/
> > >
> > > I do not think this is packaged by default. It's such a basic library
> > > for parsing test output that maybe we can keep this somewhere within
> > > the python src tree? Not sure ...
> >
> > It's pretty small for sure. Another submodule?
>
> Unlike BITS, this one is likely going to be maintained for a while and
> will receive new releases through
> https://pypi.org/project/tap.py/
> so forking is OK but someone has to keep this updated.
>
> I am open to anything. Whatever feels right is fine to me.
>
> >
> > > >
> > > > > >
> > > > > > > >
> > > > > > > > > +
> > > > > > > > > +A brief description of the contents of this directory follows:
> > > > > > > > > +
> > > > > > > > > +├── acpi-bits-test.py
> > > > > > > > > +├── acpi-bits-test-venv.sh
> > > > > > > > > +├── bits-config
> > > > > > > > > +│   ├── bits-cfg.txt
> > > > > > > > > +│   └── meson.build
> > > > > > > > > +├── bits-tests
> > > > > > > > > +│   ├── meson.build
> > > > > > > > > +│   ├── smbios.py
> > > > > > > > > +│   ├── smilatency.py
> > > > > > > > > +│   ├── testacpi.py
> > > > > > > > > +│   └── testcpuid.py
> > > > > > > > > +├── meson.build
> > > > > > > > > +├── prebuilt
> > > > > > > > > +│   ├── bits-2095-grub.tar.gz
> > > > > > > > > +│   ├── bits-2095.zip
> > > > > > > > > +│   └── meson.build
> > > > > > > > > +├── README
> > > > > > > > > +└── requirements.txt
> > > > > > > > > +
> > > > > > > > > +acpi-bits:
> > > > > > > > > + - acpi-bits-test-venv.sh: This is a shell script that sets up the virtual
> > > > > > > > > +   environment necessary for the actual python test script to run. Amongst
> > > > > > > > > +   other things, it makes sure that QEMU python library is available within
> > > > > > > > > +   that venv so that QEMU machines can be forked. QEMU python library can be
> > > > > > > > > +   found within python/ directory in QEMU source.
> > > > > > > > > +   After setting up the virtual environment, it runs the python test script
> > > > > > > > > +   from within that environment.
> > > > > > > > > +   If you want to enable verbose mode only for bits test and run make check,
> > > > > > > > > +   one trick is to add V=1 before the call to execute the python script in
> > > > > > > > > +   this file.
> > > > > > > > > + - acpi-bits-test.py: This is the main python test script that generates a
> > > > > > > > > +   biosbits iso. It then spawns a QEMU VM with it, collects the logs and reports
> > > > > > > > > +   test failures. This is the script one would be interested in if they wanted
> > > > > > > > > +   to add or change some component of the log parsing, add a new commandline to
> > > > > > > > > +   how QEMU is spawned etc. Test writers typically would not need to modify
> > > > > > > > > +   this script unless they wanted to enhance or change the log parsing for
> > > > > > > > > +   their tests.
> > > > > > > > > + - requirements.txt: This text file contains the dependent python libraries
> > > > > > > > > +   needed for the tests to run. If a new dependent library is needed, it would
> > > > > > > > > +   be added here as a new entry and then acpi-bits-test-venv.sh would install
> > > > > > > > > +   it when setting up the python virtual environment using pip.
> > > > > > > > > + - README: This text file.
> > > > > > > > > +
> > > > > > > > > +acpi-bits/bits-config:
> > > > > > > > > +   This location contains biosbits config files that determine how the software
> > > > > > > > > +   runs the tests.
> > > > > > > > > + - bits-config.txt: this is the biosbits config file that determines what tests
> > > > > > > > > +   or actions are performed by bits. The description of the config options are
> > > > > > > > > +   provided in the file itself.
> > > > > > > > > +
> > > > > > > > > +acpi-bits/prebuilt:
> > > > > > > > > +   This location contains prebuilt biosbits binaries that are used to generate
> > > > > > > > > +   the bits iso file for testing.
> > > > > > > > > + - bits-2095.zip: The contents from this zip archive are the main contents of
> > > > > > > > > +    the iso file that are used for testing. This binary zip archive also
> > > > > > > > > +    contains the full source of the bits software including the full text of
> > > > > > > > > +    the license agreement listed here:
> > > > > > > > > +    https://github.com/biosbits/bits/blob/master/COPYING
> > > > > > > > > +    The source tarball can be found in this location in the zip file:
> > > > > > > > > +    boot/src/bits-2095.tar.gz
> > > > > > > > > +    The additional changes beyond those that are present in the official
> > > > > > > > > +    biosbits github repository can be found here:
> > > > > > > > > +    https://github.com/ani-sinha/bits/tree/bits-qemu-logging
> > > > > > > > > +
> > > > > > > > > +    Basically these changes can be categorized into the following:
> > > > > > > > > +    (a) changes to make sure biosbits builds with the latest gcc compiler
> > > > > > > > > +    (gcc 9.4).
> > > > > > > > > +    (b) upgrade of acpica to the latest version (march 2022).
> > > > > > > > > +    (c) send bits logs to the debug IO port at addresss 0x403 so that isa
> > > > > > > > > +    debugcon can be used to collect the logs.
> > > > > > > > > +    (d) send a gub halt command to shutdown the VM once all the tests have been
> > > > > > > > > +    executed.
> > > > > > > > > +
> > > > > > > > > +    This zip file is automatically generated by the bits build system. It can
> > > > > > > > > +    be found in the bits build directory and it is suffixed by the bits build
> > > > > > > > > +    number.
> > > > > > > > > +    Normally, there is no need to make any alteration to this zip archive
> > > > > > > > > +    unless one wanted to change the bits software itself (for example to add
> > > > > > > > > +    a new config option or change actions of existing options or change the
> > > > > > > > > +    debug IO port etc). When such changes are made and a new biosbits software
> > > > > > > > > +    is needed to be generated, please refresh this zip archive as well as the
> > > > > > > > > +    grub tarball at the same time. Tests will break if changes are made to bits
> > > > > > > > > +    that are incompatible with existing behavior. So care must be taken to make
> > > > > > > > > +    sure that the change is compatible with existing bits software as much as
> > > > > > > > > +    possible. When a new config option is introduced for example, bits must
> > > > > > > > > +    be upadated here first before introducing the new config option through
> > > > > > > > > +    a later commit.
> > > > > > > >
> > > > > > > > I don't think playing with source tarballs is a reasonable work environment.
> > > > > > >
> > > > > > > I agree. However, we do not do much with the source tarball. It is
> > > > > > > there as part of bits to satisfy the license requirement. If we need
> > > > > > > to manipulate any test script that is in the source file, we would
> > > > > > > copy it over and keep it in the bits-test directory and change it
> > > > > > > there.
> > > > > > >
> > > > > > > > Let's use submodules just like e.g. firmware does?
> > > > > > >
> > > > > > > Yes I also proposed this to Igor on IRC. We can/maybe need to figure
> > > > > > > out a place to store the bits source if we think my github is not the
> > > > > > > best place. However, we need the source only if/when we need to
> > > > > > > rebuild bits. I suspect it won't be too often if at all.
> > > > > >
> > > > > > Why not on git.qemu.org ?
> > > > >
> > > > > Sounds good to me. Do I get push access? :-)
> > > > >
> > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > + - prebuilt/bits-2095-grub.tar.gz: This tarbball contains bits generated grub
> > > > > > > > > +    scripts and modules to the prebuilt directory. These prebuilt grub
> > > > > > > > > +    artifacts are required in order to generate a bootable bits iso file that
> > > > > > > > > +    can run tests.
> > > > > > > > > +    In order to generate this tar archive, please put the following two
> > > > > > > > > +    directories that can be found in the bits build directory in a single
> > > > > > > > > +    tar archive file named as bits-<n>-grub.tar.gz where n is the version of
> > > > > > > > > +    bits that generated the archive:
> > > > > > > > > +
> > > > > > > > > +    grub-inst-x86_64-efi
> > > > > > > > > +    grub-inst
> > > > > > > > > +
> > > > > > > > > +    This version should be the same as the version number of bits that generated
> > > > > > > > > +    bits-<n>.zip file. In other words, the two files must be in sync and should
> > > > > > > > > +    come from the same build of biosbits of the same version. Things may still
> > > > > > > > > +    work if they come from different versions but mixing version numbers are
> > > > > > > > > +    not recommended.
> > > > > > > > > +    There is normally no need to regenerate this archive unless some fixes or
> > > > > > > > > +    changes have gone into the grub that are part of biosbits.
> > > > > > > > > +
> > > > > > > > > +acpi-bits/bits-tests:
> > > > > > > > > +   This directory contains biosbits python based tests that are run from within
> > > > > > > > > +   the biosbits environment in the spawned VM. New additions of test cases can
> > > > > > > > > +   be made in the appropriate test file. For example, new acpi tests can go
> > > > > > > > > +   into testacpi.py and one would call testsuite.add_test() to register the new
> > > > > > > > > +   test so that it gets executed as a part of the ACPI tests.
> > > > > > > > > +   It might be occassionally necessary to disable some subtests or add a new
> > > > > > > > > +   test that belongs to a test suite not already present in this directory. To
> > > > > > > > > +   do this, please extract the bits source from the zip file mentioned above.
> > > > > > > > > +   Copy the test suite/script that needs modification (addition of new tests
> > > > > > > > > +   or disabling them) from boot/python location of the extracted directory
> > > > > > > > > +   into this directory.
> > > > > > > > > +
> > > > > > > > > +   step (a): copy unmodified test script to this directory.
> > > > > > > > > +   step (b): update meson.build and add this file to the list.
> > > > > > > > > +   Commit (a) and (b) together in the same commit.
> > > > > > > > > +
> > > > > > > > > +   step (c): perform modifications to the test.
> > > > > > > > > +   Commit (c) in a separate commit.
> > > > > > > > > +
> > > > > > > > > +   The test framework will then use your modified test script to run the test.
> > > > > > > > > +   No further changes would be needed. Please check the logs to make sure that
> > > > > > > > > +   appropriate changes have taken effect.
> > > > > > > > > +
> > > > > > > > > +meson.build files makes sure that the bits qtests are appropriately integrated
> > > > > > > > > +into the QEMU qtest framework and are run with "make check-qtest".
> > > > > > > > > +Currently, the bits test is configured to run only for x86_64 architectures. Bits
> > > > > > > > > +has been built only for this architecture.
> > > > > > > > > +
> > > > > > > > > +
> > > > > > > > > +Author: Ani Sinha <ani@anisinha.ca>
> > > > > > > > > +
> > > > > > > > > +References:
> > > > > > > > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > > > > > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> > > > > > > > > --
> > > > > > > > > 2.25.1
> > > > > > > >
> > > > > >
> > > >
> >

[PATCH 12/12] MAINTAINERS: add myself as the maintainer for acpi biosbits qtests

[Ani Sinha]

I wrote the biosbits test framework, the qtest for testing QEMU with biosbits
and all the related changes. Making myself as the maintainer for biosbits
related files and test scripts.

Signed-off-by: Ani Sinha <ani@anisinha.ca>
---
 MAINTAINERS | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index aaa649a50d..f5358a1f96 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1860,6 +1860,11 @@ S: Supported
 F: hw/acpi/viot.c
 F: hw/acpi/viot.h
 
+ACPI/QTEST/BIOSBITS
+M: Ani Sinha <ani@anisinha.ca>
+S: Supported
+F: tests/qtest/acpi-bits/*
+
 ACPI/HEST/GHES
 R: Dongjiu Geng <gengdongjiu1@gmail.com>
 L: qemu-arm@nongnu.org
-- 
2.25.1

Re: [PATCH 00/12] Introduce new acpi/smbios qtests using biosbits

[Daniel P. Berrangé]

On Mon, Jun 27, 2022 at 12:58:44PM +0530, Ani Sinha wrote:
> Biosbits is a software written by Josh Triplett that can be downloaded by
> visiting https://biosbits.org/. The github codebase can be found here:
> https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> the bios components such as acpi and smbios tables directly through acpica
> bios interpreter (a freely available C based library written by Intel,
> downloadable from https://acpica.org/ and is included with biosbits) without an
> operating system getting involved in between.
> There are several advantages to directly testing the bios in a real physical
> machine or VM as opposed to indirectly discovering bios issues through the
> operating system. For one thing, the OSes tend to hide bios problems from the
> end user. The other is that we have more control of what we wanted to test
> and how by directly using acpica interpreter on top of the bios on a running
> system. More details on the inspiration for developing biosbits and its real
> life uses can be found in (a) and (b).
> This patchset contains QEMU qtests written in python that exercizes the QEMU
> bios components using biosbits and reports test failures.
> 
> Details of each of the files added by this patchset are provided in the README
> file which is part of Patch 11. Every effort to contact Josh, through various
> means including email, twitter, linkedIn etc has failed. Hence, the changes to
> build biosbits with the newer compiler, upgrade acpica and other changes are
> currently maintained in a forked project in my personal github. We may want to
> maintain bits in a separate fork in a stable repository that is accessible by
> QEMU developers.
> 
> The newly introduced qtest currently only run for x86_64 platform. They pass
> both when running make check on a baremetal box as well as from inside a vm.
> 
> Thanks to Igor M for pointing me to this work.
> 
> (a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> (b) https://www.youtube.com/watch?v=36QIepyUuhg
> 
> Ani Sinha (12):
>   qtest: meson.build changes required to integrate python based qtests
>   acpi/tests/bits: add prebuilt bios bits zip archive
>   acpi/tests/bits: add prebuilt bits generated grub modules and scripts

These two files didn't arrive on the mailing list, presumaby because
pre-built binaries made the patches way too large.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 00/12] Introduce new acpi/smbios qtests using biosbits

[Ani Sinha]

On Tue, Jun 28, 2022 at 1:39 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Mon, Jun 27, 2022 at 12:58:44PM +0530, Ani Sinha wrote:
> > Biosbits is a software written by Josh Triplett that can be downloaded by
> > visiting https://biosbits.org/. The github codebase can be found here:
> > https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > the bios components such as acpi and smbios tables directly through acpica
> > bios interpreter (a freely available C based library written by Intel,
> > downloadable from https://acpica.org/ and is included with biosbits) without an
> > operating system getting involved in between.
> > There are several advantages to directly testing the bios in a real physical
> > machine or VM as opposed to indirectly discovering bios issues through the
> > operating system. For one thing, the OSes tend to hide bios problems from the
> > end user. The other is that we have more control of what we wanted to test
> > and how by directly using acpica interpreter on top of the bios on a running
> > system. More details on the inspiration for developing biosbits and its real
> > life uses can be found in (a) and (b).
> > This patchset contains QEMU qtests written in python that exercizes the QEMU
> > bios components using biosbits and reports test failures.
> >
> > Details of each of the files added by this patchset are provided in the README
> > file which is part of Patch 11. Every effort to contact Josh, through various
> > means including email, twitter, linkedIn etc has failed. Hence, the changes to
> > build biosbits with the newer compiler, upgrade acpica and other changes are
> > currently maintained in a forked project in my personal github. We may want to
> > maintain bits in a separate fork in a stable repository that is accessible by
> > QEMU developers.
> >
> > The newly introduced qtest currently only run for x86_64 platform. They pass
> > both when running make check on a baremetal box as well as from inside a vm.
> >
> > Thanks to Igor M for pointing me to this work.
> >
> > (a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > (b) https://www.youtube.com/watch?v=36QIepyUuhg
> >
> > Ani Sinha (12):
> >   qtest: meson.build changes required to integrate python based qtests
> >   acpi/tests/bits: add prebuilt bios bits zip archive
> >   acpi/tests/bits: add prebuilt bits generated grub modules and scripts
>
> These two files didn't arrive on the mailing list, presumaby because
> pre-built binaries made the patches way too large.

Yes they are over 25 MB and my gmail account does not support
attachments more than that size.

>
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

Re: [PATCH 00/12] Introduce new acpi/smbios qtests using biosbits

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 02:03:15PM +0530, Ani Sinha wrote:
> On Tue, Jun 28, 2022 at 1:39 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Mon, Jun 27, 2022 at 12:58:44PM +0530, Ani Sinha wrote:
> > > Biosbits is a software written by Josh Triplett that can be downloaded by
> > > visiting https://biosbits.org/. The github codebase can be found here:
> > > https://github.com/biosbits/bits/tree/master. It is a software that exercizes
> > > the bios components such as acpi and smbios tables directly through acpica
> > > bios interpreter (a freely available C based library written by Intel,
> > > downloadable from https://acpica.org/ and is included with biosbits) without an
> > > operating system getting involved in between.
> > > There are several advantages to directly testing the bios in a real physical
> > > machine or VM as opposed to indirectly discovering bios issues through the
> > > operating system. For one thing, the OSes tend to hide bios problems from the
> > > end user. The other is that we have more control of what we wanted to test
> > > and how by directly using acpica interpreter on top of the bios on a running
> > > system. More details on the inspiration for developing biosbits and its real
> > > life uses can be found in (a) and (b).
> > > This patchset contains QEMU qtests written in python that exercizes the QEMU
> > > bios components using biosbits and reports test failures.
> > >
> > > Details of each of the files added by this patchset are provided in the README
> > > file which is part of Patch 11. Every effort to contact Josh, through various
> > > means including email, twitter, linkedIn etc has failed. Hence, the changes to
> > > build biosbits with the newer compiler, upgrade acpica and other changes are
> > > currently maintained in a forked project in my personal github. We may want to
> > > maintain bits in a separate fork in a stable repository that is accessible by
> > > QEMU developers.
> > >
> > > The newly introduced qtest currently only run for x86_64 platform. They pass
> > > both when running make check on a baremetal box as well as from inside a vm.
> > >
> > > Thanks to Igor M for pointing me to this work.
> > >
> > > (a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > (b) https://www.youtube.com/watch?v=36QIepyUuhg
> > >
> > > Ani Sinha (12):
> > >   qtest: meson.build changes required to integrate python based qtests
> > >   acpi/tests/bits: add prebuilt bios bits zip archive
> > >   acpi/tests/bits: add prebuilt bits generated grub modules and scripts
> >
> > These two files didn't arrive on the mailing list, presumaby because
> > pre-built binaries made the patches way too large.
> 
> Yes they are over 25 MB and my gmail account does not support
> attachments more than that size.

Ok, with that kind of size, it is definitely not something we want to
be committing to git either, nor consuming via a submodule since the
latter would bloat the release tarballs too.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 00/12] Introduce new acpi/smbios qtests using biosbits

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:06:11AM +0100, Daniel P. Berrangé wrote:
> Ok, with that kind of size, it is definitely not something we want to
> be committing to git either,

Not to qemu git I think.

> nor consuming via a submodule since the
> latter would bloat the release tarballs too.

Hmm - why? We don't have to put the submodule in the tarball if we don't
want to. People consuming tarballs probably do not need these tests
anyway - just a basic smoketest is all that is needed.

-- 
MST

Re: [PATCH 00/12] Introduce new acpi/smbios qtests using biosbits

[Daniel P. Berrangé]

On Tue, Jun 28, 2022 at 06:16:06AM -0400, Michael S. Tsirkin wrote:
> On Tue, Jun 28, 2022 at 11:06:11AM +0100, Daniel P. Berrangé wrote:
> > Ok, with that kind of size, it is definitely not something we want to
> > be committing to git either,
> 
> Not to qemu git I think.
> 
> > nor consuming via a submodule since the
> > latter would bloat the release tarballs too.
> 
> Hmm - why? We don't have to put the submodule in the tarball if we don't
> want to. People consuming tarballs probably do not need these tests
> anyway - just a basic smoketest is all that is needed.

That feels very dubious. Upstream doesnt test every scenario that users
build & run in. Especially with Fedora rawhide we've often found problems
which upstream QEMU missed, particularly where new GCC releases have bugs
causing subtle mis-compilation of code.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 00/12] Introduce new acpi/smbios qtests using biosbits

[Michael S. Tsirkin]

On Tue, Jun 28, 2022 at 11:21:44AM +0100, Daniel P. Berrangé wrote:
> On Tue, Jun 28, 2022 at 06:16:06AM -0400, Michael S. Tsirkin wrote:
> > On Tue, Jun 28, 2022 at 11:06:11AM +0100, Daniel P. Berrangé wrote:
> > > Ok, with that kind of size, it is definitely not something we want to
> > > be committing to git either,
> > 
> > Not to qemu git I think.
> > 
> > > nor consuming via a submodule since the
> > > latter would bloat the release tarballs too.
> > 
> > Hmm - why? We don't have to put the submodule in the tarball if we don't
> > want to. People consuming tarballs probably do not need these tests
> > anyway - just a basic smoketest is all that is needed.
> 
> That feels very dubious. Upstream doesnt test every scenario that users
> build & run in. Especially with Fedora rawhide we've often found problems
> which upstream QEMU missed, particularly where new GCC releases have bugs
> causing subtle mis-compilation of code.
> 
> With regards,
> Daniel

IMHO these tests are not really useful for that.

What they do is verify that our ACPI tables are sane -
in addition to the manual review with disassembler we do currently.

We already have tests that verify that qemu generates expected ACPI
tables and that is enough for what you describe.

A miscompiled qemu will generate acpi tables that differ from expected
ones and the simple bit for bit test will fail.
No need to run acpipica within guest.




> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|