[Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Eduardo Habkost]

The following changes since commit 185b7ccc11354cbd69b6d53bf8d831dd964f6c88:

  Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20190607-2' into staging (2019-06-07 15:24:13 +0100)

are available in the Git repository at:

  git://github.com/ehabkost/qemu.git tags/python-next-pull-request

for you to fetch changes up to 8297719709f1339506d6da93ec69e6118ace8676:

  travis: Make check-acceptance job more verbose (2019-06-07 18:11:02 -0300)

----------------------------------------------------------------
Python queue, 2019-06-07

* New boot_linux_console test cases (Philippe Mathieu-Daudé)
* Deprecate Python 2 support (Eduardo Habkost)
* Require python3 >= 3.5 (Eduardo Habkost)
* Make check-acceptance Travis job more verbose (Eduardo Habkost)

----------------------------------------------------------------

Eduardo Habkost (3):
  Deprecate Python 2 support
  configure: Require python3 >= 3.5
  travis: Make check-acceptance job more verbose

Philippe Mathieu-Daudé (5):
  tests/boot_linux_console: Let extract_from_deb handle various
    compressions
  BootLinuxConsoleTest: Do not log empty lines
  BootLinuxConsoleTest: Test the SmartFusion2 board
  BootLinuxConsoleTest: Test nanoMIPS kernels on the I7200 CPU
  BootLinuxConsoleTest: Run kerneltests BusyBox on Malta

 configure                              |  13 ++-
 .travis.yml                            |   4 +-
 qemu-deprecated.texi                   |   8 ++
 tests/acceptance/boot_linux_console.py | 145 ++++++++++++++++++++++++-
 4 files changed, 162 insertions(+), 8 deletions(-)

-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 1/8] tests/boot_linux_console: Let extract_from_deb handle various compressions

[Eduardo Habkost]

From: Philippe Mathieu-Daudé <philmd@redhat.com>

Debian binary package format supports various compressions.

Per man deb(5):

  NAME
    deb - Debian binary package format

  FORMAT
    ...
    The third, last required member is named data.tar.  It contains the
    filesystem as a tar archive, either not compressed (supported since
    dpkg 1.10.24), or compressed with gzip (with .gz extension),
    xz (with .xz extension, supported since dpkg 1.15.6),
    bzip2 (with .bz2 extension, supported since dpkg 1.10.24) or
    lzma (with .lzma extension, supported since dpkg 1.13.25).

List the archive files to have the 3rd name with the correct extension.

The function avocado.utils.archive.extract() will handle the different
compression format for us.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20190312234541.2887-2-philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index d5c500ea30..6aa084e049 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -55,8 +55,9 @@ class BootLinuxConsole(Test):
         """
         cwd = os.getcwd()
         os.chdir(self.workdir)
-        process.run("ar x %s data.tar.gz" % deb)
-        archive.extract("data.tar.gz", self.workdir)
+        file_path = process.run("ar t %s" % deb).stdout_text.split()[2]
+        process.run("ar x %s %s" % (deb, file_path))
+        archive.extract(file_path, self.workdir)
         os.chdir(cwd)
         return self.workdir + path
 
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 2/8] Deprecate Python 2 support

[Eduardo Habkost]

Python 2 will reach end of life in January 1 2020.  Declare it as
deprecated.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20190503193721.18459-1-ehabkost@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
[ehabkost: print "warning:" in lowercase]
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 configure            | 8 ++++++++
 qemu-deprecated.texi | 8 ++++++++
 2 files changed, 16 insertions(+)

diff --git a/configure b/configure
index b091b82cb3..dcc732ff1e 100755
--- a/configure
+++ b/configure
@@ -6506,6 +6506,14 @@ if test "$supported_os" = "no"; then
     echo "us upstream at qemu-devel@nongnu.org."
 fi
 
+# Note that if the Python conditional here evaluates True we will exit
+# with status 1 which is a shell 'false' value.
+if ! $python -c 'import sys; sys.exit(sys.version_info < (3,0))'; then
+  echo
+  echo "warning: Python 2 support is deprecated" >&2
+  echo "warning: Python 3 will be required for building future versions of QEMU" >&2
+fi
+
 config_host_mak="config-host.mak"
 
 echo "# Automatically generated by configure - do not modify" >config-all-disas.mak
diff --git a/qemu-deprecated.texi b/qemu-deprecated.texi
index 50292d820b..65411c130e 100644
--- a/qemu-deprecated.texi
+++ b/qemu-deprecated.texi
@@ -237,3 +237,11 @@ Note that if you are exposing the export via /dev/nbd0, it is easier
 to just export the entire image and then mount only /dev/nbd0p1 than
 it is to reinvoke @command{qemu-nbd -c /dev/nbd0} limited to just a
 subset of the image.
+
+@section Build system
+
+@subsection Python 2 support (since 4.1.0)
+
+In the future, QEMU will require Python 3 to be available at
+build time.  Support for Python 2 in scripts shipped with QEMU
+is deprecated.
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 3/8] configure: Require python3 >= 3.5

[Eduardo Habkost]

The oldest python3 version in distros that will be supported by
QEMU 4.1 is 3.5.3 (the one in Debian Stretch).  Error out if
running python3 < 3.5.

We have a .travis.yml job configured to use Python 3.4.  Change
it to use Python 3.5.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20190508182339.22447-1-ehabkost@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 configure   | 5 +++--
 .travis.yml | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/configure b/configure
index dcc732ff1e..9dc960824f 100755
--- a/configure
+++ b/configure
@@ -1852,8 +1852,9 @@ fi
 
 # Note that if the Python conditional here evaluates True we will exit
 # with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (2,7))'; then
-  error_exit "Cannot use '$python', Python 2 >= 2.7 or Python 3 is required." \
+if ! $python -c 'import sys; sys.exit(sys.version_info < (2,7) or \
+                                      (3,0) <= sys.version_info < (3,5))'; then
+  error_exit "Cannot use '$python', Python 2 >= 2.7 or Python 3 >= 3.5 is required." \
       "Use --python=/path/to/python to specify a supported Python."
 fi
 
diff --git a/.travis.yml b/.travis.yml
index b053a836a3..75e017a5cf 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -212,7 +212,7 @@ matrix:
         - CONFIG="--target-list=x86_64-softmmu"
       language: python
       python:
-        - "3.4"
+        - "3.5"
 
 
     - env:
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 4/8] BootLinuxConsoleTest: Do not log empty lines

[Eduardo Habkost]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Avoid to log empty lines in console debug logs.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20190520220635.10961-2-f4bug@amsat.org>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Acked-by: Aleksandar Markovic <amarkovic@wavecomp.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 6aa084e049..e86e66900a 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -37,8 +37,10 @@ class BootLinuxConsole(Test):
         console = self.vm.console_socket.makefile()
         console_logger = logging.getLogger('console')
         while True:
-            msg = console.readline()
-            console_logger.debug(msg.strip())
+            msg = console.readline().strip()
+            if not msg:
+                continue
+            console_logger.debug(msg)
             if success_message in msg:
                 break
             if failure_message in msg:
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 5/8] BootLinuxConsoleTest: Test the SmartFusion2 board

[Eduardo Habkost]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Similar to the x86_64/pc test, it boots a Linux kernel on an
Emcraft board and verify the serial is working.

If ARM is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:arm" tags.

Alternatively, this test can be run using:

  $ avocado run -t arch:arm tests/acceptance
  $ avocado run -t machine:emcraft_sf2 tests/acceptance

Based on the recommended test setup from Subbaraya Sundeep:
https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg03810.html

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20190520220635.10961-3-f4bug@amsat.org>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 27 ++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index e86e66900a..e6b5ed5dda 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -179,6 +179,33 @@ class BootLinuxConsole(Test):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
+    def test_arm_emcraft_sf2(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:emcraft_sf2
+        :avocado: tags=endian:little
+        """
+        uboot_url = ('https://raw.githubusercontent.com/'
+                     'Subbaraya-Sundeep/qemu-test-binaries/'
+                     'fa030bd77a014a0b8e360d3b7011df89283a2f0b/u-boot')
+        uboot_hash = 'abba5d9c24cdd2d49cdc2a8aa92976cf20737eff'
+        uboot_path = self.fetch_asset(uboot_url, asset_hash=uboot_hash)
+        spi_url = ('https://raw.githubusercontent.com/'
+                   'Subbaraya-Sundeep/qemu-test-binaries/'
+                   'fa030bd77a014a0b8e360d3b7011df89283a2f0b/spi.bin')
+        spi_hash = '85f698329d38de63aea6e884a86fbde70890a78a'
+        spi_path = self.fetch_asset(spi_url, asset_hash=spi_hash)
+
+        self.vm.set_machine('emcraft-sf2')
+        self.vm.set_console()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
+        self.vm.add_args('-kernel', uboot_path,
+                         '-append', kernel_command_line,
+                         '-drive', 'file=' + spi_path + ',if=mtd,format=raw',
+                         '-no-reboot')
+        self.vm.launch()
+        self.wait_for_console_pattern('init started: BusyBox')
+
     def test_s390x_s390_ccw_virtio(self):
         """
         :avocado: tags=arch:s390x
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 6/8] BootLinuxConsoleTest: Test nanoMIPS kernels on the I7200 CPU

[Eduardo Habkost]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Similar to the x86_64/pc test, it boots a Linux kernel on a Malta
machine and verify the serial is working.

Use the documentation added in commit f7d257cb4a17 to test
nanoMIPS kernels and the I7200 CPU.

This test can be run using:

  $ avocado --show=console run -t arch:mipsel tests/acceptance/boot_linux_console.py
  console: [    0.000000] Linux version 4.15.18-00432-gb2eb9a8b (emubuild@mipscs563) (gcc version 6.3.0 (Codescape GNU Tools 2018.04-02 for nanoMIPS Linux)) #1 SMP Wed Jun 27 11:10:08 PDT 2018
  console: [    0.000000] GCRs appear to have been moved (expected them at 0x1fbf8000)!
  console: [    0.000000] GCRs appear to have been moved (expected them at 0x1fbf8000)!
  console: [    0.000000] CPU0 revision is: 00010000 (MIPS GENERIC QEMU)
  console: [    0.000000] MIPS: machine is mti,malta
  console: [    0.000000] Determined physical RAM map:
  console: [    0.000000]  memory: 08000000 @ 00000000 (usable)
  console: [    0.000000] earlycon: ns16550a0 at I/O port 0x3f8 (options '38400n8')
  console: [    0.000000] bootconsole [ns16550a0] enabled
  console: [    0.000000] User-defined physical RAM map:
  console: [    0.000000]  memory: 10000000 @ 00000000 (usable)
  console: [    0.000000] Initrd not found or empty - disabling initrd
  console: [    0.000000] MIPS CPS SMP unable to proceed without a CM
  console: [    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
  console: [    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
  console: [    0.000000] This processor doesn't support highmem. -262144k highmem ignored
  console: [    0.000000] Zone ranges:
  console: [    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
  console: [    0.000000]   HighMem  empty
  console: [    0.000000] Movable zone start for each node
  console: [    0.000000] Early memory node ranges
  console: [    0.000000]   node   0: [mem 0x0000000000000000-0x000000000fffffff]
  console: [    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000000fffffff]
  console: [    0.000000] random: get_random_bytes called from start_kernel+0x60/0x2f0 with crng_init=0
  console: [    0.000000] percpu: Embedded 16 pages/cpu @(ptrval) s36620 r8192 d20724 u65536
  console: [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64960
  console: [    0.000000] Kernel command line: printk.time=0 mem=256m@@0x0 console=ttyS0 earlycon

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20190520231910.12184-3-f4bug@amsat.org>
Acked-by: Aleksandar Markovic <amarkovic@wavecomp.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 58 ++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index e6b5ed5dda..5d8380a23b 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -10,6 +10,8 @@
 
 import os
 import logging
+import lzma
+import shutil
 
 from avocado_qemu import Test
 from avocado.utils import process
@@ -138,6 +140,62 @@ class BootLinuxConsole(Test):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
+    def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
+        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        kernel_path = self.workdir + "kernel"
+        with lzma.open(kernel_path_xz, 'rb') as f_in:
+            with open(kernel_path, 'wb') as f_out:
+                shutil.copyfileobj(f_in, f_out)
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
+                               + 'mem=256m@@0x0 '
+                               + 'console=ttyS0')
+        self.vm.add_args('-no-reboot',
+                         '-cpu', 'I7200',
+                         '-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
+
+    def test_mips_malta32el_nanomips_4k(self):
+        """
+        :avocado: tags=arch:mipsel
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:little
+        """
+        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+                      'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
+                      'generic_nano32r6el_page4k.xz')
+        kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
+        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+
+    def test_mips_malta32el_nanomips_16k_up(self):
+        """
+        :avocado: tags=arch:mipsel
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:little
+        """
+        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+                      'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
+                      'generic_nano32r6el_page16k_up.xz')
+        kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
+        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+
+    def test_mips_malta32el_nanomips_64k_dbg(self):
+        """
+        :avocado: tags=arch:mipsel
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:little
+        """
+        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+                      'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
+                      'generic_nano32r6el_page64k_dbg.xz')
+        kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
+        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+
     def test_aarch64_virt(self):
         """
         :avocado: tags=arch:aarch64
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 7/8] BootLinuxConsoleTest: Run kerneltests BusyBox on Malta

[Eduardo Habkost]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

This tests boots a Linux kernel on a Malta machine up to a
busybox shell on the serial console. Few commands are executed
before halting the machine (via reboot).

We use the initrd cpio image from the kerneltests project:
https://kerneltests.org/

If MIPS is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:mips" tags.

Alternatively, this test can be run using:

  $ avocado --show=console run -t arch:mips tests/acceptance/boot_linux_console.py
  [...]
  console: Boot successful.
  [...]
  console: / # uname -a
  console: Linux buildroot 4.5.0-2-4kc-malta #1 Debian 4.5.5-1 (2016-05-29) mips GNU/Linux
  console: / # reboot
  console: / # reboot: Restarting system

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20190520231910.12184-4-f4bug@amsat.org>
Acked-by: Aleksandar Markovic <amarkovic@wavecomp.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 49 ++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 5d8380a23b..32159503e9 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -11,6 +11,7 @@
 import os
 import logging
 import lzma
+import gzip
 import shutil
 
 from avocado_qemu import Test
@@ -49,6 +50,11 @@ class BootLinuxConsole(Test):
                 fail = 'Failure message found in console: %s' % failure_message
                 self.fail(fail)
 
+    def exec_command_and_wait_for_pattern(self, command, success_message):
+        command += '\n'
+        self.vm.console_socket.sendall(command.encode())
+        self.wait_for_console_pattern(success_message)
+
     def extract_from_deb(self, deb, path):
         """
         Extracts a file from a deb package into the test workdir
@@ -140,6 +146,49 @@ class BootLinuxConsole(Test):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
+    def test_mips_malta_cpio(self):
+        """
+        :avocado: tags=arch:mips
+        :avocado: tags=machine:malta
+        :avocado: tags=endian:big
+        """
+        deb_url = ('http://snapshot.debian.org/archive/debian/'
+                   '20160601T041800Z/pool/main/l/linux/'
+                   'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
+        deb_hash = 'a3c84f3e88b54e06107d65a410d1d1e8e0f340f8'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        kernel_path = self.extract_from_deb(deb_path,
+                                            '/boot/vmlinux-4.5.0-2-4kc-malta')
+        initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
+                      '8584a59ed9e5eb5ee7ca91f6d74bbb06619205b8/rootfs/'
+                      'mips/rootfs.cpio.gz')
+        initrd_hash = 'bf806e17009360a866bf537f6de66590de349a99'
+        initrd_path_gz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
+        initrd_path = self.workdir + "rootfs.cpio"
+
+        with gzip.open(initrd_path_gz, 'rb') as f_in:
+            with open(initrd_path, 'wb') as f_out:
+                shutil.copyfileobj(f_in, f_out)
+
+        self.vm.set_machine('malta')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
+                               + 'console=ttyS0 console=tty '
+                               + 'rdinit=/sbin/init noreboot')
+        self.vm.add_args('-kernel', kernel_path,
+                         '-initrd', initrd_path,
+                         '-append', kernel_command_line,
+                         '-no-reboot')
+        self.vm.launch()
+        self.wait_for_console_pattern('Boot successful.')
+
+        self.exec_command_and_wait_for_pattern('cat /proc/cpuinfo',
+                                               'BogoMIPS')
+        self.exec_command_and_wait_for_pattern('uname -a',
+                                               'Debian')
+        self.exec_command_and_wait_for_pattern('reboot',
+                                               'reboot: Restarting system')
+
     def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
         kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
         kernel_path = self.workdir + "kernel"
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 8/8] travis: Make check-acceptance job more verbose

[Eduardo Habkost]

It will help us debug issues when tests fail.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 75e017a5cf..82c74673e1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -225,7 +225,7 @@ matrix:
     # Acceptance (Functional) tests
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="make AVOCADO_SHOW=test check-acceptance"
       addons:
         apt:
           packages:
-- 
2.18.0.rc1.1.g3f1ff2140

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Peter Maydell]

On Fri, 7 Jun 2019 at 22:16, Eduardo Habkost <ehabkost@redhat.com> wrote:
>
> The following changes since commit 185b7ccc11354cbd69b6d53bf8d831dd964f6c88:
>
>   Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20190607-2' into staging (2019-06-07 15:24:13 +0100)
>
> are available in the Git repository at:
>
>   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
>
> for you to fetch changes up to 8297719709f1339506d6da93ec69e6118ace8676:
>
>   travis: Make check-acceptance job more verbose (2019-06-07 18:11:02 -0300)
>
> ----------------------------------------------------------------
> Python queue, 2019-06-07
>
> * New boot_linux_console test cases (Philippe Mathieu-Daudé)
> * Deprecate Python 2 support (Eduardo Habkost)
> * Require python3 >= 3.5 (Eduardo Habkost)
> * Make check-acceptance Travis job more verbose (Eduardo Habkost)
>
> ----------------------------------------------------------------

Hi. This fails to build on one of my buildtest machines:

ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
       Use --python=/path/to/python to specify a supported Python.

The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
box; it's one of the gcc compile farm machines so upgrades to its
OS are not really under my control.)

The configure check also spits out deprecation warnings for
the NetBSD/FreeBSD/OpenBSD tests/vm configurations. It would be nice
to get those updated.

thanks
-- PMM

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Eduardo Habkost]

On Mon, Jun 10, 2019 at 01:58:50PM +0100, Peter Maydell wrote:
> On Fri, 7 Jun 2019 at 22:16, Eduardo Habkost <ehabkost@redhat.com> wrote:
> >
> > The following changes since commit 185b7ccc11354cbd69b6d53bf8d831dd964f6c88:
> >
> >   Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20190607-2' into staging (2019-06-07 15:24:13 +0100)
> >
> > are available in the Git repository at:
> >
> >   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
> >
> > for you to fetch changes up to 8297719709f1339506d6da93ec69e6118ace8676:
> >
> >   travis: Make check-acceptance job more verbose (2019-06-07 18:11:02 -0300)
> >
> > ----------------------------------------------------------------
> > Python queue, 2019-06-07
> >
> > * New boot_linux_console test cases (Philippe Mathieu-Daudé)
> > * Deprecate Python 2 support (Eduardo Habkost)
> > * Require python3 >= 3.5 (Eduardo Habkost)
> > * Make check-acceptance Travis job more verbose (Eduardo Habkost)
> >
> > ----------------------------------------------------------------
> 
> Hi. This fails to build on one of my buildtest machines:
> 
> ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
>        Use --python=/path/to/python to specify a supported Python.
> 
> The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> box; it's one of the gcc compile farm machines so upgrades to its
> OS are not really under my control.)

Ubuntu 16.04 LTS (Xenial) was released in April 2016.  Doesn't
that mean Trusty is not a supported build platform since April
2018?

> 
> The configure check also spits out deprecation warnings for
> the NetBSD/FreeBSD/OpenBSD tests/vm configurations. It would be nice
> to get those updated.

I'll take a look.

-- 
Eduardo

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Peter Maydell]

On Mon, 10 Jun 2019 at 14:11, Eduardo Habkost <ehabkost@redhat.com> wrote:
>
> On Mon, Jun 10, 2019 at 01:58:50PM +0100, Peter Maydell wrote:
> > Hi. This fails to build on one of my buildtest machines:
> >
> > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> >        Use --python=/path/to/python to specify a supported Python.
> >
> > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > box; it's one of the gcc compile farm machines so upgrades to its
> > OS are not really under my control.)
>
> Ubuntu 16.04 LTS (Xenial) was released in April 2016.  Doesn't
> that mean Trusty is not a supported build platform since April
> 2018?

Possibly, but as I say the gcc compile farm is what it is.
Regardless of the strict letter of the deprecation policy,
when we start running into issues with the set of build test
machines I tend to feel we may be being a bit over-hasty in
deprecating things.

PS: re the BSD VMs, you probably want to start with Alex's
recent update which did a fairly major overhaul there (he
sent a pullreq but it didn't pass build tests so isn't merged yet).

thanks
-- PMM

[Qemu-devel] Ubuntu Trusty as supported build platform (was Re: [PULL 0/8] Python queue, 2019-06-07)

[Eduardo Habkost]

On Mon, Jun 10, 2019 at 02:21:57PM +0100, Peter Maydell wrote:
> On Mon, 10 Jun 2019 at 14:11, Eduardo Habkost <ehabkost@redhat.com> wrote:
> >
> > On Mon, Jun 10, 2019 at 01:58:50PM +0100, Peter Maydell wrote:
> > > Hi. This fails to build on one of my buildtest machines:
> > >
> > > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> > >        Use --python=/path/to/python to specify a supported Python.
> > >
> > > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > > box; it's one of the gcc compile farm machines so upgrades to its
> > > OS are not really under my control.)
> >
> > Ubuntu 16.04 LTS (Xenial) was released in April 2016.  Doesn't
> > that mean Trusty is not a supported build platform since April
> > 2018?
> 
> Possibly, but as I say the gcc compile farm is what it is.
> Regardless of the strict letter of the deprecation policy,
> when we start running into issues with the set of build test
> machines I tend to feel we may be being a bit over-hasty in
> deprecating things.

I understand this as a request to make Trusty a supported build
platform.  Can we please update the documentation to reflect
that, then?

-- 
Eduardo

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Daniel P. Berrangé]

On Mon, Jun 10, 2019 at 02:21:57PM +0100, Peter Maydell wrote:
> On Mon, 10 Jun 2019 at 14:11, Eduardo Habkost <ehabkost@redhat.com> wrote:
> >
> > On Mon, Jun 10, 2019 at 01:58:50PM +0100, Peter Maydell wrote:
> > > Hi. This fails to build on one of my buildtest machines:
> > >
> > > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> > >        Use --python=/path/to/python to specify a supported Python.
> > >
> > > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > > box; it's one of the gcc compile farm machines so upgrades to its
> > > OS are not really under my control.)
> >
> > Ubuntu 16.04 LTS (Xenial) was released in April 2016.  Doesn't
> > that mean Trusty is not a supported build platform since April
> > 2018?
> 
> Possibly, but as I say the gcc compile farm is what it is.
> Regardless of the strict letter of the deprecation policy,
> when we start running into issues with the set of build test
> machines I tend to feel we may be being a bit over-hasty in
> deprecating things.

There's been two new LTS releases of Ubuntu since then
with Xenial and Bionic, so IMHO, it is pretty reasonable
to want to drop Trusty rather than continuing to spend time
on software versions from 2014 and before. Both our Travis
and Docker setups use Xenial as minimum and the number of
our developers stuck using Trusty is likely negligible.

What gap is your gcc compile farm filling & can we find an
alternate way to address that gap that's viable ?

Does the gcc compile farm include Docker to let us run the
build in a container from the compile farm. Or can we use
a Docker container on a modern x86_64 host to test the same
thing. We have docker containers for all Linux OS we need
to target, and docker containers for non-x86_64 arches that
cross compile.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Peter Maydell]

On Mon, 10 Jun 2019 at 18:12, Daniel P. Berrangé <berrange@redhat.com> wrote:
> There's been two new LTS releases of Ubuntu since then
> with Xenial and Bionic, so IMHO, it is pretty reasonable
> to want to drop Trusty rather than continuing to spend time
> on software versions from 2014 and before. Both our Travis
> and Docker setups use Xenial as minimum and the number of
> our developers stuck using Trusty is likely negligible.
>
> What gap is your gcc compile farm filling & can we find an
> alternate way to address that gap that's viable ?

In this case, it's the aarch64 host. I can probably
find something else to run this on, but it's not going
to happen immediately.

> docker containers for non-x86_64 arches that cross compile.

You need to actually run the tests, so merely cross
compiling doesn't suffice. (Running 'make check' catches
a fair number of bugs.)

I've asked the gcc farm admins if they have plans for an
OS upgrade on those boxes.

thanks
-- PMM

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Daniel P. Berrangé]

On Mon, Jun 10, 2019 at 06:15:46PM +0100, Peter Maydell wrote:
> On Mon, 10 Jun 2019 at 18:12, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > There's been two new LTS releases of Ubuntu since then
> > with Xenial and Bionic, so IMHO, it is pretty reasonable
> > to want to drop Trusty rather than continuing to spend time
> > on software versions from 2014 and before. Both our Travis
> > and Docker setups use Xenial as minimum and the number of
> > our developers stuck using Trusty is likely negligible.
> >
> > What gap is your gcc compile farm filling & can we find an
> > alternate way to address that gap that's viable ?
> 
> In this case, it's the aarch64 host. I can probably
> find something else to run this on, but it's not going
> to happen immediately.

Do you have access to any machine in the compile farm or is access
granted on a per-machine basis ?   If i'm reading this page right:

  https://cfarm.tetaneutral.net/machines/list/

there is now one aarch64 machine (gcc117) that is running
Debian 9 / Stretch and another (gcc118) with OpenSUSE Leap 15.
In terms of OS version at least, either of those could be viable
for QEMU, if it is possible for you to access them.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Peter Maydell]

On Mon, 10 Jun 2019 at 18:30, Daniel P. Berrangé <berrange@redhat.com> wrote:
> Do you have access to any machine in the compile farm or is access
> granted on a per-machine basis ?   If i'm reading this page right:
>
>   https://cfarm.tetaneutral.net/machines/list/
>
> there is now one aarch64 machine (gcc117) that is running
> Debian 9 / Stretch and another (gcc118) with OpenSUSE Leap 15.
> In terms of OS version at least, either of those could be viable
> for QEMU, if it is possible for you to access them.

They're all available, but those machines have less RAM and
fewer CPUs; I think the A1100 is also not as powerful as the X-Gene.
I'm pretty sure Linaro has access to something better than that,
but it will take me a bit to find out and set up to use it.
(I do actually have a Mustang board myself but it's running
the aarch32 builds and I'd prefer not to make it also do the
aarch64 builds at the same time.)

thanks
-- PMM

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Daniel P. Berrangé]

On Mon, Jun 10, 2019 at 06:38:05PM +0100, Peter Maydell wrote:
> On Mon, 10 Jun 2019 at 18:30, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > Do you have access to any machine in the compile farm or is access
> > granted on a per-machine basis ?   If i'm reading this page right:
> >
> >   https://cfarm.tetaneutral.net/machines/list/
> >
> > there is now one aarch64 machine (gcc117) that is running
> > Debian 9 / Stretch and another (gcc118) with OpenSUSE Leap 15.
> > In terms of OS version at least, either of those could be viable
> > for QEMU, if it is possible for you to access them.
> 
> They're all available, but those machines have less RAM and
> fewer CPUs; I think the A1100 is also not as powerful as the X-Gene.
> I'm pretty sure Linaro has access to something better than that,
> but it will take me a bit to find out and set up to use it.
> (I do actually have a Mustang board myself but it's running
> the aarch32 builds and I'd prefer not to make it also do the
> aarch64 builds at the same time.)

Ok, thanks for the explanation.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PULL 8/8] travis: Make check-acceptance job more verbose

[Wainer dos Santos Moschetta]

Hi Eduardo,

On 06/07/2019 06:15 PM, Eduardo Habkost wrote:
> It will help us debug issues when tests fail.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>   .travis.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 75e017a5cf..82c74673e1 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -225,7 +225,7 @@ matrix:
>       # Acceptance (Functional) tests
>       - env:
>           - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
> -        - TEST_CMD="make check-acceptance"
> +        - TEST_CMD="make AVOCADO_SHOW=test check-acceptance"

Please see Cleber's "[PATCH 1/8] Travis: print acceptance tests logs in 
case of job failure". His patch will print the job.log content only and 
only if some test failed, which IMO is preferable over making avocado 
more verbose as you proposed.

Thanks!

- Wainer

>         addons:
>           apt:
>             packages:

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Peter Maydell]

On Mon, 10 Jun 2019 at 13:58, Peter Maydell <peter.maydell@linaro.org> wrote:
> Hi. This fails to build on one of my buildtest machines:
>
> ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
>        Use --python=/path/to/python to specify a supported Python.
>
> The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> box; it's one of the gcc compile farm machines so upgrades to its
> OS are not really under my control.)

Rereading this, I realise that either the check or the error
message is wrong here. The machine has 2.7.6, which satisfies
"python 2 >= 2.7", so we should be OK to build. The bug
seems to be that we say "prefer python3 over plain python
on python2" early, but don't revisit that decision if the
python3 we found isn't actually good enough for us.

(I am also setting up to do builds on a different aarch64 box
which has a newer python3.)

thanks
-- PMM

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Eduardo Habkost]

On Tue, Jun 11, 2019 at 04:50:34PM +0100, Peter Maydell wrote:
> On Mon, 10 Jun 2019 at 13:58, Peter Maydell <peter.maydell@linaro.org> wrote:
> > Hi. This fails to build on one of my buildtest machines:
> >
> > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> >        Use --python=/path/to/python to specify a supported Python.
> >
> > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > box; it's one of the gcc compile farm machines so upgrades to its
> > OS are not really under my control.)
> 
> Rereading this, I realise that either the check or the error
> message is wrong here. The machine has 2.7.6, which satisfies
> "python 2 >= 2.7", so we should be OK to build. The bug
> seems to be that we say "prefer python3 over plain python
> on python2" early, but don't revisit that decision if the
> python3 we found isn't actually good enough for us.

Right.  The error message is technically correct, but misleading.
python3 is too old, but python2 would work.

We can make configure not use python3 by default if it's too old,
and fall back to python2 in this case.

-- 
Eduardo

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Peter Maydell]

On Tue, 11 Jun 2019 at 17:03, Eduardo Habkost <ehabkost@redhat.com> wrote:
>
> On Tue, Jun 11, 2019 at 04:50:34PM +0100, Peter Maydell wrote:
> > On Mon, 10 Jun 2019 at 13:58, Peter Maydell <peter.maydell@linaro.org> wrote:
> > > Hi. This fails to build on one of my buildtest machines:
> > >
> > > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> > >        Use --python=/path/to/python to specify a supported Python.
> > >
> > > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > > box; it's one of the gcc compile farm machines so upgrades to its
> > > OS are not really under my control.)
> >
> > Rereading this, I realise that either the check or the error
> > message is wrong here. The machine has 2.7.6, which satisfies
> > "python 2 >= 2.7", so we should be OK to build. The bug
> > seems to be that we say "prefer python3 over plain python
> > on python2" early, but don't revisit that decision if the
> > python3 we found isn't actually good enough for us.
>
> Right.  The error message is technically correct, but misleading.
> python3 is too old, but python2 would work.
>
> We can make configure not use python3 by default if it's too old,
> and fall back to python2 in this case.

Sounds good. Since I have now managed to get my alternate
aarch64 box set up, how about I apply this pullreq and you
send a followup patch which does the fallback to python/python2 ?

thanks
-- PMM

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Eduardo Habkost]

On Tue, Jun 11, 2019 at 05:07:55PM +0100, Peter Maydell wrote:
> On Tue, 11 Jun 2019 at 17:03, Eduardo Habkost <ehabkost@redhat.com> wrote:
> >
> > On Tue, Jun 11, 2019 at 04:50:34PM +0100, Peter Maydell wrote:
> > > On Mon, 10 Jun 2019 at 13:58, Peter Maydell <peter.maydell@linaro.org> wrote:
> > > > Hi. This fails to build on one of my buildtest machines:
> > > >
> > > > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> > > >        Use --python=/path/to/python to specify a supported Python.
> > > >
> > > > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > > > box; it's one of the gcc compile farm machines so upgrades to its
> > > > OS are not really under my control.)
> > >
> > > Rereading this, I realise that either the check or the error
> > > message is wrong here. The machine has 2.7.6, which satisfies
> > > "python 2 >= 2.7", so we should be OK to build. The bug
> > > seems to be that we say "prefer python3 over plain python
> > > on python2" early, but don't revisit that decision if the
> > > python3 we found isn't actually good enough for us.
> >
> > Right.  The error message is technically correct, but misleading.
> > python3 is too old, but python2 would work.
> >
> > We can make configure not use python3 by default if it's too old,
> > and fall back to python2 in this case.
> 
> Sounds good. Since I have now managed to get my alternate
> aarch64 box set up, how about I apply this pullreq and you
> send a followup patch which does the fallback to python/python2 ?

I will remove the python2/python3 patches and send a new pull
request.

-- 
Eduardo

[Qemu-devel] Python 2 in tests/vm (was Re: [PULL 0/8] Python queue, 2019-06-07)

[Eduardo Habkost]

On Mon, Jun 10, 2019 at 01:58:50PM +0100, Peter Maydell wrote:
> On Fri, 7 Jun 2019 at 22:16, Eduardo Habkost <ehabkost@redhat.com> wrote:
> >
> > The following changes since commit 185b7ccc11354cbd69b6d53bf8d831dd964f6c88:
> >
> >   Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20190607-2' into staging (2019-06-07 15:24:13 +0100)
> >
> > are available in the Git repository at:
> >
> >   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
> >
> > for you to fetch changes up to 8297719709f1339506d6da93ec69e6118ace8676:
> >
> >   travis: Make check-acceptance job more verbose (2019-06-07 18:11:02 -0300)
> >
> > ----------------------------------------------------------------
> > Python queue, 2019-06-07
> >
> > * New boot_linux_console test cases (Philippe Mathieu-Daudé)
> > * Deprecate Python 2 support (Eduardo Habkost)
> > * Require python3 >= 3.5 (Eduardo Habkost)
> > * Make check-acceptance Travis job more verbose (Eduardo Habkost)
> >
> > ----------------------------------------------------------------
> 
> Hi. This fails to build on one of my buildtest machines:
> 
> ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
>        Use --python=/path/to/python to specify a supported Python.
> 
> The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> box; it's one of the gcc compile farm machines so upgrades to its
> OS are not really under my control.)
> 
> The configure check also spits out deprecation warnings for
> the NetBSD/FreeBSD/OpenBSD tests/vm configurations. It would be nice
> to get those updated.

CCing the test/vm maintainers.

Fam, Alex, are you able to fix this and create new BSD VM images
with Python 3 available?  I thought the VM image configurations
were stored in the source tree, but they are downloaded from
download.patchew.org.

-- 
Eduardo

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Kevin Wolf]

Am 11.06.2019 um 19:12 hat Eduardo Habkost geschrieben:
> On Tue, Jun 11, 2019 at 05:07:55PM +0100, Peter Maydell wrote:
> > On Tue, 11 Jun 2019 at 17:03, Eduardo Habkost <ehabkost@redhat.com> wrote:
> > >
> > > On Tue, Jun 11, 2019 at 04:50:34PM +0100, Peter Maydell wrote:
> > > > On Mon, 10 Jun 2019 at 13:58, Peter Maydell <peter.maydell@linaro.org> wrote:
> > > > > Hi. This fails to build on one of my buildtest machines:
> > > > >
> > > > > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> > > > >        Use --python=/path/to/python to specify a supported Python.
> > > > >
> > > > > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > > > > box; it's one of the gcc compile farm machines so upgrades to its
> > > > > OS are not really under my control.)
> > > >
> > > > Rereading this, I realise that either the check or the error
> > > > message is wrong here. The machine has 2.7.6, which satisfies
> > > > "python 2 >= 2.7", so we should be OK to build. The bug
> > > > seems to be that we say "prefer python3 over plain python
> > > > on python2" early, but don't revisit that decision if the
> > > > python3 we found isn't actually good enough for us.
> > >
> > > Right.  The error message is technically correct, but misleading.
> > > python3 is too old, but python2 would work.
> > >
> > > We can make configure not use python3 by default if it's too old,
> > > and fall back to python2 in this case.
> > 
> > Sounds good. Since I have now managed to get my alternate
> > aarch64 box set up, how about I apply this pullreq and you
> > send a followup patch which does the fallback to python/python2 ?
> 
> I will remove the python2/python3 patches and send a new pull
> request.

What is the plan forward with this? Are the patches dropped for good?

I think the plan was to drop Python 2 after QEMU 4.2, and then it
becomes really relevant what our minimum Python 3 version is. We've just
had another Python version discussion in the context of iotests (John
suggested using function annotations, but these are >= 3.5 only).

Also, the fallback to Python 2 obviously makes no sense any more then,
so maybe it's not that important to add for a single QEMU release?

As Peter seems to have indicated above that he found a replacement for
the test machine with an OS that isn't out of support, can we just
revive this patch as it is?

Kevin

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Eduardo Habkost]

On Tue, Sep 17, 2019 at 03:57:26PM +0200, Kevin Wolf wrote:
> Am 11.06.2019 um 19:12 hat Eduardo Habkost geschrieben:
> > On Tue, Jun 11, 2019 at 05:07:55PM +0100, Peter Maydell wrote:
> > > On Tue, 11 Jun 2019 at 17:03, Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > >
> > > > On Tue, Jun 11, 2019 at 04:50:34PM +0100, Peter Maydell wrote:
> > > > > On Mon, 10 Jun 2019 at 13:58, Peter Maydell <peter.maydell@linaro.org> wrote:
> > > > > > Hi. This fails to build on one of my buildtest machines:
> > > > > >
> > > > > > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> > > > > >        Use --python=/path/to/python to specify a supported Python.
> > > > > >
> > > > > > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > > > > > box; it's one of the gcc compile farm machines so upgrades to its
> > > > > > OS are not really under my control.)
> > > > >
> > > > > Rereading this, I realise that either the check or the error
> > > > > message is wrong here. The machine has 2.7.6, which satisfies
> > > > > "python 2 >= 2.7", so we should be OK to build. The bug
> > > > > seems to be that we say "prefer python3 over plain python
> > > > > on python2" early, but don't revisit that decision if the
> > > > > python3 we found isn't actually good enough for us.
> > > >
> > > > Right.  The error message is technically correct, but misleading.
> > > > python3 is too old, but python2 would work.
> > > >
> > > > We can make configure not use python3 by default if it's too old,
> > > > and fall back to python2 in this case.
> > > 
> > > Sounds good. Since I have now managed to get my alternate
> > > aarch64 box set up, how about I apply this pullreq and you
> > > send a followup patch which does the fallback to python/python2 ?
> > 
> > I will remove the python2/python3 patches and send a new pull
> > request.
> 
> What is the plan forward with this? Are the patches dropped for good?
> 
> I think the plan was to drop Python 2 after QEMU 4.2, and then it
> becomes really relevant what our minimum Python 3 version is. We've just
> had another Python version discussion in the context of iotests (John
> suggested using function annotations, but these are >= 3.5 only).
> 
> Also, the fallback to Python 2 obviously makes no sense any more then,
> so maybe it's not that important to add for a single QEMU release?
> 
> As Peter seems to have indicated above that he found a replacement for
> the test machine with an OS that isn't out of support, can we just
> revive this patch as it is?

My plan is to remove Python 2 support in QEMU 4.2 (making the
fallback to Python 2 a non-issue), and require Python >= 3.5.

Now, even if my plan is rejected and we keep supporting Python 2
when building QEMU 4.2, my suggestion for the iotest maintainers
is to make it require Python 3.5+ immediately, just like we do
for tests/acceptance.  I don't see why we should keep wasting our
energy supporting ancient Python versions in a test suite that is
not a requirement for building QEMU.

-- 
Eduardo

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[John Snow]

On 9/17/19 5:48 PM, Eduardo Habkost wrote:
> On Tue, Sep 17, 2019 at 03:57:26PM +0200, Kevin Wolf wrote:
>> Am 11.06.2019 um 19:12 hat Eduardo Habkost geschrieben:
>>> On Tue, Jun 11, 2019 at 05:07:55PM +0100, Peter Maydell wrote:
>>>> On Tue, 11 Jun 2019 at 17:03, Eduardo Habkost <ehabkost@redhat.com> wrote:
>>>>>
>>>>> On Tue, Jun 11, 2019 at 04:50:34PM +0100, Peter Maydell wrote:
>>>>>> On Mon, 10 Jun 2019 at 13:58, Peter Maydell <peter.maydell@linaro.org> wrote:
>>>>>>> Hi. This fails to build on one of my buildtest machines:
>>>>>>>
>>>>>>> ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
>>>>>>>        Use --python=/path/to/python to specify a supported Python.
>>>>>>>
>>>>>>> The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
>>>>>>> box; it's one of the gcc compile farm machines so upgrades to its
>>>>>>> OS are not really under my control.)
>>>>>>
>>>>>> Rereading this, I realise that either the check or the error
>>>>>> message is wrong here. The machine has 2.7.6, which satisfies
>>>>>> "python 2 >= 2.7", so we should be OK to build. The bug
>>>>>> seems to be that we say "prefer python3 over plain python
>>>>>> on python2" early, but don't revisit that decision if the
>>>>>> python3 we found isn't actually good enough for us.
>>>>>
>>>>> Right.  The error message is technically correct, but misleading.
>>>>> python3 is too old, but python2 would work.
>>>>>
>>>>> We can make configure not use python3 by default if it's too old,
>>>>> and fall back to python2 in this case.
>>>>
>>>> Sounds good. Since I have now managed to get my alternate
>>>> aarch64 box set up, how about I apply this pullreq and you
>>>> send a followup patch which does the fallback to python/python2 ?
>>>
>>> I will remove the python2/python3 patches and send a new pull
>>> request.
>>
>> What is the plan forward with this? Are the patches dropped for good?
>>
>> I think the plan was to drop Python 2 after QEMU 4.2, and then it
>> becomes really relevant what our minimum Python 3 version is. We've just
>> had another Python version discussion in the context of iotests (John
>> suggested using function annotations, but these are >= 3.5 only).
>>
>> Also, the fallback to Python 2 obviously makes no sense any more then,
>> so maybe it's not that important to add for a single QEMU release?
>>
>> As Peter seems to have indicated above that he found a replacement for
>> the test machine with an OS that isn't out of support, can we just
>> revive this patch as it is?
> 
> My plan is to remove Python 2 support in QEMU 4.2 (making the
> fallback to Python 2 a non-issue), and require Python >= 3.5.
> 
> Now, even if my plan is rejected and we keep supporting Python 2
> when building QEMU 4.2, my suggestion for the iotest maintainers
> is to make it require Python 3.5+ immediately, just like we do
> for tests/acceptance.  I don't see why we should keep wasting our
> energy supporting ancient Python versions in a test suite that is
> not a requirement for building QEMU.
> 

It's unfortunately now part of the 'make check' target which we use in
the vm tests as a default target ... but I think we can make the push to
change the build requires to 3.5+.

--js

Re: [Qemu-devel] Python 2 in tests/vm (was Re: [PULL 0/8] Python queue, 2019-06-07)

[Eduardo Habkost]

On Mon, Jul 01, 2019 at 07:25:27PM -0300, Eduardo Habkost wrote:
> On Mon, Jun 10, 2019 at 01:58:50PM +0100, Peter Maydell wrote:
> > On Fri, 7 Jun 2019 at 22:16, Eduardo Habkost <ehabkost@redhat.com> wrote:
> > >
> > > The following changes since commit 185b7ccc11354cbd69b6d53bf8d831dd964f6c88:
> > >
> > >   Merge remote-tracking branch 'remotes/cohuck/tags/s390x-20190607-2' into staging (2019-06-07 15:24:13 +0100)
> > >
> > > are available in the Git repository at:
> > >
> > >   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
> > >
> > > for you to fetch changes up to 8297719709f1339506d6da93ec69e6118ace8676:
> > >
> > >   travis: Make check-acceptance job more verbose (2019-06-07 18:11:02 -0300)
> > >
> > > ----------------------------------------------------------------
> > > Python queue, 2019-06-07
> > >
> > > * New boot_linux_console test cases (Philippe Mathieu-Daudé)
> > > * Deprecate Python 2 support (Eduardo Habkost)
> > > * Require python3 >= 3.5 (Eduardo Habkost)
> > > * Make check-acceptance Travis job more verbose (Eduardo Habkost)
> > >
> > > ----------------------------------------------------------------
> > 
> > Hi. This fails to build on one of my buildtest machines:
> > 
> > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> >        Use --python=/path/to/python to specify a supported Python.
> > 
> > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > box; it's one of the gcc compile farm machines so upgrades to its
> > OS are not really under my control.)
> > 
> > The configure check also spits out deprecation warnings for
> > the NetBSD/FreeBSD/OpenBSD tests/vm configurations. It would be nice
> > to get those updated.
> 
> CCing the test/vm maintainers.
> 
> Fam, Alex, are you able to fix this and create new BSD VM images
> with Python 3 available?  I thought the VM image configurations
> were stored in the source tree, but they are downloaded from
> download.patchew.org.

Fam, Alex, can you help us on this?  Python 2 won't be supported
anymore, so we need the VM images to be updated.

-- 
Eduardo

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Eduardo Habkost]

On Tue, Sep 17, 2019 at 07:10:13PM -0400, John Snow wrote:
> On 9/17/19 5:48 PM, Eduardo Habkost wrote:
> > On Tue, Sep 17, 2019 at 03:57:26PM +0200, Kevin Wolf wrote:
> >> Am 11.06.2019 um 19:12 hat Eduardo Habkost geschrieben:
> >>> On Tue, Jun 11, 2019 at 05:07:55PM +0100, Peter Maydell wrote:
> >>>> On Tue, 11 Jun 2019 at 17:03, Eduardo Habkost <ehabkost@redhat.com> wrote:
> >>>>>
> >>>>> On Tue, Jun 11, 2019 at 04:50:34PM +0100, Peter Maydell wrote:
> >>>>>> On Mon, 10 Jun 2019 at 13:58, Peter Maydell <peter.maydell@linaro.org> wrote:
> >>>>>>> Hi. This fails to build on one of my buildtest machines:
> >>>>>>>
> >>>>>>> ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> >>>>>>>        Use --python=/path/to/python to specify a supported Python.
> >>>>>>>
> >>>>>>> The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> >>>>>>> box; it's one of the gcc compile farm machines so upgrades to its
> >>>>>>> OS are not really under my control.)
> >>>>>>
> >>>>>> Rereading this, I realise that either the check or the error
> >>>>>> message is wrong here. The machine has 2.7.6, which satisfies
> >>>>>> "python 2 >= 2.7", so we should be OK to build. The bug
> >>>>>> seems to be that we say "prefer python3 over plain python
> >>>>>> on python2" early, but don't revisit that decision if the
> >>>>>> python3 we found isn't actually good enough for us.
> >>>>>
> >>>>> Right.  The error message is technically correct, but misleading.
> >>>>> python3 is too old, but python2 would work.
> >>>>>
> >>>>> We can make configure not use python3 by default if it's too old,
> >>>>> and fall back to python2 in this case.
> >>>>
> >>>> Sounds good. Since I have now managed to get my alternate
> >>>> aarch64 box set up, how about I apply this pullreq and you
> >>>> send a followup patch which does the fallback to python/python2 ?
> >>>
> >>> I will remove the python2/python3 patches and send a new pull
> >>> request.
> >>
> >> What is the plan forward with this? Are the patches dropped for good?
> >>
> >> I think the plan was to drop Python 2 after QEMU 4.2, and then it
> >> becomes really relevant what our minimum Python 3 version is. We've just
> >> had another Python version discussion in the context of iotests (John
> >> suggested using function annotations, but these are >= 3.5 only).
> >>
> >> Also, the fallback to Python 2 obviously makes no sense any more then,
> >> so maybe it's not that important to add for a single QEMU release?
> >>
> >> As Peter seems to have indicated above that he found a replacement for
> >> the test machine with an OS that isn't out of support, can we just
> >> revive this patch as it is?
> > 
> > My plan is to remove Python 2 support in QEMU 4.2 (making the
> > fallback to Python 2 a non-issue), and require Python >= 3.5.
> > 
> > Now, even if my plan is rejected and we keep supporting Python 2
> > when building QEMU 4.2, my suggestion for the iotest maintainers
> > is to make it require Python 3.5+ immediately, just like we do
> > for tests/acceptance.  I don't see why we should keep wasting our
> > energy supporting ancient Python versions in a test suite that is
> > not a requirement for building QEMU.
> > 
> 
> It's unfortunately now part of the 'make check' target which we use in
> the vm tests as a default target ... but I think we can make the push to
> change the build requires to 3.5+.

In the worst case, we can make "make check" skip iotests if
Python 2 is detected (after printing a warning).  But requiring
3.5+ for the build would really be the best option.

I've just restarted the thread about Python 2 in tests/vm:
https://lore.kernel.org/qemu-devel/20190917233140.GK4082@habkost.net/

-- 
Eduardo

Re: [Qemu-devel] [PULL 0/8] Python queue, 2019-06-07

[Kevin Wolf]

Am 17.09.2019 um 23:48 hat Eduardo Habkost geschrieben:
> On Tue, Sep 17, 2019 at 03:57:26PM +0200, Kevin Wolf wrote:
> > Am 11.06.2019 um 19:12 hat Eduardo Habkost geschrieben:
> > > On Tue, Jun 11, 2019 at 05:07:55PM +0100, Peter Maydell wrote:
> > > > On Tue, 11 Jun 2019 at 17:03, Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > >
> > > > > On Tue, Jun 11, 2019 at 04:50:34PM +0100, Peter Maydell wrote:
> > > > > > On Mon, 10 Jun 2019 at 13:58, Peter Maydell <peter.maydell@linaro.org> wrote:
> > > > > > > Hi. This fails to build on one of my buildtest machines:
> > > > > > >
> > > > > > > ERROR: Cannot use 'python3', Python 2 >= 2.7 or Python 3 >= 3.5 is required.
> > > > > > >        Use --python=/path/to/python to specify a supported Python.
> > > > > > >
> > > > > > > The machine has python 2.7.6 and 3.4.3. (It's an Ubuntu trusty
> > > > > > > box; it's one of the gcc compile farm machines so upgrades to its
> > > > > > > OS are not really under my control.)
> > > > > >
> > > > > > Rereading this, I realise that either the check or the error
> > > > > > message is wrong here. The machine has 2.7.6, which satisfies
> > > > > > "python 2 >= 2.7", so we should be OK to build. The bug
> > > > > > seems to be that we say "prefer python3 over plain python
> > > > > > on python2" early, but don't revisit that decision if the
> > > > > > python3 we found isn't actually good enough for us.
> > > > >
> > > > > Right.  The error message is technically correct, but misleading.
> > > > > python3 is too old, but python2 would work.
> > > > >
> > > > > We can make configure not use python3 by default if it's too old,
> > > > > and fall back to python2 in this case.
> > > > 
> > > > Sounds good. Since I have now managed to get my alternate
> > > > aarch64 box set up, how about I apply this pullreq and you
> > > > send a followup patch which does the fallback to python/python2 ?
> > > 
> > > I will remove the python2/python3 patches and send a new pull
> > > request.
> > 
> > What is the plan forward with this? Are the patches dropped for good?
> > 
> > I think the plan was to drop Python 2 after QEMU 4.2, and then it
> > becomes really relevant what our minimum Python 3 version is. We've just
> > had another Python version discussion in the context of iotests (John
> > suggested using function annotations, but these are >= 3.5 only).
> > 
> > Also, the fallback to Python 2 obviously makes no sense any more then,
> > so maybe it's not that important to add for a single QEMU release?
> > 
> > As Peter seems to have indicated above that he found a replacement for
> > the test machine with an OS that isn't out of support, can we just
> > revive this patch as it is?
> 
> My plan is to remove Python 2 support in QEMU 4.2 (making the
> fallback to Python 2 a non-issue), and require Python >= 3.5.

Then I think it would be best to make (or propose at least) that change
early in the release cycle. In other words, now. :-)

> Now, even if my plan is rejected and we keep supporting Python 2
> when building QEMU 4.2, my suggestion for the iotest maintainers
> is to make it require Python 3.5+ immediately, just like we do
> for tests/acceptance.  I don't see why we should keep wasting our
> energy supporting ancient Python versions in a test suite that is
> not a requirement for building QEMU.

Okay, if you as the Python maintainer say so, I'll gladly follow your
advice.

Maybe we can modify iotests so that it just skips Python tests if the
minimum version isn't available to keep the impact of deviating from the
global minimum version as small as possible. Of course, this will only
be necessary if your proposal to make 3.5 the minimum for all of QEMU is
rejected.

Kevin