[PULL 0/1] Require Python >= 3.5 to build QEMU

[PULL 0/1] Require Python >= 3.5 to build QEMU

[Eduardo Habkost]

The following changes since commit 03bf012e523ecdf047ac56b2057950247256064d:

  Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2019-10-25 14:59:53 +0100)

are available in the Git repository at:

  git://github.com/ehabkost/qemu.git tags/python-next-pull-request

for you to fetch changes up to d24e417866f85229de1b75bc5c0a1d942451a842:

  configure: Require Python >= 3.5 (2019-10-25 16:34:57 -0300)

----------------------------------------------------------------
Require Python >= 3.5 to build QEMU

----------------------------------------------------------------

Eduardo Habkost (1):
  configure: Require Python >= 3.5

 configure              | 18 ++++--------------
 tests/Makefile.include |  5 -----
 2 files changed, 4 insertions(+), 19 deletions(-)

-- 
2.21.0

[PULL 1/1] configure: Require Python >= 3.5

[Eduardo Habkost]

Python 3.5 is the oldest Python version available on our
supported build platforms, and Python 2 end of life will be 3
weeks after the planned release date of QEMU 4.2.0.  Drop Python
2 support from configure completely, and require Python 3.5 or
newer.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20191016224237.26180-1-ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 configure              | 18 ++++--------------
 tests/Makefile.include |  5 -----
 2 files changed, 4 insertions(+), 19 deletions(-)

diff --git a/configure b/configure
index 145fcabbb3..bfc2b1a9d8 100755
--- a/configure
+++ b/configure
@@ -896,9 +896,9 @@ fi
 : ${install=${INSTALL-install}}
 # We prefer python 3.x. A bare 'python' is traditionally
 # python 2.x, but some distros have it as python 3.x, so
-# we check that before python2
+# we check that too
 python=
-for binary in "${PYTHON-python3}" python python2
+for binary in "${PYTHON-python3}" python
 do
     if has "$binary"
     then
@@ -1829,8 +1829,8 @@ fi
 
 # Note that if the Python conditional here evaluates True we will exit
 # with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (2,7))'; then
-  error_exit "Cannot use '$python', Python 2 >= 2.7 or Python 3 is required." \
+if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
+  error_exit "Cannot use '$python', Python >= 3.5 is required." \
       "Use --python=/path/to/python to specify a supported Python."
 fi
 
@@ -6466,15 +6466,6 @@ if test "$supported_os" = "no"; then
     echo "us upstream at qemu-devel@nongnu.org."
 fi
 
-# Note that if the Python conditional here evaluates True we will exit
-# with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,0))'; then
-  echo
-  echo "warning: Python 2 support is deprecated" >&2
-  echo "warning: Python 3 will be required for building future versions of QEMU" >&2
-  python2="y"
-fi
-
 config_host_mak="config-host.mak"
 
 echo "# Automatically generated by configure - do not modify" >config-all-disas.mak
@@ -7295,7 +7286,6 @@ echo "INSTALL_DATA=$install -c -m 0644" >> $config_host_mak
 echo "INSTALL_PROG=$install -c -m 0755" >> $config_host_mak
 echo "INSTALL_LIB=$install -c -m 0644" >> $config_host_mak
 echo "PYTHON=$python" >> $config_host_mak
-echo "PYTHON2=$python2" >> $config_host_mak
 echo "CC=$cc" >> $config_host_mak
 if $iasl -h > /dev/null 2>&1; then
   echo "IASL=$iasl" >> $config_host_mak
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 09e5b410dc..c4e656001e 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1139,7 +1139,6 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 AVOCADO_SHOW=app
 AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGET_DIRS)))
 
-ifneq ($(PYTHON2),y)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
 	$(call quiet-command, \
             $(PYTHON) -m venv --system-site-packages $@, \
@@ -1148,10 +1147,6 @@ $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
             $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
             PIP, $(TESTS_VENV_REQ))
 	$(call quiet-command, touch $@)
-else
-$(TESTS_VENV_DIR):
-	$(error "venv directory for tests requires Python 3")
-endif
 
 $(TESTS_RESULTS_DIR):
 	$(call quiet-command, mkdir -p $@, \
-- 
2.21.0

Re: [PULL 0/1] Require Python >= 3.5 to build QEMU

[Peter Maydell]

On Fri, 25 Oct 2019 at 21:34, Eduardo Habkost <ehabkost@redhat.com> wrote:
>
> The following changes since commit 03bf012e523ecdf047ac56b2057950247256064d:
>
>   Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2019-10-25 14:59:53 +0100)
>
> are available in the Git repository at:
>
>   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
>
> for you to fetch changes up to d24e417866f85229de1b75bc5c0a1d942451a842:
>
>   configure: Require Python >= 3.5 (2019-10-25 16:34:57 -0300)
>
> ----------------------------------------------------------------
> Require Python >= 3.5 to build QEMU
>
> ----------------------------------------------------------------

I can't apply this until we've fixed the tests/vm netbsd setup to
not use Python 2.

Have you tried a test run with Travis/etc/etc to check that none of
those CI configs need updating to have python3 available ?

thanks
-- PMM

Re: [PULL 0/1] Require Python >= 3.5 to build QEMU

[Eduardo Habkost]

On Thu, Oct 31, 2019 at 08:12:01AM +0000, Peter Maydell wrote:
> On Fri, 25 Oct 2019 at 21:34, Eduardo Habkost <ehabkost@redhat.com> wrote:
> >
> > The following changes since commit 03bf012e523ecdf047ac56b2057950247256064d:
> >
> >   Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2019-10-25 14:59:53 +0100)
> >
> > are available in the Git repository at:
> >
> >   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
> >
> > for you to fetch changes up to d24e417866f85229de1b75bc5c0a1d942451a842:
> >
> >   configure: Require Python >= 3.5 (2019-10-25 16:34:57 -0300)
> >
> > ----------------------------------------------------------------
> > Require Python >= 3.5 to build QEMU
> >
> > ----------------------------------------------------------------
> 
> I can't apply this until we've fixed the tests/vm netbsd setup to
> not use Python 2.

Fixing tests/vm/netbsd is being tricky.  It looks like the
configure patch will have to wait until after QEMU 4.2.0.  :(

> 
> Have you tried a test run with Travis/etc/etc to check that none of
> those CI configs need updating to have python3 available ?

I have tested this pull request on Shippable, and I will take a
look at Travis.  I'd appreciate help from the CI system
maintainers (CCed) for the rest, as I don't have accounts in all
our CI systems.

Do we expect maintainers to test their pull requests in all CI
systems listed at the QEMU wiki[1]?  Do we have an official list
of CI systems that you consider to be pull request blockers?

[1] https://wiki.qemu.org/Testing#Continuous_Integration

-- 
Eduardo

Re: [PULL 0/1] Require Python >= 3.5 to build QEMU

[Peter Maydell]

On Tue, 5 Nov 2019 at 19:57, Eduardo Habkost <ehabkost@redhat.com> wrote:
> Fixing tests/vm/netbsd is being tricky.  It looks like the
> configure patch will have to wait until after QEMU 4.2.0.  :(

I think that makes sense at this point in the release cycle, yes.

> > Have you tried a test run with Travis/etc/etc to check that none of
> > those CI configs need updating to have python3 available ?
>
> I have tested this pull request on Shippable, and I will take a
> look at Travis.  I'd appreciate help from the CI system
> maintainers (CCed) for the rest, as I don't have accounts in all
> our CI systems.
>
> Do we expect maintainers to test their pull requests in all CI
> systems listed at the QEMU wiki[1]?  Do we have an official list
> of CI systems that you consider to be pull request blockers?

No, I don't in general expect people to test on all these CI
systems. I ask in this specific case because trying to
move to python-3-only seems like a change that's quite
likely to break the CI setups unless we've already checked
that all their configs will be pulling in a python 3 already.

thanks
-- PMM

Re: [PULL 0/1] Require Python >= 3.5 to build QEMU

[Alex Bennée]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Oct 31, 2019 at 08:12:01AM +0000, Peter Maydell wrote:
>> On Fri, 25 Oct 2019 at 21:34, Eduardo Habkost <ehabkost@redhat.com> wrote:
>> >
>> > The following changes since commit 03bf012e523ecdf047ac56b2057950247256064d:
>> >
>> >   Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2019-10-25 14:59:53 +0100)
>> >
>> > are available in the Git repository at:
>> >
>> >   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
>> >
>> > for you to fetch changes up to d24e417866f85229de1b75bc5c0a1d942451a842:
>> >
>> >   configure: Require Python >= 3.5 (2019-10-25 16:34:57 -0300)
>> >
>> > ----------------------------------------------------------------
>> > Require Python >= 3.5 to build QEMU
>> >
>> > ----------------------------------------------------------------
>>
>> I can't apply this until we've fixed the tests/vm netbsd setup to
>> not use Python 2.
>
> Fixing tests/vm/netbsd is being tricky.  It looks like the
> configure patch will have to wait until after QEMU 4.2.0.  :(

I've posted fixes for the netbsd serial install but there are still
problems with the tests including what looks like a fairly serious
failure in the async code.

>
>>
>> Have you tried a test run with Travis/etc/etc to check that none of
>> those CI configs need updating to have python3 available ?
>
> I have tested this pull request on Shippable, and I will take a
> look at Travis.  I'd appreciate help from the CI system
> maintainers (CCed) for the rest, as I don't have accounts in all
> our CI systems.

Setting up accounts on the others doesn't take long. I use the
CustomCIStatus template to instantiate all the buttons for my various
maintainer branches on the wiki, e.g.:

  {{CustomCIStatus|user=stsquad|repo=qemu|branch=testing/next|ship_proj=5885eac43b653a0f00fa97f5}}

which means I just have to glance at the button state rather than going
through each individual CI's status pages.

> Do we expect maintainers to test their pull requests in all CI
> systems listed at the QEMU wiki[1]?  Do we have an official list
> of CI systems that you consider to be pull request blockers?

Well they all catch various things but none of them catch all the things
Peter's PR processing does. Historically Travis has been allowed to
slide because of test instability and timeouts. Having said that last I
checked everything was green so breaking any of the main CIs
(Travis/Shippable/Cirrus/Gitlab) indicates there is a problem that needs
to be fixed.

>
> [1] https://wiki.qemu.org/Testing#Continuous_Integration


--
Alex Bennée

Re: [PULL 0/1] Require Python >= 3.5 to build QEMU

[Eduardo Habkost]

On Tue, Nov 05, 2019 at 08:25:03PM +0000, Alex Bennée wrote:
> 
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Thu, Oct 31, 2019 at 08:12:01AM +0000, Peter Maydell wrote:
> >> On Fri, 25 Oct 2019 at 21:34, Eduardo Habkost <ehabkost@redhat.com> wrote:
> >> >
> >> > The following changes since commit 03bf012e523ecdf047ac56b2057950247256064d:
> >> >
> >> >   Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2019-10-25 14:59:53 +0100)
> >> >
> >> > are available in the Git repository at:
> >> >
> >> >   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
> >> >
> >> > for you to fetch changes up to d24e417866f85229de1b75bc5c0a1d942451a842:
> >> >
> >> >   configure: Require Python >= 3.5 (2019-10-25 16:34:57 -0300)
> >> >
> >> > ----------------------------------------------------------------
> >> > Require Python >= 3.5 to build QEMU
> >> >
> >> > ----------------------------------------------------------------
> >>
> >> I can't apply this until we've fixed the tests/vm netbsd setup to
> >> not use Python 2.
> >
> > Fixing tests/vm/netbsd is being tricky.  It looks like the
> > configure patch will have to wait until after QEMU 4.2.0.  :(
> 
> I've posted fixes for the netbsd serial install but there are still
> problems with the tests including what looks like a fairly serious
> failure in the async code.

This sounds like a known "feature": QEMU expects clients to be
constantly reading from chardev sockets until the socket is
closed.  Otherwise, VCPU threads may block waiting for the socket
to be writeable.


> 
> >
> >>
> >> Have you tried a test run with Travis/etc/etc to check that none of
> >> those CI configs need updating to have python3 available ?
> >
> > I have tested this pull request on Shippable, and I will take a
> > look at Travis.  I'd appreciate help from the CI system
> > maintainers (CCed) for the rest, as I don't have accounts in all
> > our CI systems.
> 
> Setting up accounts on the others doesn't take long. I use the
> CustomCIStatus template to instantiate all the buttons for my various
> maintainer branches on the wiki, e.g.:
> 
>   {{CustomCIStatus|user=stsquad|repo=qemu|branch=testing/next|ship_proj=5885eac43b653a0f00fa97f5}}
> 
> which means I just have to glance at the button state rather than going
> through each individual CI's status pages.

This is awesome.  Thanks for the tip!

> 
> > Do we expect maintainers to test their pull requests in all CI
> > systems listed at the QEMU wiki[1]?  Do we have an official list
> > of CI systems that you consider to be pull request blockers?
> 
> Well they all catch various things but none of them catch all the things
> Peter's PR processing does. Historically Travis has been allowed to
> slide because of test instability and timeouts. Having said that last I
> checked everything was green so breaking any of the main CIs
> (Travis/Shippable/Cirrus/Gitlab) indicates there is a problem that needs
> to be fixed.

Manually checking if 5 different CI systems are green wouldn't be
reasonable to me, but the CustomCIStatus template will be useful.

-- 
Eduardo

Re: [PULL 0/1] Require Python >= 3.5 to build QEMU

[Daniel P. Berrangé]

On Tue, Nov 05, 2019 at 08:25:03PM +0000, Alex Bennée wrote:
> 
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Thu, Oct 31, 2019 at 08:12:01AM +0000, Peter Maydell wrote:
> >> On Fri, 25 Oct 2019 at 21:34, Eduardo Habkost <ehabkost@redhat.com> wrote:
> >> >
> >> > The following changes since commit 03bf012e523ecdf047ac56b2057950247256064d:
> >> >
> >> >   Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2019-10-25 14:59:53 +0100)
> >> >
> >> > are available in the Git repository at:
> >> >
> >> >   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
> >> >
> >> > for you to fetch changes up to d24e417866f85229de1b75bc5c0a1d942451a842:
> >> >
> >> >   configure: Require Python >= 3.5 (2019-10-25 16:34:57 -0300)
> >> >
> >> > ----------------------------------------------------------------
> >> > Require Python >= 3.5 to build QEMU
> >> >
> >> > ----------------------------------------------------------------
> >>
> >> I can't apply this until we've fixed the tests/vm netbsd setup to
> >> not use Python 2.
> >
> > Fixing tests/vm/netbsd is being tricky.  It looks like the
> > configure patch will have to wait until after QEMU 4.2.0.  :(
> 
> I've posted fixes for the netbsd serial install but there are still
> problems with the tests including what looks like a fairly serious
> failure in the async code.

At what point do we declare that NetBSD CI is broken and is no longer
considered a supported platform from POV of blocking the merging of
PULL requests ? It has been preventing the dropping of python2 for
quite a while now. It isn't the end of the world in this particular
case, as dropping py2 is mostly just a cleanup, but I feel like we
might benefit from setting expectations for ongoing platform maintenance,
otherwise these kind of issues could drag on indefinitely.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PULL 0/1] Require Python >= 3.5 to build QEMU

[Peter Maydell]

On Wed, 6 Nov 2019 at 10:36, Daniel P. Berrangé <berrange@redhat.com> wrote:
> At what point do we declare that NetBSD CI is broken and is no longer
> considered a supported platform from POV of blocking the merging of
> PULL requests ? It has been preventing the dropping of python2 for
> quite a while now. It isn't the end of the world in this particular
> case, as dropping py2 is mostly just a cleanup, but I feel like we
> might benefit from setting expectations for ongoing platform maintenance,
> otherwise these kind of issues could drag on indefinitely.

It works fine for me, and it means we have coverage of a host
OS we otherwise would not. To me that is definitely more important
than being able to drop Python 2 support. Also, AIUI the problem
that's blocking updating the NetBSD image isn't related to
NetBSD at all but is a bug in some combination of QEMU itself
and our test framework -- both of those are things we need to
fix anyway.

thanks
-- PMM

Re: [PULL 0/1] Require Python >= 3.5 to build QEMU

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 6 Nov 2019 at 10:36, Daniel P. Berrangé <berrange@redhat.com> wrote:
>> At what point do we declare that NetBSD CI is broken and is no longer
>> considered a supported platform from POV of blocking the merging of
>> PULL requests ? It has been preventing the dropping of python2 for
>> quite a while now. It isn't the end of the world in this particular
>> case, as dropping py2 is mostly just a cleanup, but I feel like we
>> might benefit from setting expectations for ongoing platform maintenance,
>> otherwise these kind of issues could drag on indefinitely.
>
> It works fine for me, and it means we have coverage of a host
> OS we otherwise would not. To me that is definitely more important
> than being able to drop Python 2 support. Also, AIUI the problem
> that's blocking updating the NetBSD image isn't related to
> NetBSD at all but is a bug in some combination of QEMU itself
> and our test framework

These have been addressed in:

  Subject: [PATCH  v1 0/6] testing/next (netbsd stuff)
  Date: Mon,  4 Nov 2019 17:36:48 +0000
  Message-Id: <20191104173654.30125-1-alex.bennee@linaro.org>

but I have a non-trivial failure rate running tests (~20% of runs fail)

> -- both of those are things we need to
> fix anyway.
>
> thanks
> -- PMM


--
Alex Bennée