[PATCH 0/2] iotests: Run pylint and mypy in a testcase

[PATCH 0/2] iotests: Run pylint and mypy in a testcase

[Kevin Wolf]

Kevin Wolf (2):
  iotests: Fix incomplete type declarations
  iotests: Run pylint and mypy in a testcase

 tests/qemu-iotests/iotests.py |  8 +++----
 tests/qemu-iotests/297        | 44 +++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/297.out    |  3 +++
 tests/qemu-iotests/group      |  1 +
 4 files changed, 52 insertions(+), 4 deletions(-)
 create mode 100755 tests/qemu-iotests/297
 create mode 100644 tests/qemu-iotests/297.out

-- 
2.25.3

[PATCH 1/2] iotests: Fix incomplete type declarations

[Kevin Wolf]

We need to fix only a few places so that iotests.py can pass
mypy --disallow-incomplete-defs, which seems to be a desirable option to
have enabled in the long run.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/qemu-iotests/iotests.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 6c0e781af7..1d7f6fd7cf 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -1040,7 +1040,7 @@ def _verify_cache_mode(supported_cache_modes: Sequence[str] = ()) -> None:
     if supported_cache_modes and (cachemode not in supported_cache_modes):
         notrun('not suitable for this cache mode: %s' % cachemode)
 
-def _verify_aio_mode(supported_aio_modes: Sequence[str] = ()):
+def _verify_aio_mode(supported_aio_modes: Sequence[str] = ()) -> None:
     if supported_aio_modes and (aiomode not in supported_aio_modes):
         notrun('not suitable for this aio mode: %s' % aiomode)
 
@@ -1087,7 +1087,8 @@ def skip_if_unsupported(required_formats=(), read_only=False):
     '''Skip Test Decorator
        Runs the test if all the required formats are whitelisted'''
     def skip_test_decorator(func):
-        def func_wrapper(test_case: QMPTestCase, *args, **kwargs):
+        def func_wrapper(test_case: QMPTestCase, *args: List[Any],
+                         **kwargs: Dict[str, Any]) -> None:
             if callable(required_formats):
                 fmts = required_formats(test_case)
             else:
@@ -1097,9 +1098,8 @@ def skip_if_unsupported(required_formats=(), read_only=False):
             if usf_list:
                 msg = f'{test_case}: formats {usf_list} are not whitelisted'
                 test_case.case_skip(msg)
-                return None
             else:
-                return func(test_case, *args, **kwargs)
+                func(test_case, *args, **kwargs)
         return func_wrapper
     return skip_test_decorator
 
-- 
2.25.3

[PATCH 2/2] iotests: Run pylint and mypy in a testcase

[Kevin Wolf]

We made sure that iotests.py passes pylint. It would be a shame if we
allowed new patches in that break this again, so let's just add a
meta-test case that runs pylint on it.

While we don't pass mypy --strict yet, we can already run it with a few
options that would be part of --strict to make sure that we won't
regress on these aspects at least until we can enable the full thing.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 tests/qemu-iotests/297     | 44 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/297.out |  3 +++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 48 insertions(+)
 create mode 100755 tests/qemu-iotests/297
 create mode 100644 tests/qemu-iotests/297.out

diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
new file mode 100755
index 0000000000..5c5420712b
--- /dev/null
+++ b/tests/qemu-iotests/297
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2020 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+# get standard environment
+. ./common.rc
+
+if ! type -p "pylint-3" > /dev/null; then
+    _notrun "pylint-3 not found"
+fi
+if ! type -p "mypy" > /dev/null; then
+    _notrun "mypy not found"
+fi
+
+pylint-3 --score=n iotests.py
+
+MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \
+    --disallow-any-generics --disallow-incomplete-defs \
+    --disallow-untyped-decorators --no-implicit-optional \
+    --warn-redundant-casts --warn-unused-ignores \
+    --no-implicit-reexport iotests.py
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/297.out b/tests/qemu-iotests/297.out
new file mode 100644
index 0000000000..6acc843649
--- /dev/null
+++ b/tests/qemu-iotests/297.out
@@ -0,0 +1,3 @@
+QA output created by 297
+Success: no issues found in 1 source file
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index fe649c5b73..c514975880 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -299,3 +299,4 @@
 289 rw quick
 290 rw auto quick
 292 rw auto quick
+297 meta
-- 
2.25.3

Re: [PATCH 1/2] iotests: Fix incomplete type declarations

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 417 bytes --]

On 11.05.20 18:35, Kevin Wolf wrote:
> We need to fix only a few places so that iotests.py can pass
> mypy --disallow-incomplete-defs, which seems to be a desirable option to
> have enabled in the long run.
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 2/2] iotests: Run pylint and mypy in a testcase

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 1362 bytes --]

On 11.05.20 18:35, Kevin Wolf wrote:
> We made sure that iotests.py passes pylint. It would be a shame if we
> allowed new patches in that break this again, so let's just add a
> meta-test case that runs pylint on it.
> 
> While we don't pass mypy --strict yet, we can already run it with a few
> options that would be part of --strict to make sure that we won't
> regress on these aspects at least until we can enable the full thing.
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  tests/qemu-iotests/297     | 44 ++++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/297.out |  3 +++
>  tests/qemu-iotests/group   |  1 +
>  3 files changed, 48 insertions(+)
>  create mode 100755 tests/qemu-iotests/297
>  create mode 100644 tests/qemu-iotests/297.out

Bit of a shame that this takes 8 s (on my machine at least) and will run
with every format/protocol combination unless I explictly exclude it
with -x meta...  But I suppose the actual problem here is the fact that
tests still can’t just define a “This is the format/protocol combination
I require” and then you can just let all tests run once with that
default combination.  (And maybe afterwards run all tests again with
some custom combinations, but only when that makes sense.)

Well.  Not a new problem.

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 2/2] iotests: Run pylint and mypy in a testcase

[Kevin Wolf]

[-- Attachment #1: Type: text/plain, Size: 1752 bytes --]

Am 13.05.2020 um 14:14 hat Max Reitz geschrieben:
> On 11.05.20 18:35, Kevin Wolf wrote:
> > We made sure that iotests.py passes pylint. It would be a shame if we
> > allowed new patches in that break this again, so let's just add a
> > meta-test case that runs pylint on it.
> > 
> > While we don't pass mypy --strict yet, we can already run it with a few
> > options that would be part of --strict to make sure that we won't
> > regress on these aspects at least until we can enable the full thing.
> > 
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> >  tests/qemu-iotests/297     | 44 ++++++++++++++++++++++++++++++++++++++
> >  tests/qemu-iotests/297.out |  3 +++
> >  tests/qemu-iotests/group   |  1 +
> >  3 files changed, 48 insertions(+)
> >  create mode 100755 tests/qemu-iotests/297
> >  create mode 100644 tests/qemu-iotests/297.out
> 
> Bit of a shame that this takes 8 s (on my machine at least) and will run
> with every format/protocol combination unless I explictly exclude it
> with -x meta...

Yes, it's surprising how slow these tools are. At least mypy caches some
stuff, so the second run is considerably faster, but pylint doesn't do
that.

I wonder if there is some overlap between mypy and pylint that we could
configure away in pylint to speed it up.

> But I suppose the actual problem here is the fact that
> tests still can’t just define a “This is the format/protocol combination
> I require” and then you can just let all tests run once with that
> default combination.  (And maybe afterwards run all tests again with
> some custom combinations, but only when that makes sense.)

It's probably not hard to find more "actual problems" in the test
harness...

Kevin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 0/2] iotests: Run pylint and mypy in a testcase

[John Snow]

On 5/11/20 12:35 PM, Kevin Wolf wrote:
> Kevin Wolf (2):
>   iotests: Fix incomplete type declarations
>   iotests: Run pylint and mypy in a testcase
> 
>  tests/qemu-iotests/iotests.py |  8 +++----
>  tests/qemu-iotests/297        | 44 +++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/297.out    |  3 +++
>  tests/qemu-iotests/group      |  1 +
>  4 files changed, 52 insertions(+), 4 deletions(-)
>  create mode 100755 tests/qemu-iotests/297
>  create mode 100644 tests/qemu-iotests/297.out
> 

Cool!

Maybe the approach of putting this in 'make check' somewhere can
alleviate that expensive test being performed in iotests directly (as
Max notes), but, uhm. whatever.

For now, this will help prevent the backslide.

Reviewed-by: John Snow <jsnow@redhat.com>