From: Hanna Reitz <hreitz@redhat.com>
To: John Snow <jsnow@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>,
Eduardo Habkost <ehabkost@redhat.com>,
qemu-devel <qemu-devel@nongnu.org>,
qemu-block@nongnu.org, Cleber Rosa <crosa@redhat.com>
Subject: Re: [PATCH v2 11/15] iotests: split linters.py out from 297
Date: Thu, 28 Oct 2021 12:34:06 +0200 [thread overview]
Message-ID: <f9dd697b-02ef-7f67-32ff-595e9f2d9457@redhat.com> (raw)
In-Reply-To: <CAFn=p-agMGtbryADMxgiG5OLFSXyZd+Zf9YrJyyyRVXgN2UGgA@mail.gmail.com>
On 26.10.21 20:30, John Snow wrote:
>
>
> On Tue, Oct 26, 2021 at 6:51 AM Hanna Reitz <hreitz@redhat.com> wrote:
>
> On 19.10.21 16:49, John Snow wrote:
> > Now, 297 is just the iotests-specific incantations and
> linters.py is as
> > minimal as I can think to make it. The only remaining element in
> here
> > that ought to be configuration and not code is the list of skip
> files,
> > but they're still numerous enough that repeating them for mypy and
> > pylint configurations both would be ... a hassle.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> > tests/qemu-iotests/297 | 72
> +++++----------------------------
> > tests/qemu-iotests/linters.py | 76
> +++++++++++++++++++++++++++++++++++
> > 2 files changed, 87 insertions(+), 61 deletions(-)
> > create mode 100644 tests/qemu-iotests/linters.py
>
> Reviewed-by: Hanna Reitz <hreitz@redhat.com>
>
>
> Thanks!
>
> I wonder about `check_linter()`, though. By not moving it to
> linters.py, we can’t use it in its entry point, and so the Python
> test
> infrastructure will have a strong dependency on these linters. Though
> then again, it probably already does, and I suppose that’s one of the
> points hindering us from running this from make check?
>
>
> That one is left behind because it uses iotests API to skip a test.
> Environment setup that guarantees we won't *need* to skip the test is
> handled by the virtual environment setup magic in qemu/python/Makefile.
>
> Hanna
>
>
> More info than you require:
>
> There's maybe about four ways you could run the python tests that
> might make sense depending on who you are and what you're trying to
> accomplish; they're documented in "make help" and again in
> qemu/python/README.rst;
>
> (1) make check-dev -- makes a venv with whatever python you happen to
> have, installs the latest packages, runs the tests
> (2) make check-pipenv -- requires python 3.6 specifically, installs
> the *oldest* packages, runs the tests
> (3) make check-tox -- requires python 3.6 through 3.10, installs the
> newest packages, runs the tests per each python version
> (4) make check -- perform no setup at all, just run the tests in the
> current environment. (Used directly by all three prior invocations)
AFAIU these are all to be run in build/python? Isn’t any of them hooked
up to the global `make check` in build? Because...
> In general, I assume that human beings that aren't working on Python
> stuff actively will be using (1) at their interactive console, if they
> decide to run any of these at all.
...I’m just running `make check` in the build directory.
I would have hoped that this is hooked up to something that won’t fail
because I don’t have some necessary tools installed or perhaps even
because I have the wrong version of some tools installed (although the
latter would be kind of questionable...). Would be nice if the global
`make check` had a summary on what was skipped...
> It imposes the least pre-requirements while still endeavoring to be a
> target that will "just work".
> Options (2) and (3) are what power the CI tests 'check-python-pipenv'
> and 'check-python-tox', respectively; with(2) being the one that
> actually gates the CI.
> Option (4) is only really a convenience for bypassing the venv-setup
> stuff if you want to construct your own (or not use one at all). So
> the tests just assume that the tools they have available will work.
> It's kind of a 'power user' target.
>
> The way the CI works at the moment is that it uses a "fedora:latest"
> base image and installs python interpreters 3.6 through 3.10
> inclusive, pipenv, and tox. From there, it has enough juice to run the
> makefile targets which take care of all of the actual linting
> dependencies and their different versions to get a wider matrix on the
> version testing to ensure we're still keeping compatibility with the
> 3.6 platform while also checking for new problems that show up on the
> bleeding edge.
Perhaps it’s unreasonable, but I’d prefer if a local `make check` would
already run most tests in some form or another and that I don’t have to
push to gitlab and wait for the CI there.
I mean, I can of course also integrate a `cd python && make check-dev`
invocation into my test scripts, but it doesn’t feel right to
special-case one test area.
> iotests 297 right now doesn't do any python environment setup at all,
> so we can't guarantee that the linters are present, so we decide to
> allow the test to be skipped. My big hesitation of adding this test
> directly into 'make check' is that I will need to do some environment
> probing to make sure that the 'pylint' version isn't too old -- or
> otherwise set up a venv in the build directory that can be used to run
> tests. I know we already set one up for the acceptance tests, so maybe
> there's an opportunity to re-use that venv, but I need to make sure
> that the dependencies between the two sets of tests are aligned. I
> don't know if they agree, currently.
I see.
> I think I probably want to minimize the number of different virtual
> python environments we create during the build, so I will look into
> what problems might exist in re-purposing the acceptance test venv. If
> that can get squared away easily, there's likely no real big barrier
> to adding more tests that rely on a python venv to get cooking during
> the normal build/check process, including iotest 297.
OK, thanks for the explanation!
Hanna
next prev parent reply other threads:[~2021-10-28 10:43 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-19 14:49 [PATCH v2 00/15] python/iotests: Run iotest linters during Python CI John Snow
2021-10-19 14:49 ` [PATCH v2 01/15] iotests/297: Move pylint config into pylintrc John Snow
2021-10-19 14:49 ` [PATCH v2 02/15] iotests/297: Split mypy configuration out into mypy.ini John Snow
2021-10-19 14:49 ` [PATCH v2 03/15] iotests/297: Add get_files() function John Snow
2021-10-19 14:49 ` [PATCH v2 04/15] iotests/297: Create main() function John Snow
2021-10-19 14:49 ` [PATCH v2 05/15] iotests/297: Don't rely on distro-specific linter binaries John Snow
2021-10-19 14:49 ` [PATCH v2 06/15] iotests/297: Split run_linters apart into run_pylint and run_mypy John Snow
2021-10-19 14:49 ` [PATCH v2 07/15] iotests/297: refactor run_[mypy|pylint] as generic execution shim John Snow
2021-10-26 10:01 ` Hanna Reitz
2021-10-19 14:49 ` [PATCH v2 08/15] iotests/297: Change run_linter() to raise an exception on failure John Snow
2021-10-26 10:10 ` Hanna Reitz
2021-10-26 17:59 ` John Snow
2021-10-19 14:49 ` [PATCH v2 09/15] iotests/297: update tool availability checks John Snow
2021-10-26 10:19 ` Hanna Reitz
2021-10-19 14:49 ` [PATCH v2 10/15] iotests/297: split test into sub-cases John Snow
2021-10-26 10:29 ` Hanna Reitz
2021-10-26 18:02 ` John Snow
2021-10-19 14:49 ` [PATCH v2 11/15] iotests: split linters.py out from 297 John Snow
2021-10-26 10:51 ` Hanna Reitz
2021-10-26 18:30 ` John Snow
2021-10-28 10:34 ` Hanna Reitz [this message]
2021-10-28 16:27 ` John Snow
2021-10-19 14:49 ` [PATCH v2 12/15] iotests/linters: Add entry point for linting via Python CI John Snow
2021-10-26 10:57 ` Hanna Reitz
2021-10-26 18:36 ` John Snow
2021-10-26 19:45 ` John Snow
2021-10-28 10:36 ` Hanna Reitz
2021-10-19 14:49 ` [PATCH v2 13/15] iotests/linters: Add workaround for mypy bug #9852 John Snow
2021-10-19 14:49 ` [PATCH v2 14/15] python: Add iotest linters to test suite John Snow
2021-10-19 14:49 ` [PATCH v2 15/15] iotests: [RFC] drop iotest 297 John Snow
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f9dd697b-02ef-7f67-32ff-595e9f2d9457@redhat.com \
--to=hreitz@redhat.com \
--cc=crosa@redhat.com \
--cc=ehabkost@redhat.com \
--cc=jsnow@redhat.com \
--cc=kwolf@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).