From: John Snow <jsnow@redhat.com>
To: qemu-devel@nongnu.org
Cc: "Kevin Wolf" <kwolf@redhat.com>, "Fam Zheng" <fam@euphon.net>,
"Thomas Huth" <thuth@redhat.com>,
"Eduardo Habkost" <ehabkost@redhat.com>,
qemu-block@nongnu.org,
"Philippe Mathieu-Daudé" <philmd@redhat.com>,
"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
"Max Reitz" <mreitz@redhat.com>,
"Willian Rampazzo" <willianr@redhat.com>,
"Willian Rampazzo" <wrampazz@redhat.com>,
"Cleber Rosa" <crosa@redhat.com>,
"Alex Bennée" <alex.bennee@linaro.org>,
"Beraldo Leal" <bleal@redhat.com>
Subject: Re: [PATCH v6 00/25] python: create installable package
Date: Mon, 17 May 2021 14:52:33 -0400 [thread overview]
Message-ID: <d59b6295-58a4-90bd-f898-75d8fc5326ed@redhat.com> (raw)
In-Reply-To: <20210512231241.2816122-1-jsnow@redhat.com>
On 5/12/21 7:12 PM, John Snow wrote:
> Based-on: <20210512214642.2803189-1-jsnow@redhat.com>
> CI: https://gitlab.com/jsnow/qemu/-/pipelines/302010131
> GitLab: https://gitlab.com/jsnow/qemu/-/tree/python-package-mk3
> MR: https://gitlab.com/jsnow/qemu/-/merge_requests/4
>
Patchset updated and rebased on top of new linting pre-req series.
(Gitlab branch and MR rebased and updated.)
Based-on: <20210517184808.3562549-1-jsnow@redhat.com>
> I invite you to leave review comments on my mock merge request on
> gitlab, submitted against my own mirror. I will, as always, also respond
> to feedback on-list.
>
> ABOUT
> =====
>
> This series factors the python/qemu directory as an installable
> package. It does not yet actually change the mechanics of how any other
> python source in the tree actually consumes it (yet), beyond the import
> path -- some import statements change in a few places.
>
> RATIONALE
> =========
>
> The primary motivation of this series is primarily to formalize our
> dependencies on mypy, flake8, isort, and pylint alongside versions that
> are known to work. It does this using the setup.cfg and setup.py
> files. It also adds explicitly pinned versions (using Pipfile.lock) of
> these dependencies that should behave in a repeatable and known way for
> developers and CI environments both. Lastly, it enables those CI checks
> such that we can enforce Python coding quality checks via the CI tests.
>
> An auxiliary motivation is that this package is formatted in such a way
> that it COULD be uploaded to https://pypi.org/project/qemu and installed
> independently of qemu.git with `pip install qemu`, but that button
> remains *unpushed* and this series *will not* cause any such
> releases. We have time to debate finer points like API guarantees and
> versioning even after this series is merged.
>
> Other bits of interest
> ----------------------
>
> With the python tooling as a proper package, you can install this
> package in editable or production mode to a virtual environment, your
> local user environment, or your system packages. The primary benefit of
> this is to gain access to QMP tooling regardless of CWD, without needing
> to battle sys.path (and confounding other python analysis tools).
>
> For example: when developing, you may go to qemu/python/ and run `make
> venv` followed by `pipenv shell` to activate a virtual environment that
> contains the qemu python packages. These packages will always reflect
> the current version of the source files in the tree. When you are
> finished, you can simply exit the shell (^d) to remove these packages
> from your python environment.
>
> When not developing, you could install a version of this package to your
> environment outright to gain access to the QMP and QEMUMachine classes
> for lightweight scripting and testing by using pip: "pip install
> [--user] ."
>
> TESTING THIS SERIES
> ===================
>
> First of all, nothing should change. Without any intervention,
> everything should behave exactly as it did before. The only new
> information here comes from how to interact with and run the linters
> that will be enforcing code quality standards in this subdirectory.
>
> There are various invocations available that will test subtly different
> combinations using subtly different environments. I am assuming some
> light knowledge of Python environments and installing Python packages
> here. If you have questions, I would be delighted to answer them.
>
> To test the new tests, CD to ./python/ first, and then:
>
> 0. Try "make" or "make help" to get a sense of this series.
>
> 1. Try "make venv && pipenv shell" to get a venv with the package
> installed to it in editable mode. Ctrl+d exits this venv shell. While
> in this shell, any python script that uses "from qemu.[qmp|machine]
> import ..." should work correctly regardless of where the script is,
> or what your CWD is.
>
> This will pull some packages from PyPI and install them into the
> virtual environment, leaving your normal environment untouched.
>
> You will need Python 3.6 and pipenv installed on your system to do
> this step. For Fedora: "dnf install python36 pipenv" will do the
> trick. If you don't have this, skip down to #4 and onwards.
>
> 2. Try "make check" while still in the shell to run the Python linters
> using the venv built in the previous step. This will run avocado, which
> will in turn execute mypy, flake8, isort and pylint with the correct
> arguments.
>
> 3. Having exited the shell from above, try "make venv-check". This will
> create and update the venv if needed, then run 'make check' within the
> context of that shell. It should pass as long as the above did. You
> should be able to run "make distclean" prior to running "make
> venv-check" and have the entire process work start to finish.
>
> 4. Still outside of the venv, you may try running "make check". This
> will not install anything, but unless you have the right Python
> dependencies installed, these tests may fail for you. You might try
> using "pip install --user .[devel]" to install the development packages
> needed to run the tests successfully to your local user's python
> environment. Once done, you will probably want to "pip uninstall qemu"
> to remove the qemu packages to avoid them interfering with other things.
>
> 5. "make distclean" will delete the venv and any temporary files that
> may have been created by packaging, installing, testing, etc.
>
> 6. You may also (if you wish) create your own environment using a Python
> other than 3.6, bypassing pipenv. This may be useful for environments
> in which you simply do not have Python 3.6 readily available, or
> explicitly wish to test against a different version.
>
> Using the Python of your choice:
>
> - `> make distclean` (Remove the .venv, if you created one.)
> - `> /usr/bin/python3.9 -m venv .venv`
> - `> source .venv/bin/activate` (or activate.[fish|csh])
> - `> make develop` (Installs linter deps to this venv)
> - `> make check` (Runs tests using current env)
> - `> deactivate` (Leaves the venv)
>
> This *should* pass, and if it doesn't, it's a bug. Due to
> dependencies changing like shifting sands, it's impossible to promise
> support for every last combination of dependency and python
> version. This is why the "make venv-check" target uses Python 3 and a
> very explicit set of packages instead. Still, this form *should*
> pass.
>
> 7. If you are running Fedora or a distro that packages multiple Python
> versions, you can try running "make check-tox" to run the linters under
> all Python versions from Python 3.6 to 3.9. This will create a series of
> virtual environments under the .tox folder, leaving your existing
> environment unscathed. This should pass, provided you have the requisite
> Python versions on hand. When you are done, "make distclean" should
> remove any testing files.
>
> CHANGELOG
> =========
>
> V6
> --
>
> ```
> 001/25:[----] [--] 'iotests/297: add --namespace-packages to mypy arguments'
> 002/25:[----] [-C] 'python: create qemu packages'
> 003/25:[----] [--] 'python: create utils sub-package'
> 004/25:[0005] [FC] 'python: add qemu package installer'
> 005/25:[0002] [FC] 'python: add VERSION file'
> 006/25:[----] [--] 'python: add directory structure README.rst files'
> 007/25:[down] 'python: add MANIFEST.in'
> 008/25:[0001] [FC] 'python: Add pipenv support'
> 009/25:[----] [--] 'python: add pylint import exceptions'
> 010/25:[----] [--] 'python: move pylintrc into setup.cfg'
> 011/25:[0083] [FC] 'python: add pylint to pipenv'
> 012/25:[----] [--] 'python: move flake8 config to setup.cfg'
> 013/25:[----] [--] 'python: add excluded dirs to flake8 config'
> 014/25:[0038] [FC] 'python: Add flake8 to pipenv'
> 015/25:[----] [--] 'python: move mypy.ini into setup.cfg'
> 016/25:[0050] [FC] 'python: add mypy to pipenv'
> 017/25:[----] [--] 'python: move .isort.cfg into setup.cfg'
> 018/25:[0006] [FC] 'python/qemu: add isort to pipenv'
> 019/25:[0004] [FC] 'python/qemu: add qemu package itself to pipenv'
> 020/25:[0014] [FC] 'python: add devel package requirements to setuptools'
> 021/25:[down] 'python: add avocado-framework and tests'
> 022/25:[0015] [FC] 'python: add Makefile for some common tasks'
> 023/25:[0018] [FC] 'python: add .gitignore'
> 024/25:[0004] [FC] 'gitlab: add python linters to CI'
> 025/25:[down] 'python: add tox support'
> ```
>
> - 04: Changed maintainer blurb in PACKAGE.rst
> - 05: VERSION is now 0.6.1a0
> - 07: New, needed for both tox and using Pipenv to install the qemu
> package(s) in editable mode.
> - 08: Add README entries explaining the purpose of the pipenv files.
> - 11: pylint dependency was bumped to 2.7.0.
> - 14: pipenv changes.
> - 16: pipenv changes.
> - 18: isort pinned at >= 5.1.2 now; see commit message.
> - 19: pipenv changes.
> - 20: Context differences from isort, pylint, and pipenv changes.
> - 22:
> - Add 'make develop'
> - Add more README.rst entries.
> - Change pytest invocation to avocado.
> - 23:
> - Add comments to gitignore, split ignores out by function/category
> - Ignore vscode files. (I like vscode now!)
> - 24: Use the 'needs' feature of Gitlab CI and move the test to 'test'
> stage
>
> STATUS
> =======
>
> ```
> + [01] iotests-297-add-namespace # [RB] CR [SOB] JS
> + [02] python-create-qemu-packages # [SOB] JS
> + [03] python-create-utils-sub # [SOB] JS
> + [04] python-add-qemu-package # [SOB] JS
> + [05] python-add-version-file # [RB] CR [SOB] JS
> + [06] python-add-directory-structure # [SOB] JS
> + [07] python-add-manifest-in # [SOB] JS
> + [08] python-add-pipenv-support # [RB] CR [SOB] JS
> + [09] python-add-pylint-import # [RB] CR [SOB] JS
> + [10] python-move-pylintrc-into # [RB] CR [TB] CR [SOB] JS
> + [11] python-add-pylint-to-pipenv # [SOB] JS
> + [12] python-move-flake8-config-to # [RB] CR [SOB] JS
> + [13] python-add-excluded-dirs-to # [SOB] JS
> + [14] python-add-flake8-to-pipenv # [RB] CR [TB] CR [SOB] JS
> + [15] python-move-mypy-ini-into # [SOB] JS
> + [16] python-add-mypy-to-pipenv # [RB] CR [TB] CR [SOB] JS
> + [17] python-move-isort-cfg-into # [RB] CR [SOB] JS
> + [18] python-qemu-add-isort-to # [SOB] JS
> + [19] python-qemu-add-qemu-package # [RB] CR [TB] CR [SOB] JS
> + [20] python-add-devel-package # [SOB] JS
> + [21] python-add-avocado-framework # [SOB] JS
> + [22] python-add-makefile-for-some # [SOB] JS
> + [23] python-add-gitignore # [SOB] JS
> + [24] gitlab-add-python-linters-to # [SOB] JS
>> [25] python-add-tox-support # [SOB] JS
> ```
>
> John Snow (25):
> iotests/297: add --namespace-packages to mypy arguments
> python: create qemu packages
> python: create utils sub-package
> python: add qemu package installer
> python: add VERSION file
> python: add directory structure README.rst files
> python: add MANIFEST.in
> python: Add pipenv support
> python: add pylint import exceptions
> python: move pylintrc into setup.cfg
> python: add pylint to pipenv
> python: move flake8 config to setup.cfg
> python: add excluded dirs to flake8 config
> python: Add flake8 to pipenv
> python: move mypy.ini into setup.cfg
> python: add mypy to pipenv
> python: move .isort.cfg into setup.cfg
> python/qemu: add isort to pipenv
> python/qemu: add qemu package itself to pipenv
> python: add devel package requirements to setuptools
> python: add avocado-framework and tests
> python: add Makefile for some common tasks
> python: add .gitignore
> gitlab: add python linters to CI
> python: add tox support
>
> python/PACKAGE.rst | 33 +++
> python/README.rst | 53 +++++
> python/qemu/README.rst | 8 +
> python/qemu/machine/README.rst | 9 +
> python/qemu/qmp/README.rst | 9 +
> python/qemu/utils/README.rst | 7 +
> .gitlab-ci.yml | 12 +
> python/.gitignore | 20 ++
> python/MANIFEST.in | 2 +
> python/Makefile | 47 ++++
> python/Pipfile | 13 ++
> python/Pipfile.lock | 231 ++++++++++++++++++++
> python/VERSION | 1 +
> python/avocado.cfg | 10 +
> python/mypy.ini | 4 -
> python/qemu/.flake8 | 2 -
> python/qemu/.isort.cfg | 7 -
> python/qemu/__init__.py | 11 -
> python/qemu/machine/__init__.py | 36 +++
> python/qemu/{ => machine}/console_socket.py | 0
> python/qemu/{ => machine}/machine.py | 16 +-
> python/qemu/{ => machine}/qtest.py | 3 +-
> python/qemu/pylintrc | 58 -----
> python/qemu/{qmp.py => qmp/__init__.py} | 12 +-
> python/qemu/utils/__init__.py | 23 ++
> python/qemu/{ => utils}/accel.py | 0
> python/setup.cfg | 79 +++++++
> python/setup.py | 23 ++
> python/tests/flake8.sh | 2 +
> python/tests/isort.sh | 2 +
> python/tests/mypy.sh | 2 +
> python/tests/pylint.sh | 2 +
> python/tox.ini | 13 ++
> tests/acceptance/avocado_qemu/__init__.py | 4 +-
> tests/acceptance/virtio-gpu.py | 2 +-
> tests/docker/dockerfiles/fedora.docker | 2 +
> tests/qemu-iotests/297 | 1 +
> tests/qemu-iotests/300 | 4 +-
> tests/qemu-iotests/iotests.py | 2 +-
> tests/vm/aarch64vm.py | 2 +-
> tests/vm/basevm.py | 2 +-
> 41 files changed, 672 insertions(+), 97 deletions(-)
> create mode 100644 python/PACKAGE.rst
> create mode 100644 python/README.rst
> create mode 100644 python/qemu/README.rst
> create mode 100644 python/qemu/machine/README.rst
> create mode 100644 python/qemu/qmp/README.rst
> create mode 100644 python/qemu/utils/README.rst
> create mode 100644 python/.gitignore
> create mode 100644 python/MANIFEST.in
> create mode 100644 python/Makefile
> create mode 100644 python/Pipfile
> create mode 100644 python/Pipfile.lock
> create mode 100644 python/VERSION
> create mode 100644 python/avocado.cfg
> delete mode 100644 python/mypy.ini
> delete mode 100644 python/qemu/.flake8
> delete mode 100644 python/qemu/.isort.cfg
> delete mode 100644 python/qemu/__init__.py
> create mode 100644 python/qemu/machine/__init__.py
> rename python/qemu/{ => machine}/console_socket.py (100%)
> rename python/qemu/{ => machine}/machine.py (98%)
> rename python/qemu/{ => machine}/qtest.py (98%)
> delete mode 100644 python/qemu/pylintrc
> rename python/qemu/{qmp.py => qmp/__init__.py} (96%)
> create mode 100644 python/qemu/utils/__init__.py
> rename python/qemu/{ => utils}/accel.py (100%)
> create mode 100644 python/setup.cfg
> create mode 100755 python/setup.py
> create mode 100755 python/tests/flake8.sh
> create mode 100755 python/tests/isort.sh
> create mode 100755 python/tests/mypy.sh
> create mode 100755 python/tests/pylint.sh
> create mode 100644 python/tox.ini
>
prev parent reply other threads:[~2021-05-17 19:30 UTC|newest]
Thread overview: 64+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-12 23:12 [PATCH v6 00/25] python: create installable package John Snow
2021-05-12 23:12 ` [PATCH v6 01/25] iotests/297: add --namespace-packages to mypy arguments John Snow
2021-05-12 23:12 ` [PATCH v6 02/25] python: create qemu packages John Snow
2021-05-18 14:31 ` Cleber Rosa
2021-05-21 17:15 ` Willian Rampazzo
2021-05-12 23:12 ` [PATCH v6 03/25] python: create utils sub-package John Snow
2021-05-18 15:12 ` Cleber Rosa
2021-05-18 15:20 ` John Snow
2021-05-21 17:18 ` Willian Rampazzo
2021-05-21 17:39 ` John Snow
2021-05-12 23:12 ` [PATCH v6 04/25] python: add qemu package installer John Snow
2021-05-21 4:00 ` Cleber Rosa
2021-05-21 15:52 ` John Snow
2021-05-12 23:12 ` [PATCH v6 05/25] python: add VERSION file John Snow
2021-05-12 23:12 ` [PATCH v6 06/25] python: add directory structure README.rst files John Snow
2021-05-25 2:33 ` Cleber Rosa
2021-05-25 17:14 ` John Snow
2021-05-25 20:30 ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 07/25] python: add MANIFEST.in John Snow
2021-05-25 2:42 ` Cleber Rosa
2021-05-25 14:12 ` John Snow
2021-05-12 23:12 ` [PATCH v6 08/25] python: Add pipenv support John Snow
2021-05-12 23:12 ` [PATCH v6 09/25] python: add pylint import exceptions John Snow
2021-05-12 23:12 ` [PATCH v6 10/25] python: move pylintrc into setup.cfg John Snow
2021-05-12 23:12 ` [PATCH v6 11/25] python: add pylint to pipenv John Snow
2021-05-25 3:33 ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 12/25] python: move flake8 config to setup.cfg John Snow
2021-05-12 23:12 ` [PATCH v6 13/25] python: add excluded dirs to flake8 config John Snow
2021-05-25 15:50 ` Cleber Rosa
2021-05-25 17:18 ` John Snow
2021-05-12 23:12 ` [PATCH v6 14/25] python: Add flake8 to pipenv John Snow
2021-05-12 23:12 ` [PATCH v6 15/25] python: move mypy.ini into setup.cfg John Snow
2021-05-25 15:52 ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 16/25] python: add mypy to pipenv John Snow
2021-05-12 23:12 ` [PATCH v6 17/25] python: move .isort.cfg into setup.cfg John Snow
2021-05-12 23:12 ` [PATCH v6 18/25] python/qemu: add isort to pipenv John Snow
2021-05-25 15:56 ` Cleber Rosa
2021-05-25 17:21 ` John Snow
2021-05-25 20:34 ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 19/25] python/qemu: add qemu package itself " John Snow
2021-05-12 23:12 ` [PATCH v6 20/25] python: add devel package requirements to setuptools John Snow
2021-05-25 16:13 ` Cleber Rosa
2021-05-25 17:43 ` John Snow
2021-05-25 20:38 ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 21/25] python: add avocado-framework and tests John Snow
2021-05-25 18:58 ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 22/25] python: add Makefile for some common tasks John Snow
2021-05-25 19:24 ` Cleber Rosa
2021-05-25 19:45 ` John Snow
2021-05-25 20:39 ` Cleber Rosa
2021-05-12 23:12 ` [PATCH v6 23/25] python: add .gitignore John Snow
2021-05-25 19:36 ` Cleber Rosa
2021-05-25 20:10 ` John Snow
2021-05-25 20:42 ` Cleber Rosa
2021-05-25 23:54 ` John Snow
2021-05-12 23:12 ` [PATCH v6 24/25] gitlab: add python linters to CI John Snow
2021-05-25 19:55 ` Cleber Rosa
2021-05-25 20:33 ` John Snow
2021-05-12 23:12 ` [PATCH v6 25/25] python: add tox support John Snow
2021-05-25 20:15 ` Cleber Rosa
2021-05-25 20:25 ` John Snow
2021-05-25 20:46 ` Cleber Rosa
2021-05-25 22:15 ` John Snow
2021-05-17 18:52 ` John Snow [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d59b6295-58a4-90bd-f898-75d8fc5326ed@redhat.com \
--to=jsnow@redhat.com \
--cc=alex.bennee@linaro.org \
--cc=bleal@redhat.com \
--cc=crosa@redhat.com \
--cc=ehabkost@redhat.com \
--cc=f4bug@amsat.org \
--cc=fam@euphon.net \
--cc=kwolf@redhat.com \
--cc=mreitz@redhat.com \
--cc=philmd@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=thuth@redhat.com \
--cc=wainersm@redhat.com \
--cc=willianr@redhat.com \
--cc=wrampazz@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).