Re: [PATCH v2 03/15] python: add VERSION file

[John Snow <jsnow@redhat.com>]

On 10/19/20 6:02 AM, Daniel P. Berrangé wrote:
> On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote:
>> On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote:
>>> Python infrastructure as it exists today is not capable reliably of
>>> single-sourcing a package version from a parent directory. The authors
>>> of pip are working to correct this, but as of today this is not possible
>>> to my knowledge.
>>>
>>> The problem is that when using pip to build and install a python
>>> package, it copies files over to a temporary directory and performs its
>>> build there. This loses access to any information in the parent
>>> directory, including git itself.
>>>
>>> Further, Python versions have a standard (PEP 440) that may or may not
>>> follow QEMU's versioning. In general, it does; but naturally QEMU does
>>> not follow PEP 440. To avoid any automatically-generated conflict, a
>>> manual version file is preferred.
>>>
>>>
>>> I am proposing:
>>>
>>> - Python core tooling synchronizes with the QEMU version directly
>>>    (5.2.0, 5.1.1, 5.3.0, etc.)
>>>
>>> - In the event that a Python package needs to be updated independently
>>>    of the QEMU version, a pre-release alpha version should be preferred,
>>>    but *only* after inclusion to the qemu development or stable branches.
>>>
>>>    e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's
>>>    release.
>>>
>>> - The Python core tooling makes absolutely no version compatibility
>>>    checks or constraints. It *may* work with releases of QEMU from the
>>>    past or future, but it is not required to.
>>>
>>>    i.e., "qemu.core" will always remain in lock-step with QEMU.
>>>
>>> - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp
>>>    and begin indepedently versioning such a package separately from the
>>>    QEMU version it accompanies.
>>
>> I think this need to be considered very carefully.
>>
>> I'm not overly familiar with the Python ecosystem but it would appear
>> that, despite PEP 440 not mandating this, many (most?) of the
>> packages uploaded to PyPi are using semantic versioning.
> 
>    https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme
> 
> Semver is the recommended approach, but they explicitly list date
> based versioning as a valid alternative
> 
>    "Semantic versioning is not a suitable choice for all projects,
>     such as those with a regular time based release cadence and a
>     deprecation process that provides warnings for a number of
>     releases prior to removal of a feature."
> 
> That paragraph describes QEMU's scenario.
> 
> NB, historically we've made arbitrary changes to the python code
> since it was not considered public API. If we make it official
> public API, then we would actually need to start following our
> deprecation process for the python code too.
> 

I think our deprecation process is not tightly compatible with how 
Python programmers at-large expect packages to work. Semver is more or 
less the norm, despite the fact that it isn't explicitly required.

setting requirements in requirements.txt, setup.[cfg|py], Pipfile, etc 
often hinge on a major version, e.g.

qemu >= 5.0
qemu >= 3.0, < 6.0

would both be common forms of describing a requirement.

Pinning specific versions is considered bad form, but in the context of 
releasing a package, I often see maintainers hedging their bets and 
preventing upgrades across a major version line.

For that reason I am a little weary of adopting the deprecation policy 
as it exists in QEMU directly, and would propose a modification for my 
purposes here:

- Features must be marked as deprecated
- They must remain in a deprecated state for [at least] 2 releases
- Deprecated features may not be removed until a major version change.

In practice, this modification is a change from "2 releases" to "at 
least 2".

However, I didn't intend to pay any mind to the deprecation policy 
"yet", as I have the package metadata listing the package status as 
"Alpha", see below:

>> With that in mind, I think it would be unwise for qemu.* not to do
>> the same; in particular, using a version number that's not <1.0.0 for
>> a package that is very much in flux will almost certainly break
>> people's expectations, and is also not something that you can easily
>> take back at a later time.
> 
> I don't think it is that big a deal, and there is clear benefit to
> having the python code version match the QEMU version that it is
> the companioon to.
> 

Do you think it's fine if I start versioning at, say, "0.5.2", I could 
ignore a deprecation policy for now? I have a lot of changes I want to 
make and expect a lot of breaking changes very quickly.

I just wanted to try -- somehow -- to conjure up a relationship to the 
QEMU package it's designed to work with.

> Ultimately the versioning scheme just impacts on the version string
> conditionals people list for their dependancies. Apps consuming QEMU
> can handle any of the version schemes without much difference.
> 
> Regards,
> Daniel
> 

Thanks for your input. This is the trickiest part of the process for me.

I believe there is value in distributing these tools for other 
developers to help them prototype and experiment with new features, but 
realize it's a tightrope walk because we're flying dangerously close to 
providing a management utility that needs to care about cross-version 
compatibility and so on.

I have no interest in providing stringent cross-version compatibility, 
but at the same time, libraries like QMP are not really at risk of 
changing all that much, actually. It will *probably* work for most QEMU 
versions.

I have some further patches where I clean up the scripts in 
./scripts/qmp and move them to ./python/qemu/qmp -- and that work is 
making me consider a rework to this series. I think that rework matches 
the spirit of an earlier suggestion of yours:

- move ./python/qemu/core/qmp.py to ./python/qemu/qmp/qmp.py
- (in a follow-up series) move ./scripts/qmp/* to ./python/qemu/qmp/*
- rename ./python/qemu/core/ to ./python/qemu/machine
     (move accel.py, machine.py and qtest.py to a 'machine' pkg.)


In effect, we'd then have:

qemu.machine -- primarily a test interface for QEMU instances
qemu.qmp -- fairly ageless QMP tools/lib for talking to QEMU

and the different levels of support and "use at your own risk" become 
slightly more clear between the two packages.

--js