From: John Snow <jsnow@redhat.com>
To: Markus Armbruster <armbru@redhat.com>
Cc: michael.roth@amd.com, qemu-devel@nongnu.org, marcandre.lureau@redhat.com
Subject: Re: [PATCH 10/28] qapi: Rework name checking in preparation of stricter checking
Date: Thu, 25 Mar 2021 13:48:34 -0400 [thread overview]
Message-ID: <6b74a262-10c1-a857-00dd-736d29eec23f@redhat.com> (raw)
In-Reply-To: <87im5fah92.fsf@dusky.pond.sub.org>
On 3/25/21 2:18 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
>
>> On 3/24/21 1:57 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> On 3/23/21 5:40 AM, Markus Armbruster wrote:
>>>>> Naming rules differ for the various kinds of names. To prepare
>>>>> enforcing them, define functions to check them: check_name_upper(),
>>>>> check_name_lower(), and check_name_camel(). For now, these merely
>>>>> wrap around check_name_str(), but that will change shortly. Replace
>>>>> the other uses of check_name_str() by appropriate uses of the
>>>>> wrappers. No change in behavior just yet.
>>>>> Signed-off-by: Markus Armbruster <armbru@redhat.com>
>>>>> ---
>>>>> scripts/qapi/expr.py | 51 +++++++++++++++++++++++++++++++-------------
>>>>> 1 file changed, 36 insertions(+), 15 deletions(-)
>>>>> diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
>>>>> index e00467636c..30285fe334 100644
>>>>> --- a/scripts/qapi/expr.py
>>>>> +++ b/scripts/qapi/expr.py
>>>>> @@ -21,11 +21,12 @@
>>>>> from .error import QAPISemError
>>>>> -# Names must be letters, numbers, -, and _. They must start
>>>>> with letter,
>>>>> -# except for downstream extensions which must start with __RFQDN_.
>>>>> -# Dots are only valid in the downstream extension prefix.
>>>>> -valid_name = re.compile(r'^(__[a-zA-Z0-9.-]+_)?'
>>>>> - '[a-zA-Z][a-zA-Z0-9_-]*$')
>>>>> +# Names consist of letters, digits, -, and _, starting with a letter.
>>>>> +# An experimental name is prefixed with x-. A name of a downstream
>>>>> +# extension is prefixed with __RFQDN_. The latter prefix goes first.
>>>>> +valid_name = re.compile(r'(__[a-z0-9.-]+_)?'
>>>>> + r'(x-)?'
>>>>> + r'([a-z][a-z0-9_-]*)$', re.IGNORECASE)
>>>>> def check_name_is_str(name, info, source):
>>>>> @@ -37,16 +38,38 @@ def check_name_str(name, info, source,
>>>>> permit_upper=False):
>>>>> # Reserve the entire 'q_' namespace for c_name(), and for 'q_empty'
>>>>> # and 'q_obj_*' implicit type names.
>>>>> - if not valid_name.match(name) or \
>>>>> - c_name(name, False).startswith('q_'):
>>>>> + match = valid_name.match(name)
>>>>> + if not match or c_name(name, False).startswith('q_'):
>>>>> raise QAPISemError(info, "%s has an invalid name" % source)
>>>>> if not permit_upper and name.lower() != name:
>>>>> raise QAPISemError(
>>>>> info, "%s uses uppercase in name" % source)
>>>>> + return match.group(3)
>>>>> +
>>>>> +
>>>>> +def check_name_upper(name, info, source):
>>>>> + stem = check_name_str(name, info, source, permit_upper=True)
>>>>> + # TODO reject '[a-z-]' in @stem
>>>>> +
>>>>
>>>> Creates (presumably) temporary errors in flake8 for the dead
>>>> assignment here and below.
>>>
>>> All gone by the end of the series.
>>>
>>> "make check" and checkpatch were content. Anything else you'd like me
>>> to run?
>>
>> Eventually it'll be part of CI, with targets to run locally.
>>
>> I never expected the process to take this long, so I did not invest my
>> time in developing an interim solution.
>>
>> I use a hastily written script to do my own testing, which I run for
>> every commit that touches QAPI:
>>
>> #!/usr/bin/env bash
>> set -e
>>
>> if [[ -f qapi/.flake8 ]]; then
>> echo "flake8 --config=qapi/.flake8 qapi/"
>> flake8 --config=qapi/.flake8 qapi/
>> fi
>> if [[ -f qapi/pylintrc ]]; then
>> echo "pylint --rcfile=qapi/pylintrc qapi/"
>> pylint --rcfile=qapi/pylintrc qapi/
>> fi
>> if [[ -f qapi/mypy.ini ]]; then
>> echo "mypy --config-file=qapi/mypy.ini qapi/"
>> mypy --config-file=qapi/mypy.ini qapi/
>> fi
>>
>> if [[ -f qapi/.isort.cfg ]]; then
>> pushd qapi
>> echo "isort -c ."
>> isort -c .
>> popd
>> fi
>>
>> pushd ../bin/git
>> make -j9
>> make check-qapi-schema
>> popd
>
> Thanks for sharing this!
>
My intent, eventually, was to get scripts/qapi under python/qemu/qapi;
under which there is a testing framework thing I have been debating on
and off with Cleber for a while that does (basically) the same thing as
what my hasty script does, but more integrated with Python ecosystem.
It'll do a few things:
(1) Gets these checks on CI
(2) Allows developers to manually run the checks locally
(3) Allows developers to manually run the checks locally using a
pre-determined set of pinned linter versions to guarantee synchronicity
with CI
cd python/qemu && "make check" would do more or less the same as the
hacky script above, "make venv-check" would create a virtual environment
with precisely the right packages and then run the same.
I have been stalled there for a while, and I missed freeze deadline from
illness. Anguish. For now, the dumb script in my scripts directory does
the lifting I need it to, using packages I have selected in my local
environment that just-so-happen to pass.
> Apropos qapi-gen testing scripts. I have scripts to show me how the
> generated code changes along the way in a branch. They evolved over a
> long time, and try to cope with changes in the tree that are hardly
> relevant anymore. By now, they could quite possibly make Frankenstein
> recoil in horror.
>
Are they in the tree? Largely if the generated code changes it's
invisible to me, but I rely heavily on the unit tests. I guess maybe if
they are not in a state to upstream it might not be worth the hassle to
clean them, but I don't know.
> As a secondary purpose, the scripts show me how output of pycodestyle-3
> and pylint change. This would be uninteresting if the code in master
> was clean against a useful configuration of these tools. Your work has
> been making it less interesting.
>
--js
next prev parent reply other threads:[~2021-03-25 17:52 UTC|newest]
Thread overview: 93+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-23 9:39 [PATCH 00/28] qapi: Enforce naming rules Markus Armbruster
2021-03-23 9:39 ` [PATCH 01/28] qapi/pragma: Tidy up after removal of deprecated commands Markus Armbruster
2021-03-23 12:50 ` John Snow
2021-03-23 9:39 ` [PATCH 02/28] tests/qapi-schema: Drop redundant flat-union-inline test Markus Armbruster
2021-03-23 12:54 ` John Snow
2021-03-23 9:40 ` [PATCH 03/28] tests/qapi-schema: Rework comments on longhand member definitions Markus Armbruster
2021-03-23 13:00 ` John Snow
2021-03-23 13:58 ` Eric Blake
2021-03-23 14:25 ` John Snow
2021-03-23 13:59 ` Eric Blake
2021-03-23 14:27 ` John Snow
2021-03-23 9:40 ` [PATCH 04/28] tests/qapi-schema: Belatedly update comment on alternate clash Markus Armbruster
2021-03-23 13:12 ` John Snow
2021-03-23 9:40 ` [PATCH 05/28] tests/qapi-schema: Drop TODO comment on simple unions Markus Armbruster
2021-03-23 13:16 ` John Snow
2021-03-23 9:40 ` [PATCH 06/28] tests/qapi-schema: Tweak to demonstrate buggy member name check Markus Armbruster
2021-03-23 13:20 ` John Snow
2021-03-23 15:44 ` Markus Armbruster
2021-03-23 17:09 ` John Snow
2021-03-23 20:42 ` Markus Armbruster
2021-03-23 9:40 ` [PATCH 07/28] qapi: Fix to reject optional members with reserved names Markus Armbruster
2021-03-23 13:27 ` John Snow
2021-03-23 15:50 ` Markus Armbruster
2021-03-23 9:40 ` [PATCH 08/28] qapi: Support flat unions tag values with leading digit Markus Armbruster
2021-03-23 14:11 ` Eric Blake
2021-03-23 14:49 ` John Snow
2021-03-23 16:18 ` Markus Armbruster
2021-03-23 21:07 ` Markus Armbruster
2021-03-23 9:40 ` [PATCH 09/28] qapi: Lift enum-specific code out of check_name_str() Markus Armbruster
2021-03-23 14:13 ` Eric Blake
2021-03-23 21:44 ` John Snow
2021-03-23 22:11 ` John Snow
2021-03-24 5:55 ` Markus Armbruster
2021-03-23 9:40 ` [PATCH 10/28] qapi: Rework name checking in preparation of stricter checking Markus Armbruster
2021-03-23 14:20 ` Eric Blake
2021-03-23 14:30 ` John Snow
2021-03-23 14:40 ` Eric Blake
2021-03-23 16:25 ` Markus Armbruster
2021-03-23 21:14 ` Markus Armbruster
2021-03-23 22:15 ` John Snow
2021-03-24 5:57 ` Markus Armbruster
2021-03-24 20:11 ` John Snow
2021-03-25 6:18 ` Markus Armbruster
2021-03-25 17:48 ` John Snow [this message]
2021-03-26 5:25 ` Markus Armbruster
2021-03-23 9:40 ` [PATCH 11/28] qapi: Move uppercase rejection to check_name_lower() Markus Armbruster
2021-03-23 14:29 ` Eric Blake
2021-03-23 22:21 ` John Snow
2021-03-23 9:40 ` [PATCH 12/28] qapi: Consistently permit any case in downstream prefixes Markus Armbruster
2021-03-23 14:30 ` Eric Blake
2021-03-23 22:26 ` John Snow
2021-03-23 9:40 ` [PATCH 13/28] qapi: Enforce event naming rules Markus Armbruster
2021-03-23 14:32 ` Eric Blake
2021-03-23 22:31 ` John Snow
2021-03-24 6:22 ` Markus Armbruster
2021-03-24 20:07 ` John Snow
2021-03-25 6:22 ` Markus Armbruster
2021-03-25 17:50 ` John Snow
2021-03-23 9:40 ` [PATCH 14/28] qapi: Enforce type " Markus Armbruster
2021-03-23 14:50 ` Eric Blake
2021-03-23 16:27 ` Markus Armbruster
2021-03-23 9:40 ` [PATCH 15/28] tests/qapi-schema: Rename redefined-builtin to redefined-predefined Markus Armbruster
2021-03-23 14:55 ` Eric Blake
2021-03-23 9:40 ` [PATCH 16/28] qapi: Factor out QAPISchemaParser._check_pragma_list_of_str() Markus Armbruster
2021-03-23 15:01 ` Eric Blake
2021-03-23 9:40 ` [PATCH 17/28] tests/qapi-schema: Rename pragma-*-crap to pragma-value-not-* Markus Armbruster
2021-03-23 15:02 ` Eric Blake
2021-03-23 9:40 ` [PATCH 18/28] tests/qapi-schema: Rename returns-whitelist to returns-bad-type Markus Armbruster
2021-03-23 15:06 ` Eric Blake
2021-03-23 9:40 ` [PATCH 19/28] qapi: Rename pragma *-whitelist to *-exceptions Markus Armbruster
2021-03-23 15:09 ` Eric Blake
2021-03-23 16:35 ` Markus Armbruster
2021-03-23 9:40 ` [PATCH 20/28] qapi/pragma: Streamline comments on member-name-exceptions Markus Armbruster
2021-03-23 15:10 ` Eric Blake
2021-03-23 9:40 ` [PATCH 21/28] tests-qmp-cmds: Drop unused and incorrect qmp_TestIfCmd() Markus Armbruster
2021-03-23 15:11 ` Eric Blake
2021-03-23 9:40 ` [PATCH 22/28] qapi: Prepare for rejecting underscore in command and member names Markus Armbruster
2021-03-23 15:15 ` Eric Blake
2021-03-23 9:40 ` [PATCH 23/28] qapi: Enforce feature naming rules Markus Armbruster
2021-03-23 15:16 ` Eric Blake
2021-03-23 9:40 ` [PATCH 24/28] qapi: Enforce command " Markus Armbruster
2021-03-23 15:23 ` Eric Blake
2021-03-23 21:19 ` Markus Armbruster
2021-03-23 9:40 ` [PATCH 25/28] tests/qapi-schema: Switch member name clash test to struct Markus Armbruster
2021-03-23 15:42 ` Eric Blake
2021-03-23 9:40 ` [PATCH 26/28] qapi: Enforce struct member naming rules Markus Armbruster
2021-03-23 15:46 ` Eric Blake
2021-03-23 21:23 ` Markus Armbruster
2021-03-23 9:40 ` [PATCH 27/28] qapi: Enforce enum " Markus Armbruster
2021-03-23 15:47 ` Eric Blake
2021-03-23 9:40 ` [PATCH 28/28] qapi: Enforce union and alternate branch " Markus Armbruster
2021-03-23 16:05 ` Eric Blake
2021-03-23 21:24 ` Markus Armbruster
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6b74a262-10c1-a857-00dd-736d29eec23f@redhat.com \
--to=jsnow@redhat.com \
--cc=armbru@redhat.com \
--cc=marcandre.lureau@redhat.com \
--cc=michael.roth@amd.com \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).