Re: [Qemu-devel] [RFC 17/18] validator.py script

From: Eduardo Habkost <ehabkost@redhat.com>
To: Markus Armbruster <armbru@redhat.com>
Cc: qemu-devel@nongnu.org, "Thomas Huth" <thuth@redhat.com>,
	"Amador Pahim" <apahim@redhat.com>,
	"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
	"Cleber Rosa" <crosa@redhat.com>,
	"Marcel Apfelbaum" <marcel@redhat.com>,
	"Paolo Bonzini" <pbonzini@redhat.com>
Subject: Re: [Qemu-devel] [RFC 17/18] validator.py script
Date: Tue, 17 Apr 2018 11:42:56 -0300	[thread overview]
Message-ID: <20180417144256.GW29865@localhost.localdomain> (raw)
In-Reply-To: <87vacqf3da.fsf@dusky.pond.sub.org>

On Tue, Apr 17, 2018 at 02:01:53PM +0200, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > See cover letter for a description of the new test system.
> >
> > TODO: code clean up
> > TODO: write description.
> >
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> >  scripts/validator.py | 724 +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 724 insertions(+)
> >  create mode 100755 scripts/validator.py
> >
> > diff --git a/scripts/validator.py b/scripts/validator.py
> > new file mode 100755
> > index 0000000000..4312571feb
> > --- /dev/null
> > +++ b/scripts/validator.py
> > @@ -0,0 +1,724 @@
> > +#!/usr/bin/env python
> > +#
> > +#  Copyright (c) 2018 Red Hat Inc
> > +#
> > +# Author:
> > +#  Eduardo Habkost <ehabkost@redhat.com>
> > +#
> > +# This program is free software; you can redistribute it and/or modify
> > +# it under the terms of the GNU General Public License as published by
> > +# the Free Software Foundation; either version 2 of the License, or
> > +# (at your option) any later version.
> > +#
> > +# This program is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License along
> > +# with this program; if not, write to the Free Software Foundation, Inc.,
> > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> > +
> > +"""
> > +QEMU validator script
> > +=====================
> > +
> > +This script will get test YAML test case specifications or Python
> > +modules as input, and generate/run test cases based on them.
> > +
> > +USAGE
> > +-----
> > +
> > +validator.py <specification-file>... -V VAR1=value1 VAR1=value2 VAR2=value3
> > +
> > +specification-file is a YAML file containing the test specification.
> 
> I can see the "test YAML test case specifications", but not the "Python
> modules".
> 
> Should we introduce yet another markup language into QEMU?  (pardon the
> pun).

Fair question.  What are the existing markup languages in QEMU we
could use?

JSON is an option, but I believe YAML is more readable.

> 
> > +
> > +Example::
> > +
> > +    # this test specification is equivalent to the
> > +    # "device/introspect/list" test case in device-introspect-test.c
> 
> If we merge this, not for long :)
> 
> > +    command-line: '$QEMU -nodefaults -machine none'
> > +    monitor-commands:
> > +    - qmp:
> > +      - execute: qom-list-types
> > +        arguments:
> > +          implements: 'device'
> > +          abstract: true
> > +    - hmp: 'device_add help'
> > +
> > +
> > +VARIABLE EXPANSION
> > +------------------
> > +
> > +The test runner will try to run the test cases with all possible values
> > +for variables appearing in the test specification.
> > +
> > +Some built-in variables are automatically expanded:
> > +
> > +* `$MACHINE` - Expands to a machine-type name supported by $QEMU
> > +* `$ACCEL` - Expands to an accelerator name supported by $QEMU
> > +* `$DEVICE` - Expands to a (user-creatable) device type name supported by $QEMU
> > +* `$CPU` - Expands to a CPU model name supported by $QEMU
> > +
> > +Note that the $QEMU variable must be specified in th
> 
> Yes?

Heh.  It must be specified in the command-line.  I will fix that.

> 
> > +
> > +TEST SPECIFICATION FIELDS
> > +-------------------------
> > +
> > +command-line
> > +~~~~~~~~~~~~
> > +
> > +List or string, containing the QEMU command-line to be run.
> > +
> > +Default: '$QEMU'
> > +
> > +
> > +monitor-commands
> > +~~~~~~~~~~~~~~~~
> > +
> > +Mapping or list-of-mappings containing monitor commands to run.  The key on each
> > +item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
> > +mapping, or list.
> > +
> > +Default: None.
> > +
> > +TODO: not implemented yet.
> 
> The whole monitor-commands feature?

Outdated comment, sorry.

> 
> Can I do
> 
>     monitor-commands:
>     - qmp:
>       - execute: eins
>     - hmp: zwei
>     - qmp:
>       - execute: drei
> 
> to execute eins, zwei, drei in this order?

Yes, it can.  See the test specification examples.

> 
> > +
> > +
> > +qmp
> > +~~~
> > +
> > +Boolean.  If true (the default), a QMP monitor is configured on the command-line
> > +automatically.
> > +
> > +If true, the test runner will issue a ``quit`` command automatically when
> > +testing is finished.  If false, the test runner will wait until QEMU exits by
> > +itself.
> > +
> > +Example::
> > +
> > +    # just run $QEMU -help and ensure it won't crash
> > +    command-line: ['$QEMU', '-help']
> > +    qmp: false
> > +
> > +
> > +TODO: whitelist
> > +TODO: validate output against reference output
> > +TODO: configure defaults for variables
> > +TODO: compatibility with Avocado multiplexer?
> > +"""
> 
> This is a DSL to write tests.  I applaud the idea to write more tests at
> a higher level than C.  We just need to be careful not to create too
> many different ways to write tests, or else readability will suffer.
> Ideally, the way to use for a test should be fairly obvious.

Agreed we need to keep this in mind.

Replacing the DSL with very short and simple Python code is a
possibility I was considering, but I'm not sure yet if this is
the way to go.  I don't want this to create a whole new test
framework.

> 
> Looking forward to your next iteration.

Thanks!

-- 
Eduardo