[Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Stefan Hajnoczi]

There is not much getting started documentation for qemu-iotests.  This
patch explains how to create a new test and covers the overall testing
approach.

Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 tests/qemu-iotests/README | 83 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)

diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README
index 6079b40..8259b9f 100644
--- a/tests/qemu-iotests/README
+++ b/tests/qemu-iotests/README
@@ -14,8 +14,91 @@ Just run ./check to run all tests for the raw image format, or ./check
 -qcow2 to test the qcow2 image format.  The output of ./check -h explains
 additional options to test further image formats or I/O methods.
 
+* Testing approach
+
+Each test is an executable file (usually a bash script) that is run by the
+./check test harness.  Standard out and standard error are captured to an
+output file.  If the output file differs from the "golden master" output file
+for the test then it fails.
+
+Tests are simply a sequence of commands that produce output and the test itself
+does not judge whether it passed or failed.  If you find yourself writing
+checks to determine success or failure then you should rethink the test and
+rely on output diffing instead.
+
+** Filtering volatile output
+
+When output contains absolute file paths, timestamps, process IDs, hostnames,
+or other volatile strings, the diff against golden master output will fail.
+Such output must be filtered to replace volatile strings with fixed
+placeholders.
+
+For example, the path to the temporary working directory changes between test
+runs so it must be filtered:
+
+  sed -e "s#$TEST_DIR/#TEST_DIR/#g"
+
+Commonly needed filters are available in ./common.filter.
+
+** Python tests
+
+Most tests are implemented in bash but it is difficult to interact with the QMP
+monitor.  A Python module called 'iotests' is available for tests that require
+JSON and interacting with QEMU.
+
+* How to create a test
+
+1. Choose an unused test number
+
+Tests are identified by a unique number.  Look for the highest test case number
+by looking at the test files.  Then search the qemu-devel@nongnu.org mailing
+list to check if anyone has already sent patches using the next available
+number.  You may need to increment the number a few times to reach an unused
+number.
+
+2. Create the test file
+
+Copy an existing test (one that most closely resembles what you wish to test)
+to the new test number:
+
+  cp 001 <test-number>
+
+3. Assign groups to the test
+
+Add your test to the ./group file.  This file is the index of tests and assigns
+them to functional groups like "rw" for read-write tests.  Most tests belong to
+the "rw" and "auto" groups.  "auto" means the test runs when ./check is invoked
+without a -g argument.
+
+Consider adding your test to the "quick" group if it executes quickly (<1s).
+This group is run by "make check-block" and is often included as part of build
+tests in continuous integration systems.
+
+4. Write the test
+
+Edit the test script.  Look at existing tests for examples.
+
+5. Generate the golden master file
+
+Run your test with "./check <test-number>".  You may need to pass additional
+options to use an image format or protocol.
+
+The test will fail because there is no golden master yet.  Inspect the output
+that your test generated with "cat <test-number>.out.bad".
+
+Verify that the output is as expected and contains no volatile strings like
+timestamps.  You may need to add filters to your test to remove volatile
+strings.
+
+Once you are happy with the test output it can be used as the golden master
+with "mv <test-number>.out.bad <test-number>.out".  Rerun the test to verify
+that it passes.
+
+Congratulations, you've created a new test!
+
 * Feedback and patches
 
 Please send improvements to the test suite, general feedback or just
 reports of failing tests cases to qemu-devel@nongnu.org with a CC:
 to qemu-block@nongnu.org.
+
-- 
2.9.4

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1750 bytes --]

On 07/21/2017 04:34 AM, Stefan Hajnoczi wrote:
> There is not much getting started documentation for qemu-iotests.  This
> patch explains how to create a new test and covers the overall testing
> approach.
> 
> Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---

> +3. Assign groups to the test
> +
> +Add your test to the ./group file.  This file is the index of tests and assigns
> +them to functional groups like "rw" for read-write tests.  Most tests belong to
> +the "rw" and "auto" groups.  "auto" means the test runs when ./check is invoked
> +without a -g argument.
> +
> +Consider adding your test to the "quick" group if it executes quickly (<1s).

We have several tests going up to 5s (and I have a patch pending to
remove two tests that took longer) - I think 1s is a bit on the short
end for still classifying a test as quick.

> +This group is run by "make check-block" and is often included as part of build
> +tests in continuous integration systems.

It would still be nice to have 'make check' run 'make check-block'...
but that's independent of this patch.

> +Once you are happy with the test output it can be used as the golden master
> +with "mv <test-number>.out.bad <test-number>.out".  Rerun the test to verify
> +that it passes.
> +
> +Congratulations, you've created a new test!

Maybe a reminder to 'git add' the new files, then submit the patch?

Better than what we have, so whether or not you make further tweaks
according to my suggestions,
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]

On Fri, Jul 21, 2017 at 07:16:34AM -0500, Eric Blake wrote:
> On 07/21/2017 04:34 AM, Stefan Hajnoczi wrote:
> > There is not much getting started documentation for qemu-iotests.  This
> > patch explains how to create a new test and covers the overall testing
> > approach.
> > 
> > Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > ---
> 
> > +3. Assign groups to the test
> > +
> > +Add your test to the ./group file.  This file is the index of tests and assigns
> > +them to functional groups like "rw" for read-write tests.  Most tests belong to
> > +the "rw" and "auto" groups.  "auto" means the test runs when ./check is invoked
> > +without a -g argument.
> > +
> > +Consider adding your test to the "quick" group if it executes quickly (<1s).
> 
> We have several tests going up to 5s (and I have a patch pending to
> remove two tests that took longer) - I think 1s is a bit on the short
> end for still classifying a test as quick.

I'm happy to accept any number blessed by Kevin.  I do think that 1
second is a safe maximum and no one should get in trouble for adding a
test that takes 1 second to the "quick" group.

> > +This group is run by "make check-block" and is often included as part of build
> > +tests in continuous integration systems.
> 
> It would still be nice to have 'make check' run 'make check-block'...
> but that's independent of this patch.

Yes, there is a separate discussion about that on the list right now.
Hopefully it will be added back.

> > +Once you are happy with the test output it can be used as the golden master
> > +with "mv <test-number>.out.bad <test-number>.out".  Rerun the test to verify
> > +that it passes.
> > +
> > +Congratulations, you've created a new test!
> 
> Maybe a reminder to 'git add' the new files, then submit the patch?

I thought about this too but decided not to get into the git and patch
submission business.  I carefully worded it to be about "creating" tests
rather than "adding" them to qemu.git because I wanted to limit the
scope of this README :).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [Qemu-block] [PATCH] qemu-iotests: add a "how to" to ./README

[Manos Pitsidianakis]

[-- Attachment #1: Type: text/plain, Size: 4690 bytes --]

On Fri, Jul 21, 2017 at 10:34:16AM +0100, Stefan Hajnoczi wrote:
>There is not much getting started documentation for qemu-iotests.  This
>patch explains how to create a new test and covers the overall testing
>approach.
>
>Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
>Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>---
> tests/qemu-iotests/README | 83 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 83 insertions(+)
>
>diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README
>index 6079b40..8259b9f 100644
>--- a/tests/qemu-iotests/README
>+++ b/tests/qemu-iotests/README
>@@ -14,8 +14,91 @@ Just run ./check to run all tests for the raw image format, or ./check
> -qcow2 to test the qcow2 image format.  The output of ./check -h explains
> additional options to test further image formats or I/O methods.
>
>+* Testing approach
>+
>+Each test is an executable file (usually a bash script) that is run by the
>+./check test harness.  Standard out and standard error are captured to an
>+output file.  If the output file differs from the "golden master" output file
>+for the test then it fails.
>+
>+Tests are simply a sequence of commands that produce output and the test itself
>+does not judge whether it passed or failed.  If you find yourself writing
>+checks to determine success or failure then you should rethink the test and
>+rely on output diffing instead.
>+
>+** Filtering volatile output
>+
>+When output contains absolute file paths, timestamps, process IDs, hostnames,
>+or other volatile strings, the diff against golden master output will fail.
>+Such output must be filtered to replace volatile strings with fixed
>+placeholders.
>+
>+For example, the path to the temporary working directory changes between test
>+runs so it must be filtered:
>+
>+  sed -e "s#$TEST_DIR/#TEST_DIR/#g"
>+
>+Commonly needed filters are available in ./common.filter.
>+
>+** Python tests
>+
>+Most tests are implemented in bash but it is difficult to interact with the QMP
>+monitor.  A Python module called 'iotests' is available for tests that require
>+JSON and interacting with QEMU.
>+
>+* How to create a test
>+
>+1. Choose an unused test number
>+
>+Tests are identified by a unique number.  Look for the highest test case number
>+by looking at the test files.  Then search the qemu-devel@nongnu.org mailing
>+list to check if anyone has already sent patches using the next available
>+number.  You may need to increment the number a few times to reach an unused
>+number.
>+
>+2. Create the test file
>+
>+Copy an existing test (one that most closely resembles what you wish to test)
>+to the new test number:
>+
>+  cp 001 <test-number>
>+
>+3. Assign groups to the test
>+
>+Add your test to the ./group file.  This file is the index of tests and assigns
>+them to functional groups like "rw" for read-write tests.  Most tests belong to
>+the "rw" and "auto" groups.  "auto" means the test runs when ./check is invoked
>+without a -g argument.
>+
>+Consider adding your test to the "quick" group if it executes quickly (<1s).
>+This group is run by "make check-block" and is often included as part of build
>+tests in continuous integration systems.
>+
>+4. Write the test
>+
>+Edit the test script.  Look at existing tests for examples.
>+
>+5. Generate the golden master file
>+
>+Run your test with "./check <test-number>".  You may need to pass additional
>+options to use an image format or protocol.
>+
>+The test will fail because there is no golden master yet.  Inspect the output
>+that your test generated with "cat <test-number>.out.bad".
>+
>+Verify that the output is as expected and contains no volatile strings like
>+timestamps.  You may need to add filters to your test to remove volatile
>+strings.
>+
>+Once you are happy with the test output it can be used as the golden master
>+with "mv <test-number>.out.bad <test-number>.out".  Rerun the test to verify
>+that it passes.
>+
>+Congratulations, you've created a new test!
>+
> * Feedback and patches
>
> Please send improvements to the test suite, general feedback or just
> reports of failing tests cases to qemu-devel@nongnu.org with a CC:
> to qemu-block@nongnu.org.
>+
>-- 
>2.9.4
>
>
>

It is mentioned in the existing README that ./check runs just the raw 
format but it might be a good idea to mention that it's the default in 
the Howto as well. For this patch though:
Reviewed-by: Manos Pitsidianakis <el13635@mail.ntua.gr>

While we're on the topic of qemu-iotests, it'd be a nice thing to 
parallelize the tests with a Makefile.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Philippe Mathieu-Daudé]

On 07/21/2017 12:51 PM, Stefan Hajnoczi wrote:
> On Fri, Jul 21, 2017 at 07:16:34AM -0500, Eric Blake wrote:
>> On 07/21/2017 04:34 AM, Stefan Hajnoczi wrote:
>>> There is not much getting started documentation for qemu-iotests.  This
>>> patch explains how to create a new test and covers the overall testing
>>> approach.
>>>
>>> Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
>>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>>> ---
>>
>>> +3. Assign groups to the test
>>> +
>>> +Add your test to the ./group file.  This file is the index of tests and assigns
>>> +them to functional groups like "rw" for read-write tests.  Most tests belong to
>>> +the "rw" and "auto" groups.  "auto" means the test runs when ./check is invoked
>>> +without a -g argument.
>>> +
>>> +Consider adding your test to the "quick" group if it executes quickly (<1s).
>>
>> We have several tests going up to 5s (and I have a patch pending to
>> remove two tests that took longer) - I think 1s is a bit on the short
>> end for still classifying a test as quick.
> 
> I'm happy to accept any number blessed by Kevin.  I do think that 1
> second is a safe maximum and no one should get in trouble for adding a
> test that takes 1 second to the "quick" group.
> 
>>> +This group is run by "make check-block" and is often included as part of build
>>> +tests in continuous integration systems.
>>
>> It would still be nice to have 'make check' run 'make check-block'...
>> but that's independent of this patch.
> 
> Yes, there is a separate discussion about that on the list right now.
> Hopefully it will be added back.
> 
>>> +Once you are happy with the test output it can be used as the golden master
>>> +with "mv <test-number>.out.bad <test-number>.out".  Rerun the test to verify
>>> +that it passes.
>>> +
>>> +Congratulations, you've created a new test!
>>
>> Maybe a reminder to 'git add' the new files, then submit the patch?
> 
> I thought about this too but decided not to get into the git and patch
> submission business.  I carefully worded it to be about "creating" tests
> rather than "adding" them to qemu.git because I wanted to limit the
> scope of this README :).

Maybe something tiny like:

   Congratulations, you've created a new test!

   To share your test to upstream QEMU (highly recommended!) just follow
   these recommendations: http://wiki.qemu.org/Contribute/SubmitAPatch

> 
> Stefan
> 

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Kevin Wolf]

Am 21.07.2017 um 11:34 hat Stefan Hajnoczi geschrieben:
> There is not much getting started documentation for qemu-iotests.  This
> patch explains how to create a new test and covers the overall testing
> approach.
> 
> Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  tests/qemu-iotests/README | 83 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 83 insertions(+)
> 
> diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README
> index 6079b40..8259b9f 100644
> --- a/tests/qemu-iotests/README
> +++ b/tests/qemu-iotests/README
> @@ -14,8 +14,91 @@ Just run ./check to run all tests for the raw image format, or ./check
>  -qcow2 to test the qcow2 image format.  The output of ./check -h explains
>  additional options to test further image formats or I/O methods.
>  
> +* Testing approach
> +
> +Each test is an executable file (usually a bash script) that is run by the
> +./check test harness.  Standard out and standard error are captured to an
> +output file.  If the output file differs from the "golden master" output file
> +for the test then it fails.
> +
> +Tests are simply a sequence of commands that produce output and the test itself
> +does not judge whether it passed or failed.  If you find yourself writing
> +checks to determine success or failure then you should rethink the test and
> +rely on output diffing instead.
> +
> +** Filtering volatile output
> +
> +When output contains absolute file paths, timestamps, process IDs, hostnames,
> +or other volatile strings, the diff against golden master output will fail.
> +Such output must be filtered to replace volatile strings with fixed
> +placeholders.
> +
> +For example, the path to the temporary working directory changes between test
> +runs so it must be filtered:
> +
> +  sed -e "s#$TEST_DIR/#TEST_DIR/#g"
> +
> +Commonly needed filters are available in ./common.filter.
> +
> +** Python tests
> +
> +Most tests are implemented in bash but it is difficult to interact with the QMP
> +monitor.  A Python module called 'iotests' is available for tests that require
> +JSON and interacting with QEMU.
> +
> +* How to create a test
> +
> +1. Choose an unused test number
> +
> +Tests are identified by a unique number.  Look for the highest test case number
> +by looking at the test files.  Then search the qemu-devel@nongnu.org mailing
> +list to check if anyone has already sent patches using the next available
> +number.  You may need to increment the number a few times to reach an unused
> +number.
> +
> +2. Create the test file
> +
> +Copy an existing test (one that most closely resembles what you wish to test)
> +to the new test number:
> +
> +  cp 001 <test-number>
> +
> +3. Assign groups to the test
> +
> +Add your test to the ./group file.  This file is the index of tests and assigns
> +them to functional groups like "rw" for read-write tests.  Most tests belong to
> +the "rw" and "auto" groups.  "auto" means the test runs when ./check is invoked
> +without a -g argument.
> +
> +Consider adding your test to the "quick" group if it executes quickly (<1s).
> +This group is run by "make check-block" and is often included as part of build
> +tests in continuous integration systems.
> +
> +4. Write the test
> +
> +Edit the test script.  Look at existing tests for examples.
> +
> +5. Generate the golden master file
> +
> +Run your test with "./check <test-number>".  You may need to pass additional
> +options to use an image format or protocol.

./check refuses to even run a test if the reference output is missing.
So in practice you need a 'touch <test-number>.out' first.

> +The test will fail because there is no golden master yet.  Inspect the output
> +that your test generated with "cat <test-number>.out.bad".
> +
> +Verify that the output is as expected and contains no volatile strings like
> +timestamps.  You may need to add filters to your test to remove volatile
> +strings.
> +
> +Once you are happy with the test output it can be used as the golden master
> +with "mv <test-number>.out.bad <test-number>.out".  Rerun the test to verify
> +that it passes.
> +
> +Congratulations, you've created a new test!

Looks good otherwise.

Kevin

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2758 bytes --]

On Sun, Jul 23, 2017 at 11:18:50AM -0300, Philippe Mathieu-Daudé wrote:
> On 07/21/2017 12:51 PM, Stefan Hajnoczi wrote:
> > On Fri, Jul 21, 2017 at 07:16:34AM -0500, Eric Blake wrote:
> > > On 07/21/2017 04:34 AM, Stefan Hajnoczi wrote:
> > > > There is not much getting started documentation for qemu-iotests.  This
> > > > patch explains how to create a new test and covers the overall testing
> > > > approach.
> > > > 
> > > > Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
> > > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > > > ---
> > > 
> > > > +3. Assign groups to the test
> > > > +
> > > > +Add your test to the ./group file.  This file is the index of tests and assigns
> > > > +them to functional groups like "rw" for read-write tests.  Most tests belong to
> > > > +the "rw" and "auto" groups.  "auto" means the test runs when ./check is invoked
> > > > +without a -g argument.
> > > > +
> > > > +Consider adding your test to the "quick" group if it executes quickly (<1s).
> > > 
> > > We have several tests going up to 5s (and I have a patch pending to
> > > remove two tests that took longer) - I think 1s is a bit on the short
> > > end for still classifying a test as quick.
> > 
> > I'm happy to accept any number blessed by Kevin.  I do think that 1
> > second is a safe maximum and no one should get in trouble for adding a
> > test that takes 1 second to the "quick" group.
> > 
> > > > +This group is run by "make check-block" and is often included as part of build
> > > > +tests in continuous integration systems.
> > > 
> > > It would still be nice to have 'make check' run 'make check-block'...
> > > but that's independent of this patch.
> > 
> > Yes, there is a separate discussion about that on the list right now.
> > Hopefully it will be added back.
> > 
> > > > +Once you are happy with the test output it can be used as the golden master
> > > > +with "mv <test-number>.out.bad <test-number>.out".  Rerun the test to verify
> > > > +that it passes.
> > > > +
> > > > +Congratulations, you've created a new test!
> > > 
> > > Maybe a reminder to 'git add' the new files, then submit the patch?
> > 
> > I thought about this too but decided not to get into the git and patch
> > submission business.  I carefully worded it to be about "creating" tests
> > rather than "adding" them to qemu.git because I wanted to limit the
> > scope of this README :).
> 
> Maybe something tiny like:
> 
>   Congratulations, you've created a new test!
> 
>   To share your test to upstream QEMU (highly recommended!) just follow
>   these recommendations: http://wiki.qemu.org/Contribute/SubmitAPatch

Sounds good.

Kevin: Any thoughts on this patch?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Peter Maydell]

On 21 July 2017 at 10:34, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> There is not much getting started documentation for qemu-iotests.  This
> patch explains how to create a new test and covers the overall testing
> approach.
>
> Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  tests/qemu-iotests/README | 83 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 83 insertions(+)
>
> diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README
> index 6079b40..8259b9f 100644
> --- a/tests/qemu-iotests/README
> +++ b/tests/qemu-iotests/README
> @@ -14,8 +14,91 @@ Just run ./check to run all tests for the raw image format, or ./check
>  -qcow2 to test the qcow2 image format.  The output of ./check -h explains
>  additional options to test further image formats or I/O methods.
>
> +* Testing approach
> +
> +Each test is an executable file (usually a bash script) that is run by the
> +./check test harness.  Standard out and standard error are captured to an
> +output file.  If the output file differs from the "golden master" output file
> +for the test then it fails.

Should ./check be run from the source tree, or the build tree? The
existing README text doesn't say and I don't think your additions
do either.

thanks
-- PMM

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

On 07/21/2017 04:34 AM, Stefan Hajnoczi wrote:
> There is not much getting started documentation for qemu-iotests.  This
> patch explains how to create a new test and covers the overall testing
> approach.
> 
> +2. Create the test file
> +
> +Copy an existing test (one that most closely resembles what you wish to test)
> +to the new test number:
> +
> +  cp 001 <test-number>

And mark it executable (chmod a+x <test-number>)

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Kevin Wolf]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

Am 24.07.2017 um 16:28 hat Eric Blake geschrieben:
> On 07/21/2017 04:34 AM, Stefan Hajnoczi wrote:
> > There is not much getting started documentation for qemu-iotests.  This
> > patch explains how to create a new test and covers the overall testing
> > approach.
> > 
> > +2. Create the test file
> > +
> > +Copy an existing test (one that most closely resembles what you wish to test)
> > +to the new test number:
> > +
> > +  cp 001 <test-number>
> 
> And mark it executable (chmod a+x <test-number>)

If you copy an existing test, it's already executable.

Kevin

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Peter Maydell]

On 24 July 2017 at 15:28, Eric Blake <eblake@redhat.com> wrote:
> On 07/21/2017 04:34 AM, Stefan Hajnoczi wrote:
>> There is not much getting started documentation for qemu-iotests.  This
>> patch explains how to create a new test and covers the overall testing
>> approach.
>>
>> +2. Create the test file
>> +
>> +Copy an existing test (one that most closely resembles what you wish to test)
>> +to the new test number:
>> +
>> +  cp 001 <test-number>
>
> And mark it executable (chmod a+x <test-number>)

It should already be executable because the test file being
copied (001 in this case) is executable, shouldn't it?

thanks
-- PMM

Re: [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

On 07/24/2017 09:34 AM, Peter Maydell wrote:
> On 24 July 2017 at 15:28, Eric Blake <eblake@redhat.com> wrote:
>> On 07/21/2017 04:34 AM, Stefan Hajnoczi wrote:
>>> There is not much getting started documentation for qemu-iotests.  This
>>> patch explains how to create a new test and covers the overall testing
>>> approach.
>>>
>>> +2. Create the test file
>>> +
>>> +Copy an existing test (one that most closely resembles what you wish to test)
>>> +to the new test number:
>>> +
>>> +  cp 001 <test-number>
>>
>> And mark it executable (chmod a+x <test-number>)
> 
> It should already be executable because the test file being
> copied (001 in this case) is executable, shouldn't it?

Oh, I see what happened.  Rather than dropping to shell 'cp', I copied
via emacs' "ctrl-x ctrl-w" (write-file) and creating a new file name,
and that does not preserve executable bit.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [Qemu-block] [PATCH] qemu-iotests: add a "how to" to ./README

[John Snow]

On 07/21/2017 11:51 AM, Stefan Hajnoczi wrote:
> On Fri, Jul 21, 2017 at 07:16:34AM -0500, Eric Blake wrote:
>> On 07/21/2017 04:34 AM, Stefan Hajnoczi wrote:
>>> There is not much getting started documentation for qemu-iotests.  This
>>> patch explains how to create a new test and covers the overall testing
>>> approach.
>>>
>>> Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
>>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>>> ---
>>
>>> +3. Assign groups to the test
>>> +
>>> +Add your test to the ./group file.  This file is the index of tests and assigns
>>> +them to functional groups like "rw" for read-write tests.  Most tests belong to
>>> +the "rw" and "auto" groups.  "auto" means the test runs when ./check is invoked
>>> +without a -g argument.
>>> +
>>> +Consider adding your test to the "quick" group if it executes quickly (<1s).
>>
>> We have several tests going up to 5s (and I have a patch pending to
>> remove two tests that took longer) - I think 1s is a bit on the short
>> end for still classifying a test as quick.
> 
> I'm happy to accept any number blessed by Kevin.  I do think that 1
> second is a safe maximum and no one should get in trouble for adding a
> test that takes 1 second to the "quick" group.
> 
>>> +This group is run by "make check-block" and is often included as part of build
>>> +tests in continuous integration systems.
>>
>> It would still be nice to have 'make check' run 'make check-block'...
>> but that's independent of this patch.
> 
> Yes, there is a separate discussion about that on the list right now.
> Hopefully it will be added back.
> 

If we do, it'd be nice to have a `make check-block-alt` or something 
else that checks the exact set of tests not covered by `make check-block`.

Re: [Qemu-devel] [Qemu-block] [PATCH] qemu-iotests: add a "how to" to ./README

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1587 bytes --]

On Mon, Jul 24, 2017 at 11:20:44AM +0100, Peter Maydell wrote:
> On 21 July 2017 at 10:34, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > There is not much getting started documentation for qemu-iotests.  This
> > patch explains how to create a new test and covers the overall testing
> > approach.
> >
> > Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > ---
> >  tests/qemu-iotests/README | 83 +++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 83 insertions(+)
> >
> > diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README
> > index 6079b40..8259b9f 100644
> > --- a/tests/qemu-iotests/README
> > +++ b/tests/qemu-iotests/README
> > @@ -14,8 +14,91 @@ Just run ./check to run all tests for the raw image format, or ./check
> >  -qcow2 to test the qcow2 image format.  The output of ./check -h explains
> >  additional options to test further image formats or I/O methods.
> >
> > +* Testing approach
> > +
> > +Each test is an executable file (usually a bash script) that is run by the
> > +./check test harness.  Standard out and standard error are captured to an
> > +output file.  If the output file differs from the "golden master" output file
> > +for the test then it fails.
> 
> Should ./check be run from the source tree, or the build tree? The
> existing README text doesn't say and I don't think your additions
> do either.

It doesn't matter, both should work.  The script detects both
possibilities and rejigs itself to compensate.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [Qemu-block] [PATCH] qemu-iotests: add a "how to" to ./README

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

On Mon, Jul 24, 2017 at 11:11:28AM +0200, Kevin Wolf wrote:
> Am 21.07.2017 um 11:34 hat Stefan Hajnoczi geschrieben:
> > +5. Generate the golden master file
> > +
> > +Run your test with "./check <test-number>".  You may need to pass additional
> > +options to use an image format or protocol.
> 
> ./check refuses to even run a test if the reference output is missing.
> So in practice you need a 'touch <test-number>.out' first.

Oops, will fix!

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [Qemu-block] [PATCH] qemu-iotests: add a "how to" to ./README

[Peter Maydell]

On 25 July 2017 at 16:20, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Mon, Jul 24, 2017 at 11:20:44AM +0100, Peter Maydell wrote:
>> Should ./check be run from the source tree, or the build tree? The
>> existing README text doesn't say and I don't think your additions
>> do either.
>
> It doesn't matter, both should work.  The script detects both
> possibilities and rejigs itself to compensate.

Does that mean
 if you run it from the source tree in a tree configured
 for out of tree build it will:
 (a) pollute your source tree with test output and binaries
 (b) give you a helpful message about what to do
 (c) magically find the build tree somehow
 (d) not need to write any binaries/test output/temp files at all

?

thanks
-- PMM

Re: [Qemu-devel] [Qemu-block] [PATCH] qemu-iotests: add a "how to" to ./README

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1644 bytes --]

On Tue, Jul 25, 2017 at 04:34:19PM +0100, Peter Maydell wrote:
> On 25 July 2017 at 16:20, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > On Mon, Jul 24, 2017 at 11:20:44AM +0100, Peter Maydell wrote:
> >> Should ./check be run from the source tree, or the build tree? The
> >> existing README text doesn't say and I don't think your additions
> >> do either.
> >
> > It doesn't matter, both should work.  The script detects both
> > possibilities and rejigs itself to compensate.
> 
> Does that mean
>  if you run it from the source tree in a tree configured
>  for out of tree build it will:
>  (a) pollute your source tree with test output and binaries
>  (b) give you a helpful message about what to do
>  (c) magically find the build tree somehow
>  (d) not need to write any binaries/test output/temp files at all
> 
> ?

I think it would fail since the binaries would be missing in the source
tree.

./check handles either source or out-of-tree mode:

if [ -L "$0" ]
then
    # called from the build tree
    source_iotests=$(dirname "$(readlink "$0")")
    if [ -z "$source_iotests" ]
    then
        _init_error "failed to obtain source tree name from check symlink"
    fi
    source_iotests=$(cd "$source_iotests"; pwd) || _init_error "failed to enter source tree"
    build_iotests=$PWD
else
    # called from the source tree
    source_iotests=$PWD
    # this may be an in-tree build (note that in the following code we may not
    # assume that it truly is and have to test whether the build results
    # actually exist)
    build_iotests=$PWD
fi

build_root="$build_iotests/../.."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [Qemu-block] [PATCH] qemu-iotests: add a "how to" to ./README

[Peter Maydell]

On 26 July 2017 at 12:33, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> On Tue, Jul 25, 2017 at 04:34:19PM +0100, Peter Maydell wrote:
>> On 25 July 2017 at 16:20, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> > On Mon, Jul 24, 2017 at 11:20:44AM +0100, Peter Maydell wrote:
>> >> Should ./check be run from the source tree, or the build tree? The
>> >> existing README text doesn't say and I don't think your additions
>> >> do either.
>> >
>> > It doesn't matter, both should work.  The script detects both
>> > possibilities and rejigs itself to compensate.
>>
>> Does that mean
>>  if you run it from the source tree in a tree configured
>>  for out of tree build it will:
>>  (a) pollute your source tree with test output and binaries
>>  (b) give you a helpful message about what to do
>>  (c) magically find the build tree somehow
>>  (d) not need to write any binaries/test output/temp files at all
>>
>> ?
>
> I think it would fail since the binaries would be missing in the source
> tree.

That sounds like "the README should say you need to run it
from the build tree", then ?

thanks
-- PMM

Re: [Qemu-devel] [Qemu-block] [PATCH] qemu-iotests: add a "how to" to ./README

[Jeff Cody]

On Fri, Jul 21, 2017 at 10:34:16AM +0100, Stefan Hajnoczi wrote:
> There is not much getting started documentation for qemu-iotests.  This
> patch explains how to create a new test and covers the overall testing
> approach.
> 
> Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  tests/qemu-iotests/README | 83 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 83 insertions(+)
> 
> diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README
> index 6079b40..8259b9f 100644
> --- a/tests/qemu-iotests/README
> +++ b/tests/qemu-iotests/README
> @@ -14,8 +14,91 @@ Just run ./check to run all tests for the raw image format, or ./check
>  -qcow2 to test the qcow2 image format.  The output of ./check -h explains
>  additional options to test further image formats or I/O methods.
>  
> +* Testing approach
> +
> +Each test is an executable file (usually a bash script) that is run by the
> +./check test harness.  Standard out and standard error are captured to an
> +output file.  If the output file differs from the "golden master" output file
> +for the test then it fails.
> +
> +Tests are simply a sequence of commands that produce output and the test itself
> +does not judge whether it passed or failed.  If you find yourself writing
> +checks to determine success or failure then you should rethink the test and
> +rely on output diffing instead.
> +
> +** Filtering volatile output
> +
> +When output contains absolute file paths, timestamps, process IDs, hostnames,
> +or other volatile strings, the diff against golden master output will fail.
> +Such output must be filtered to replace volatile strings with fixed
> +placeholders.
> +
> +For example, the path to the temporary working directory changes between test
> +runs so it must be filtered:
> +
> +  sed -e "s#$TEST_DIR/#TEST_DIR/#g"
> +
> +Commonly needed filters are available in ./common.filter.
> +
> +** Python tests
> +
> +Most tests are implemented in bash but it is difficult to interact with the QMP
> +monitor.  A Python module called 'iotests' is available for tests that require
> +JSON and interacting with QEMU.
> +

There are some that prefer the bash tests still, and we do have a 'standard'
way to do so... so perhaps add as well:


** Bash tests

If you wish to create a test in Bash that interacts with the QMP monitor,
'common.qemu' provides functions for interacting with multiple QEMU
processes.


> +* How to create a test
> +
> +1. Choose an unused test number
> +
> +Tests are identified by a unique number.  Look for the highest test case number
> +by looking at the test files.  Then search the qemu-devel@nongnu.org mailing
> +list to check if anyone has already sent patches using the next available
> +number.  You may need to increment the number a few times to reach an unused
> +number.
> +
> +2. Create the test file
> +
> +Copy an existing test (one that most closely resembles what you wish to test)
> +to the new test number:
> +
> +  cp 001 <test-number>
> +
> +3. Assign groups to the test
> +
> +Add your test to the ./group file.  This file is the index of tests and assigns
> +them to functional groups like "rw" for read-write tests.  Most tests belong to
> +the "rw" and "auto" groups.  "auto" means the test runs when ./check is invoked
> +without a -g argument.
> +
> +Consider adding your test to the "quick" group if it executes quickly (<1s).
> +This group is run by "make check-block" and is often included as part of build
> +tests in continuous integration systems.
> +
> +4. Write the test
> +
> +Edit the test script.  Look at existing tests for examples.
> +
> +5. Generate the golden master file
> +
> +Run your test with "./check <test-number>".  You may need to pass additional
> +options to use an image format or protocol.
> +
> +The test will fail because there is no golden master yet.  Inspect the output
> +that your test generated with "cat <test-number>.out.bad".
> +
> +Verify that the output is as expected and contains no volatile strings like
> +timestamps.  You may need to add filters to your test to remove volatile
> +strings.
> +
> +Once you are happy with the test output it can be used as the golden master
> +with "mv <test-number>.out.bad <test-number>.out".  Rerun the test to verify
> +that it passes.
> +
> +Congratulations, you've created a new test!
> +
>  * Feedback and patches
>  
>  Please send improvements to the test suite, general feedback or just
>  reports of failing tests cases to qemu-devel@nongnu.org with a CC:
>  to qemu-block@nongnu.org.
> +
> -- 
> 2.9.4
> 
>