[PATCH] kvm-unit-test: more documentation and runner script

[PATCH] kvm-unit-test: more documentation and runner script

[Michael S. Tsirkin]

Add documentation about using qemu-system for unit tests.
Add runner script to select the correct binary and flags.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 README  | 10 +++++++++-
 x86-run | 27 +++++++++++++++++++++++++++
 2 files changed, 36 insertions(+), 1 deletion(-)
 create mode 100755 x86-run

diff --git a/README b/README
index 4ceb869..214397c 100644
--- a/README
+++ b/README
@@ -10,10 +10,18 @@ To create the tests' images just type 'make' in this directory.
 Tests' images created in ./<ARCH>/*.flat
 
 An example of a test invocation:
-qemu-system-x86_64 -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat
+Using qemu-kvm:
+
+qemu-kvm -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat
 This invocation runs the msr test case. The test outputs to stdio.
 
+Using qemu (supported since qemu 1.3):
+qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
 
+Or use a runner script to detect the correct invocation:
+./x86-run ./x86/msr.flat
+To select a specific qemu binary, specify the QEMU=<path> environment:
+QEMU=/tmp/qemu/x86_64-softmmu/qemu-system-x86_64 ./x86-run ./x86/msr.flat
 
 Directory structure:
 .:  Makefile and config files for the tests
diff --git a/x86-run b/x86-run
new file mode 100755
index 0000000..cf1d38a
--- /dev/null
+++ b/x86-run
@@ -0,0 +1,27 @@
+#!/usr/bin/bash
+
+qemukvm="${QEMU:-qemu-kvm}"
+qemusystem="${QEMU:-qemu-system-x86_64}"
+if
+	${qemukvm} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null;
+then
+	qemu="${qemukvm}"
+else
+	if
+		${qemsystem} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null;
+	then
+		qemu="${qemusystem}"
+	else
+		echo QEMU binary ${QEMU} has no support for test device. Exiting.
+		exit 1
+	fi
+fi
+
+if
+	${qemu} -device '?' 2>&1 | fgrep "pc-testdev" > /dev/null;
+then
+	command="${qemu} -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel"
+else
+	command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat"
+fi
+exec ${command} "$@"
-- 
MST

Re: [PATCH] kvm-unit-test: more documentation and runner script

[Lucas Meneghel Rodrigues]

On 02/27/2013 12:55 PM, Michael S. Tsirkin wrote:
> Add documentation about using qemu-system for unit tests.
> Add runner script to select the correct binary and flags.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>   README  | 10 +++++++++-
>   x86-run | 27 +++++++++++++++++++++++++++
>   2 files changed, 36 insertions(+), 1 deletion(-)
>   create mode 100755 x86-run
>
> diff --git a/README b/README
> index 4ceb869..214397c 100644
> --- a/README
> +++ b/README
> @@ -10,10 +10,18 @@ To create the tests' images just type 'make' in this directory.
>   Tests' images created in ./<ARCH>/*.flat
>
>   An example of a test invocation:
> -qemu-system-x86_64 -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat
> +Using qemu-kvm:
> +
> +qemu-kvm -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat
>   This invocation runs the msr test case. The test outputs to stdio.
>
> +Using qemu (supported since qemu 1.3):
> +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat

I think it is worth here to point out that with new qemu, after the 
unittest is done, the exit status of qemu is 1, different from the 'old 
style', whose exit status in successful completion is 0.

> +Or use a runner script to detect the correct invocation:
> +./x86-run ./x86/msr.flat
> +To select a specific qemu binary, specify the QEMU=<path> environment:
> +QEMU=/tmp/qemu/x86_64-softmmu/qemu-system-x86_64 ./x86-run ./x86/msr.flat
>
>   Directory structure:
>   .:  Makefile and config files for the tests
> diff --git a/x86-run b/x86-run
> new file mode 100755
> index 0000000..cf1d38a
> --- /dev/null
> +++ b/x86-run
> @@ -0,0 +1,27 @@
> +#!/usr/bin/bash
> +
> +qemukvm="${QEMU:-qemu-kvm}"
> +qemusystem="${QEMU:-qemu-system-x86_64}"
> +if
> +	${qemukvm} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null;
> +then
> +	qemu="${qemukvm}"
> +else
> +	if
> +		${qemsystem} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null;
> +	then
> +		qemu="${qemusystem}"
> +	else
> +		echo QEMU binary ${QEMU} has no support for test device. Exiting.
> +		exit 1
> +	fi
> +fi
> +
> +if
> +	${qemu} -device '?' 2>&1 | fgrep "pc-testdev" > /dev/null;
> +then
> +	command="${qemu} -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel"
> +else
> +	command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat"
> +fi
> +exec ${command} "$@"

^ What about checking the exit status of qemu here and print something 
like "test $@ PASS" or "test $@ FAIL"?

Re: [PATCH] kvm-unit-test: more documentation and runner script

[Michael S. Tsirkin]

On Wed, Feb 27, 2013 at 03:10:01PM -0300, Lucas Meneghel Rodrigues wrote:
> On 02/27/2013 12:55 PM, Michael S. Tsirkin wrote:
> >Add documentation about using qemu-system for unit tests.
> >Add runner script to select the correct binary and flags.
> >
> >Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >---
> >  README  | 10 +++++++++-
> >  x86-run | 27 +++++++++++++++++++++++++++
> >  2 files changed, 36 insertions(+), 1 deletion(-)
> >  create mode 100755 x86-run
> >
> >diff --git a/README b/README
> >index 4ceb869..214397c 100644
> >--- a/README
> >+++ b/README
> >@@ -10,10 +10,18 @@ To create the tests' images just type 'make' in this directory.
> >  Tests' images created in ./<ARCH>/*.flat
> >
> >  An example of a test invocation:
> >-qemu-system-x86_64 -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat
> >+Using qemu-kvm:
> >+
> >+qemu-kvm -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat
> >  This invocation runs the msr test case. The test outputs to stdio.
> >
> >+Using qemu (supported since qemu 1.3):
> >+qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
> 
> I think it is worth here to point out that with new qemu, after the
> unittest is done, the exit status of qemu is 1, different from the
> 'old style', whose exit status in successful completion is 0.
> 
> >+Or use a runner script to detect the correct invocation:
> >+./x86-run ./x86/msr.flat
> >+To select a specific qemu binary, specify the QEMU=<path> environment:
> >+QEMU=/tmp/qemu/x86_64-softmmu/qemu-system-x86_64 ./x86-run ./x86/msr.flat
> >
> >  Directory structure:
> >  .:  Makefile and config files for the tests
> >diff --git a/x86-run b/x86-run
> >new file mode 100755
> >index 0000000..cf1d38a
> >--- /dev/null
> >+++ b/x86-run
> >@@ -0,0 +1,27 @@
> >+#!/usr/bin/bash
> >+
> >+qemukvm="${QEMU:-qemu-kvm}"
> >+qemusystem="${QEMU:-qemu-system-x86_64}"
> >+if
> >+	${qemukvm} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null;
> >+then
> >+	qemu="${qemukvm}"
> >+else
> >+	if
> >+		${qemsystem} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null;
> >+	then
> >+		qemu="${qemusystem}"
> >+	else
> >+		echo QEMU binary ${QEMU} has no support for test device. Exiting.
> >+		exit 1
> >+	fi
> >+fi
> >+
> >+if
> >+	${qemu} -device '?' 2>&1 | fgrep "pc-testdev" > /dev/null;
> >+then
> >+	command="${qemu} -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel"
> >+else
> >+	command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat"
> >+fi
> >+exec ${command} "$@"
> 
> ^ What about checking the exit status of qemu here and print
> something like "test $@ PASS" or "test $@ FAIL"?

How do we know how to interpret it?
Overall I think it's best to rely on test output
than on return status.

-- 
MST

Re: [PATCH] kvm-unit-test: more documentation and runner script

[Lucas Meneghel Rodrigues]

On 02/27/2013 05:44 PM, Michael S. Tsirkin wrote:
>>> +Using qemu (supported since qemu 1.3):
>>> +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
>>
>> I think it is worth here to point out that with new qemu, after the
>> unittest is done, the exit status of qemu is 1, different from the
>> 'old style', whose exit status in successful completion is 0.

^ "comment above"

>>
>>> +exec ${command} "$@"
>>
>> ^ What about checking the exit status of qemu here and print
>> something like "test $@ PASS" or "test $@ FAIL"?
>
> How do we know how to interpret it?
> Overall I think it's best to rely on test output
> than on return status.

See comment above. Well, test output may be good for humans, but it is 
really not good for machines [1], that's why when the test suite was 
developed, the convention was to make qemu to exit with a given return 
code on success and others on failure. Anyway, it was just a suggestion, 
feel free to disregard it.

[1] having to parse the output and try to guess what is a pass or fail 
is a mess at best, and should be avoided unless we positively have no 
saner way of doing it.

Re: [PATCH] kvm-unit-test: more documentation and runner script

[Michael S. Tsirkin]

On Wed, Feb 27, 2013 at 06:03:41PM -0300, Lucas Meneghel Rodrigues wrote:
> On 02/27/2013 05:44 PM, Michael S. Tsirkin wrote:
> >>>+Using qemu (supported since qemu 1.3):
> >>>+qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
> >>
> >>I think it is worth here to point out that with new qemu, after the
> >>unittest is done, the exit status of qemu is 1, different from the
> >>'old style', whose exit status in successful completion is 0.
> 
> ^ "comment above"
> 
> >>
> >>>+exec ${command} "$@"
> >>
> >>^ What about checking the exit status of qemu here and print
> >>something like "test $@ PASS" or "test $@ FAIL"?
> >
> >How do we know how to interpret it?
> >Overall I think it's best to rely on test output
> >than on return status.
> 
> See comment above. Well, test output may be good for humans, but it
> is really not good for machines [1], that's why when the test suite
> was developed, the convention was to make qemu to exit with a given
> return code on success and others on failure.

Right but given a qemu binary, how do I find out what it is on success
and what it is on failure?

> Anyway, it was just a
> suggestion, feel free to disregard it.
> 
> [1] having to parse the output and try to guess what is a pass or
> fail is a mess at best, and should be avoided unless we positively
> have no saner way of doing it.

Re: [PATCH] kvm-unit-test: more documentation and runner script

[Gleb Natapov]

On Wed, Feb 27, 2013 at 11:40:37PM +0200, Michael S. Tsirkin wrote:
> On Wed, Feb 27, 2013 at 06:03:41PM -0300, Lucas Meneghel Rodrigues wrote:
> > On 02/27/2013 05:44 PM, Michael S. Tsirkin wrote:
> > >>>+Using qemu (supported since qemu 1.3):
> > >>>+qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
> > >>
> > >>I think it is worth here to point out that with new qemu, after the
> > >>unittest is done, the exit status of qemu is 1, different from the
> > >>'old style', whose exit status in successful completion is 0.
> > 
> > ^ "comment above"
> > 
> > >>
> > >>>+exec ${command} "$@"
> > >>
> > >>^ What about checking the exit status of qemu here and print
> > >>something like "test $@ PASS" or "test $@ FAIL"?
> > >
> > >How do we know how to interpret it?
> > >Overall I think it's best to rely on test output
> > >than on return status.
> > 
> > See comment above. Well, test output may be good for humans, but it
> > is really not good for machines [1], that's why when the test suite
> > was developed, the convention was to make qemu to exit with a given
> > return code on success and others on failure.
> 
> Right but given a qemu binary, how do I find out what it is on success
> and what it is on failure?
> 
Since you know what device you are using you know expected value for
successful/failure.

> > Anyway, it was just a
> > suggestion, feel free to disregard it.
> > 
> > [1] having to parse the output and try to guess what is a pass or
> > fail is a mess at best, and should be avoided unless we positively
> > have no saner way of doing it.

--
			Gleb.

Re: [PATCH] kvm-unit-test: more documentation and runner script

[Michael S. Tsirkin]

On Thu, Feb 28, 2013 at 11:06:20AM +0200, Gleb Natapov wrote:
> On Wed, Feb 27, 2013 at 11:40:37PM +0200, Michael S. Tsirkin wrote:
> > On Wed, Feb 27, 2013 at 06:03:41PM -0300, Lucas Meneghel Rodrigues wrote:
> > > On 02/27/2013 05:44 PM, Michael S. Tsirkin wrote:
> > > >>>+Using qemu (supported since qemu 1.3):
> > > >>>+qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
> > > >>
> > > >>I think it is worth here to point out that with new qemu, after the
> > > >>unittest is done, the exit status of qemu is 1, different from the
> > > >>'old style', whose exit status in successful completion is 0.
> > > 
> > > ^ "comment above"
> > > 
> > > >>
> > > >>>+exec ${command} "$@"
> > > >>
> > > >>^ What about checking the exit status of qemu here and print
> > > >>something like "test $@ PASS" or "test $@ FAIL"?
> > > >
> > > >How do we know how to interpret it?
> > > >Overall I think it's best to rely on test output
> > > >than on return status.
> > > 
> > > See comment above. Well, test output may be good for humans, but it
> > > is really not good for machines [1], that's why when the test suite
> > > was developed, the convention was to make qemu to exit with a given
> > > return code on success and others on failure.
> > 
> > Right but given a qemu binary, how do I find out what it is on success
> > and what it is on failure?
> > 
> Since you know what device you are using you know expected value for
> successful/failure.

So exit status is 1 for success 0 for failure?

> > > Anyway, it was just a
> > > suggestion, feel free to disregard it.
> > > 
> > > [1] having to parse the output and try to guess what is a pass or
> > > fail is a mess at best, and should be avoided unless we positively
> > > have no saner way of doing it.
> 
> --
> 			Gleb.

Re: [PATCH] kvm-unit-test: more documentation and runner script

[Gleb Natapov]

On Thu, Feb 28, 2013 at 11:45:33AM +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 28, 2013 at 11:06:20AM +0200, Gleb Natapov wrote:
> > On Wed, Feb 27, 2013 at 11:40:37PM +0200, Michael S. Tsirkin wrote:
> > > On Wed, Feb 27, 2013 at 06:03:41PM -0300, Lucas Meneghel Rodrigues wrote:
> > > > On 02/27/2013 05:44 PM, Michael S. Tsirkin wrote:
> > > > >>>+Using qemu (supported since qemu 1.3):
> > > > >>>+qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
> > > > >>
> > > > >>I think it is worth here to point out that with new qemu, after the
> > > > >>unittest is done, the exit status of qemu is 1, different from the
> > > > >>'old style', whose exit status in successful completion is 0.
> > > > 
> > > > ^ "comment above"
> > > > 
> > > > >>
> > > > >>>+exec ${command} "$@"
> > > > >>
> > > > >>^ What about checking the exit status of qemu here and print
> > > > >>something like "test $@ PASS" or "test $@ FAIL"?
> > > > >
> > > > >How do we know how to interpret it?
> > > > >Overall I think it's best to rely on test output
> > > > >than on return status.
> > > > 
> > > > See comment above. Well, test output may be good for humans, but it
> > > > is really not good for machines [1], that's why when the test suite
> > > > was developed, the convention was to make qemu to exit with a given
> > > > return code on success and others on failure.
> > > 
> > > Right but given a qemu binary, how do I find out what it is on success
> > > and what it is on failure?
> > > 
> > Since you know what device you are using you know expected value for
> > successful/failure.
> 
> So exit status is 1 for success 0 for failure?
> 
As Lucas said above upstream is different from qemu-kvm unfortunately.
On upstream you need to shift return value right for 1 bit to get to
the tests return value (why?) and then 0 will be success otherwise failure.

--
			Gleb.

Re: [PATCH] kvm-unit-test: more documentation and runner script

[Lucas Meneghel Rodrigues]

On 02/28/2013 06:51 AM, Gleb Natapov wrote:
> On Thu, Feb 28, 2013 at 11:45:33AM +0200, Michael S. Tsirkin wrote:
>> On Thu, Feb 28, 2013 at 11:06:20AM +0200, Gleb Natapov wrote:
>>> On Wed, Feb 27, 2013 at 11:40:37PM +0200, Michael S. Tsirkin wrote:
>>>> On Wed, Feb 27, 2013 at 06:03:41PM -0300, Lucas Meneghel Rodrigues wrote:
>>>>> On 02/27/2013 05:44 PM, Michael S. Tsirkin wrote:
>>>>>>>> +Using qemu (supported since qemu 1.3):
>>>>>>>> +qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
>>>>>>>
>>>>>>> I think it is worth here to point out that with new qemu, after the
>>>>>>> unittest is done, the exit status of qemu is 1, different from the
>>>>>>> 'old style', whose exit status in successful completion is 0.
>>>>>
>>>>> ^ "comment above"
>>>>>
>>>>>>>
>>>>>>>> +exec ${command} "$@"
>>>>>>>
>>>>>>> ^ What about checking the exit status of qemu here and print
>>>>>>> something like "test $@ PASS" or "test $@ FAIL"?
>>>>>>
>>>>>> How do we know how to interpret it?
>>>>>> Overall I think it's best to rely on test output
>>>>>> than on return status.
>>>>>
>>>>> See comment above. Well, test output may be good for humans, but it
>>>>> is really not good for machines [1], that's why when the test suite
>>>>> was developed, the convention was to make qemu to exit with a given
>>>>> return code on success and others on failure.
>>>>
>>>> Right but given a qemu binary, how do I find out what it is on success
>>>> and what it is on failure?
>>>>
>>> Since you know what device you are using you know expected value for
>>> successful/failure.
>>
>> So exit status is 1 for success 0 for failure?

For qemu-kvm, 0 success, non 0 for failure
For qemu upstream, shift that 1 bit.

> As Lucas said above upstream is different from qemu-kvm unfortunately.
> On upstream you need to shift return value right for 1 bit to get to
> the tests return value (why?) and then 0 will be success otherwise failure.

On the why, It was Anthony's call. See, as the upstream test device 
doesn't have any ports we can write to to make qemu exit, we used 
another device that can do that. I vaguely remember he argued for the 
exit code to be shifted by 1 on that other device, and said it wasn't a 
problem for test suites, since they can define what is a success and 
what is a failure.