[PATCH kvm-unit-tests 1/5] .gitignore: Add *.flat and config.mak

[PATCH kvm-unit-tests 1/5] .gitignore: Add *.flat and config.mak

[Cole Robinson]

---
 .gitignore | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitignore b/.gitignore
index ed857b7..52655e1 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,8 +1,10 @@
+config.mak
 .gdbinit
 *.a
 *.d
 *.o
 *.flat
+*.elf
 .pc
 patches
 .stgit-*
-- 
1.8.1.4

[PATCH kvm-unit-tests 2/5] x86/README: Drop it

[Cole Robinson]

Was out of date, and not particularly useful to begin with.
---
 x86/README | 16 ----------------
 1 file changed, 16 deletions(-)
 delete mode 100644 x86/README

diff --git a/x86/README b/x86/README
deleted file mode 100644
index d644abd..0000000
--- a/x86/README
+++ /dev/null
@@ -1,16 +0,0 @@
-Tests in this directory and what they do:
-
-access: lots of page table related access (pte/pde) (read/write)
-apic: enable x2apic, self ipi, ioapic intr, ioapic simultaneous
-emulator: move to/from regs, cmps, push, pop, to/from cr8, smsw and lmsw
-hypercall: intel and amd hypercall insn
-msr: write to msr (only KERNEL_GS_BASE for now)
-port80: lots of out to port 80
-realmode: goes back to realmode, shld, push/pop, mov immediate, cmp immediate, add immediate,
-         io, eflags instructions (clc, cli, etc.), jcc short, jcc near, call, long jmp, xchg
-sieve: heavy memory access with no paging and with paging static and with paging vmalloc'ed
-smptest: run smp_id() on every cpu and compares return value to number
-tsc: write to tsc(0) and write to tsc(100000000000) and read it back
-vmexit: long loops for each: cpuid, vmcall, mov_from_cr8, mov_to_cr8, inl_pmtimer, ipi, ipi+halt
-kvmclock_test: test of wallclock, monotonic cycle and performance of kvmclock
-pcid: basic functionality test of PCID/INVPCID feature
\ No newline at end of file
-- 
1.8.1.4

[PATCH kvm-unit-tests 3/5] x86/run-kvm-unit-tests: Drop it

[Cole Robinson]

Never appears to have done anything useful
---
 x86/run-kvm-unit-tests | 6 ------
 1 file changed, 6 deletions(-)
 delete mode 100644 x86/run-kvm-unit-tests

diff --git a/x86/run-kvm-unit-tests b/x86/run-kvm-unit-tests
deleted file mode 100644
index fed925a..0000000
--- a/x86/run-kvm-unit-tests
+++ /dev/null
@@ -1,6 +0,0 @@
-#!/usr/bin/python
-
-import sys, os, os.path
-
-prog = sys.argv[0]
-dir = os.path.dirname(prog)
-- 
1.8.1.4

[PATCH kvm-unit-tests 4/5] Rewrite x86-run in python

[Cole Robinson]

Main motivation here is in the next patch: getting access to ConfigParser
for pulling extra options out of x86/unittests.cfg.

Drop the README bits about manually invoking qemu-kvm/qemu-system: the
runner spits out the generated CLI if you want to invoke it by hand,
and since some tests want specific options the docs weren't correct
to begin with.

This also fixes the script to have a consistent exit status regardless
of the version of qemu we invoke.
---
 README  |  21 ++++--------
 x86-run | 113 ++++++++++++++++++++++++++++++++++++++++++++++++----------------
 2 files changed, 93 insertions(+), 41 deletions(-)

diff --git a/README b/README
index db525e3..9a580b8 100644
--- a/README
+++ b/README
@@ -9,24 +9,17 @@ See file testdev.txt for more details.
 To create the tests' images just type 'make' in this directory.
 Tests' images created in ./<ARCH>/*.flat
 
-An example of a test invocation:
-Using qemu-kvm:
 
-qemu-kvm -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel ./x86/msr.flat
-This invocation runs the msr test case. The test outputs to stdio.
+To launch a test, use the runner script:
 
-Using qemu (supported since qemu 1.3):
-qemu-system-x86_64 -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./x86/msr.flat
+  ./x86-run ./x86/msr.flat
 
-Or use a runner script to detect the correct invocation:
-./x86-run ./x86/msr.flat
-To select a specific qemu binary, specify the QEMU=<path> environment:
-QEMU=/tmp/qemu/x86_64-softmmu/qemu-system-x86_64 ./x86-run ./x86/msr.flat
+To select a specific qemu binary: specify the QEMU=<path> environment:
+
+  ./x86-run --qemu /path/to/x86_64-softmmu/qemu-system-x86_64 ./x86/msr.flat
+
+./x86-run returns 1 if any passed test fails, 0 otherwise.
 
-The exit status of the binary (and the script) is inconsistent: with
-qemu-system, after the unittest is done, the exit status of qemu is 1,
-different from the 'old style' qemu-kvm, whose exit status in successful
-completion is 0.
 
 Directory structure:
 .:  Makefile and config files for the tests
diff --git a/x86-run b/x86-run
index e395a70..ab9eed9 100755
--- a/x86-run
+++ b/x86-run
@@ -1,27 +1,86 @@
-#!/usr/bin/bash
-
-qemukvm="${QEMU:-qemu-kvm}"
-qemusystem="${QEMU:-qemu-system-x86_64}"
-if
-	${qemukvm} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null;
-then
-	qemu="${qemukvm}"
-else
-	if
-		${qemsystem} -device '?' 2>&1 | fgrep -e \"testdev\" -e \"pc-testdev\" > /dev/null;
-	then
-		qemu="${qemusystem}"
-	else
-		echo QEMU binary ${QEMU} has no support for test device. Exiting.
-		exit 1
-	fi
-fi
-
-if
-	${qemu} -device '?' 2>&1 | fgrep "pc-testdev" > /dev/null;
-then
-	command="${qemu} -enable-kvm -device pc-testdev -serial stdio -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel"
-else
-	command="${qemu} -device testdev,chardev=testlog -chardev file,id=testlog,path=msr.out -serial stdio -kernel"
-fi
-exec ${command} "$@"
+#!/usr/bin/env python
+
+import glob
+import optparse
+import os
+import sys
+
+
+def fail(msg):
+    print >> sys.stderr, "ERROR: %s" % msg
+    sys.exit(1)
+
+def supports_testdev(qemu):
+    return os.system("%s -device ? 2>&1 | fgrep -q "
+                     "-e 'testdev' -e 'pc-testdev'" % qemu) == 0
+
+def check_qemu(binpath):
+    if binpath:
+        if not supports_testdev(binpath):
+            fail("%s does not support testdev" % binpath)
+    elif supports_testdev("qemu-kvm"):
+        binpath = "qemu-kvm"
+    elif supports_testdev("qemu-system-x86_64"):
+        binpath = "qemu-system-x86_64"
+    else:
+        fail("Could not find qemu binary that supports testdev")
+
+    cmd = "%s -enable-kvm -serial stdio " % binpath
+
+    if os.system("%s -device ? 2>&1 | fgrep -q 'pc-testdev'" % binpath) == 0:
+        devcmd = ("-device pc-testdev "
+                  "-device isa-debug-exit,iobase=0xf4,iosize=0x4")
+    else:
+        devcmd = ("-device testdev,chardev=testlog "
+                  "-chardev file,id=testlog,path=test.out")
+
+    return cmd + devcmd
+
+
+def parse_args():
+    parser = optparse.OptionParser(
+        usage="%prog x86/$TEST1.flat [x86/$TEST2.flat] ...",
+        description="Launch a test with the correct qemu args")
+
+    parser.add_option("--qemu-bin", dest="qemu",
+                      help="Path to qemu binary to use. If not specified, "
+                           "uses qemu-kvm or qemu-system-x86_64 from $PATH.")
+    parser.add_option("--all", action="store_true", help="Run all x86 tests")
+
+    options, args = parser.parse_args()
+    if options.all:
+        args = glob.glob("x86/*.flat")
+        if not args:
+            parser.error("Didn't find any .flat files in x86/")
+
+    elif not args:
+        parser.error("A .flat test file or --all must be specified")
+
+    return options.qemu, args
+
+
+def main():
+    qemu, testfiles = parse_args()
+    basecmd = check_qemu(qemu)
+    newstyle = "pc-testdev" in basecmd
+
+    ret = 0
+    for testfile in testfiles:
+        cmd = basecmd + " -kernel %s" % testfile
+        print cmd
+
+        cmdret = os.system(cmd) >> 8
+        print
+
+        if cmdret == 0 and not newstyle:
+            continue
+        if cmdret == 1 and newstyle:
+            continue
+        ret = 1
+
+
+    return ret
+
+
+if __name__ == '__main__':
+    sys.exit(main())
-- 
1.8.1.4

[PATCH kvm-unit-tests 5/5] x86-run: Pull extra arguments from unittests.cfg

[Cole Robinson]

Some tests want extra arguments as enumerated in unittests.cfg,
use them.

unittests.cfg also has a few sections about invoking certains tests
with different combinations of options, but x86-run doesn't do
anything with that.
---
 x86-run | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/x86-run b/x86-run
index ab9eed9..7eebee2 100755
--- a/x86-run
+++ b/x86-run
@@ -1,5 +1,6 @@
 #!/usr/bin/env python
 
+import ConfigParser
 import glob
 import optparse
 import os
@@ -36,6 +37,24 @@ def check_qemu(binpath):
 
     return cmd + devcmd
 
+cfgfile = None
+def args_for_test(testfile):
+    global cfgfile
+    if not cfgfile:
+        cfgfile = ConfigParser.ConfigParser()
+        cfgfile.read("x86/unittests.cfg")
+
+    testname = os.path.splitext(os.path.basename(testfile))[0]
+    if testname not in cfgfile.sections():
+        return ""
+
+    cmd = ""
+    if cfgfile.has_option(testname, "extra_params"):
+        cmd += cfgfile.get(testname, "extra_params")
+    if cfgfile.has_option(testname, "smp"):
+        cmd += " -smp %s" % int(cfgfile.get(testname, "smp"))
+    return cmd
+
 
 def parse_args():
     parser = optparse.OptionParser(
@@ -67,6 +86,7 @@ def main():
     ret = 0
     for testfile in testfiles:
         cmd = basecmd + " -kernel %s" % testfile
+        cmd += " " + args_for_test(testfile)
         print cmd
 
         cmdret = os.system(cmd) >> 8
-- 
1.8.1.4

Re: [PATCH kvm-unit-tests 5/5] x86-run: Pull extra arguments from unittests.cfg

[Gleb Natapov]

On Fri, Mar 15, 2013 at 08:09:09PM -0400, Cole Robinson wrote:
> Some tests want extra arguments as enumerated in unittests.cfg,
> use them.
> 
> unittests.cfg also has a few sections about invoking certains tests
> with different combinations of options, but x86-run doesn't do
> anything with that.
With this it will not be possible to use x86-run outside of autotest,
no?

> ---
>  x86-run | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/x86-run b/x86-run
> index ab9eed9..7eebee2 100755
> --- a/x86-run
> +++ b/x86-run
> @@ -1,5 +1,6 @@
>  #!/usr/bin/env python
>  
> +import ConfigParser
>  import glob
>  import optparse
>  import os
> @@ -36,6 +37,24 @@ def check_qemu(binpath):
>  
>      return cmd + devcmd
>  
> +cfgfile = None
> +def args_for_test(testfile):
> +    global cfgfile
> +    if not cfgfile:
> +        cfgfile = ConfigParser.ConfigParser()
> +        cfgfile.read("x86/unittests.cfg")
> +
> +    testname = os.path.splitext(os.path.basename(testfile))[0]
> +    if testname not in cfgfile.sections():
> +        return ""
> +
> +    cmd = ""
> +    if cfgfile.has_option(testname, "extra_params"):
> +        cmd += cfgfile.get(testname, "extra_params")
> +    if cfgfile.has_option(testname, "smp"):
> +        cmd += " -smp %s" % int(cfgfile.get(testname, "smp"))
> +    return cmd
> +
>  
>  def parse_args():
>      parser = optparse.OptionParser(
> @@ -67,6 +86,7 @@ def main():
>      ret = 0
>      for testfile in testfiles:
>          cmd = basecmd + " -kernel %s" % testfile
> +        cmd += " " + args_for_test(testfile)
>          print cmd
>  
>          cmdret = os.system(cmd) >> 8
> -- 
> 1.8.1.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
			Gleb.

Re: [PATCH kvm-unit-tests 5/5] x86-run: Pull extra arguments from unittests.cfg

[Cole Robinson]

On 03/17/2013 11:25 AM, Gleb Natapov wrote:
> On Fri, Mar 15, 2013 at 08:09:09PM -0400, Cole Robinson wrote:
>> Some tests want extra arguments as enumerated in unittests.cfg,
>> use them.
>>
>> unittests.cfg also has a few sections about invoking certains tests
>> with different combinations of options, but x86-run doesn't do
>> anything with that.
> With this it will not be possible to use x86-run outside of autotest,
> no?
> 

Not true, x86-run is still meant to be the standalone helper script for
running unittests. autotest doesn't care about x86-run, and ConfigParser is a
standard python module.

x86/unittests.cfg already exists in the kvm-unit-tests repo, I assumed it was
encoding required test options but maybe I'm wrong about that. It's still
useful to build off of if there's value in running some tests with different
combinations of parameters.

- Cole

>> ---
>>  x86-run | 20 ++++++++++++++++++++
>>  1 file changed, 20 insertions(+)
>>
>> diff --git a/x86-run b/x86-run
>> index ab9eed9..7eebee2 100755
>> --- a/x86-run
>> +++ b/x86-run
>> @@ -1,5 +1,6 @@
>>  #!/usr/bin/env python
>>  
>> +import ConfigParser
>>  import glob
>>  import optparse
>>  import os
>> @@ -36,6 +37,24 @@ def check_qemu(binpath):
>>  
>>      return cmd + devcmd
>>  
>> +cfgfile = None
>> +def args_for_test(testfile):
>> +    global cfgfile
>> +    if not cfgfile:
>> +        cfgfile = ConfigParser.ConfigParser()
>> +        cfgfile.read("x86/unittests.cfg")
>> +
>> +    testname = os.path.splitext(os.path.basename(testfile))[0]
>> +    if testname not in cfgfile.sections():
>> +        return ""
>> +
>> +    cmd = ""
>> +    if cfgfile.has_option(testname, "extra_params"):
>> +        cmd += cfgfile.get(testname, "extra_params")
>> +    if cfgfile.has_option(testname, "smp"):
>> +        cmd += " -smp %s" % int(cfgfile.get(testname, "smp"))
>> +    return cmd
>> +
>>  
>>  def parse_args():
>>      parser = optparse.OptionParser(
>> @@ -67,6 +86,7 @@ def main():
>>      ret = 0
>>      for testfile in testfiles:
>>          cmd = basecmd + " -kernel %s" % testfile
>> +        cmd += " " + args_for_test(testfile)
>>          print cmd
>>  
>>          cmdret = os.system(cmd) >> 8
>> -- 
>> 1.8.1.4
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> 			Gleb.
> 

Re: [PATCH kvm-unit-tests 2/5] x86/README: Drop it

[Marcelo Tosatti]

On Fri, Mar 15, 2013 at 08:09:06PM -0400, Cole Robinson wrote:
> Was out of date, and not particularly useful to begin with.
> ---
>  x86/README | 16 ----------------
>  1 file changed, 16 deletions(-)
>  delete mode 100644 x86/README
> 
> diff --git a/x86/README b/x86/README
> deleted file mode 100644
> index d644abd..0000000
> --- a/x86/README
> +++ /dev/null
> @@ -1,16 +0,0 @@
> -Tests in this directory and what they do:
> -
> -access: lots of page table related access (pte/pde) (read/write)
> -apic: enable x2apic, self ipi, ioapic intr, ioapic simultaneous
> -emulator: move to/from regs, cmps, push, pop, to/from cr8, smsw and lmsw
> -hypercall: intel and amd hypercall insn
> -msr: write to msr (only KERNEL_GS_BASE for now)
> -port80: lots of out to port 80
> -realmode: goes back to realmode, shld, push/pop, mov immediate, cmp immediate, add immediate,
> -         io, eflags instructions (clc, cli, etc.), jcc short, jcc near, call, long jmp, xchg
> -sieve: heavy memory access with no paging and with paging static and with paging vmalloc'ed
> -smptest: run smp_id() on every cpu and compares return value to number
> -tsc: write to tsc(0) and write to tsc(100000000000) and read it back
> -vmexit: long loops for each: cpuid, vmcall, mov_from_cr8, mov_to_cr8, inl_pmtimer, ipi, ipi+halt
> -kvmclock_test: test of wallclock, monotonic cycle and performance of kvmclock
> -pcid: basic functionality test of PCID/INVPCID feature
> \ No newline at end of file
> -- 
> 1.8.1.4

The summary can be useful (would rather update it than remove it).

Re: [PATCH kvm-unit-tests 5/5] x86-run: Pull extra arguments from unittests.cfg

[Marcelo Tosatti]

On Sun, Mar 17, 2013 at 07:58:56PM -0400, Cole Robinson wrote:
> On 03/17/2013 11:25 AM, Gleb Natapov wrote:
> > On Fri, Mar 15, 2013 at 08:09:09PM -0400, Cole Robinson wrote:
> >> Some tests want extra arguments as enumerated in unittests.cfg,
> >> use them.
> >>
> >> unittests.cfg also has a few sections about invoking certains tests
> >> with different combinations of options, but x86-run doesn't do
> >> anything with that.
> > With this it will not be possible to use x86-run outside of autotest,
> > no?
> > 
> 
> Not true, x86-run is still meant to be the standalone helper script for
> running unittests. autotest doesn't care about x86-run, and ConfigParser is a
> standard python module.
> 
> x86/unittests.cfg already exists in the kvm-unit-tests repo, I assumed it was
> encoding required test options but maybe I'm wrong about that. It's still
> useful to build off of if there's value in running some tests with different
> combinations of parameters.

I fail to see what is the point here.

unittests.cfg has been originally (and continues to be, AFAIK), intended
for autotest:
http://kerneltrap.org/mailarchive/linux-kvm/2010/6/24/6264146

Please don't remove manual execution from README.

Re: [PATCH kvm-unit-tests 5/5] x86-run: Pull extra arguments from unittests.cfg

[Cole Robinson]

On 03/20/2013 03:06 PM, Marcelo Tosatti wrote:
> On Sun, Mar 17, 2013 at 07:58:56PM -0400, Cole Robinson wrote:
>> On 03/17/2013 11:25 AM, Gleb Natapov wrote:
>>> On Fri, Mar 15, 2013 at 08:09:09PM -0400, Cole Robinson wrote:
>>>> Some tests want extra arguments as enumerated in unittests.cfg,
>>>> use them.
>>>>
>>>> unittests.cfg also has a few sections about invoking certains tests
>>>> with different combinations of options, but x86-run doesn't do
>>>> anything with that.
>>> With this it will not be possible to use x86-run outside of autotest,
>>> no?
>>>
>>
>> Not true, x86-run is still meant to be the standalone helper script for
>> running unittests. autotest doesn't care about x86-run, and ConfigParser is a
>> standard python module.
>>
>> x86/unittests.cfg already exists in the kvm-unit-tests repo, I assumed it was
>> encoding required test options but maybe I'm wrong about that. It's still
>> useful to build off of if there's value in running some tests with different
>> combinations of parameters.
> 
> I fail to see what is the point here.
> 
> unittests.cfg has been originally (and continues to be, AFAIK), intended
> for autotest:
> http://kerneltrap.org/mailarchive/linux-kvm/2010/6/24/6264146
> 
> Please don't remove manual execution from README.

I guess I'm misunderstanding things here.

I understood that unittests.cfg was only being used by autotest. What confused
me was that this config enumerates different command line options for some of
the tests. Why is autotest invoking these test cases with different options
than the README suggests? Which one is right?

If unittests.cfg is the canonical list of options, than the README needs to
document different command lines for each test. If the README is canonical,
then I don't understand why autotest is using their own options.

My intent was to unify the two, so that autotest wouldn't reproduce an error
that the kvm-unit-test docs might miss. And if there's ever a new unit test
added that requires non-default qemu options, or is useful to run with a few
permutations of cli options, unittests.cfg can be extended and x86-run will
just do the right thing.

Does that make sense?

Thanks,
Cole

Re: [PATCH kvm-unit-tests 2/5] x86/README: Drop it

[Cole Robinson]

On 03/20/2013 02:54 PM, Marcelo Tosatti wrote:
> On Fri, Mar 15, 2013 at 08:09:06PM -0400, Cole Robinson wrote:
>> Was out of date, and not particularly useful to begin with.
>> ---
>>  x86/README | 16 ----------------
>>  1 file changed, 16 deletions(-)
>>  delete mode 100644 x86/README
>>
>> diff --git a/x86/README b/x86/README
>> deleted file mode 100644
>> index d644abd..0000000
>> --- a/x86/README
>> +++ /dev/null
>> @@ -1,16 +0,0 @@
>> -Tests in this directory and what they do:
>> -
>> -access: lots of page table related access (pte/pde) (read/write)
>> -apic: enable x2apic, self ipi, ioapic intr, ioapic simultaneous
>> -emulator: move to/from regs, cmps, push, pop, to/from cr8, smsw and lmsw
>> -hypercall: intel and amd hypercall insn
>> -msr: write to msr (only KERNEL_GS_BASE for now)
>> -port80: lots of out to port 80
>> -realmode: goes back to realmode, shld, push/pop, mov immediate, cmp immediate, add immediate,
>> -         io, eflags instructions (clc, cli, etc.), jcc short, jcc near, call, long jmp, xchg
>> -sieve: heavy memory access with no paging and with paging static and with paging vmalloc'ed
>> -smptest: run smp_id() on every cpu and compares return value to number
>> -tsc: write to tsc(0) and write to tsc(100000000000) and read it back
>> -vmexit: long loops for each: cpuid, vmcall, mov_from_cr8, mov_to_cr8, inl_pmtimer, ipi, ipi+halt
>> -kvmclock_test: test of wallclock, monotonic cycle and performance of kvmclock
>> -pcid: basic functionality test of PCID/INVPCID feature
>> \ No newline at end of file
>> -- 
>> 1.8.1.4
> 
> The summary can be useful (would rather update it than remove it).
> 

Okay, I'm not really qualified to update it, so I'll drop this patch.

Thanks,
Cole

Re: [PATCH kvm-unit-tests 5/5] x86-run: Pull extra arguments from unittests.cfg

[Gleb Natapov]

On Mon, Mar 25, 2013 at 02:30:27PM -0400, Cole Robinson wrote:
> On 03/20/2013 03:06 PM, Marcelo Tosatti wrote:
> > On Sun, Mar 17, 2013 at 07:58:56PM -0400, Cole Robinson wrote:
> >> On 03/17/2013 11:25 AM, Gleb Natapov wrote:
> >>> On Fri, Mar 15, 2013 at 08:09:09PM -0400, Cole Robinson wrote:
> >>>> Some tests want extra arguments as enumerated in unittests.cfg,
> >>>> use them.
> >>>>
> >>>> unittests.cfg also has a few sections about invoking certains tests
> >>>> with different combinations of options, but x86-run doesn't do
> >>>> anything with that.
> >>> With this it will not be possible to use x86-run outside of autotest,
> >>> no?
> >>>
> >>
> >> Not true, x86-run is still meant to be the standalone helper script for
> >> running unittests. autotest doesn't care about x86-run, and ConfigParser is a
> >> standard python module.
> >>
> >> x86/unittests.cfg already exists in the kvm-unit-tests repo, I assumed it was
> >> encoding required test options but maybe I'm wrong about that. It's still
> >> useful to build off of if there's value in running some tests with different
> >> combinations of parameters.
> > 
> > I fail to see what is the point here.
> > 
> > unittests.cfg has been originally (and continues to be, AFAIK), intended
> > for autotest:
> > http://kerneltrap.org/mailarchive/linux-kvm/2010/6/24/6264146
> > 
> > Please don't remove manual execution from README.
> 
> I guess I'm misunderstanding things here.
> 
> I understood that unittests.cfg was only being used by autotest. What confused
> me was that this config enumerates different command line options for some of
> the tests. Why is autotest invoking these test cases with different options
> than the README suggests? Which one is right?
> 
Neither. README just have examples how to enable test device in various
versions of QEMU. unittests.cfg suppose to have reasonable configuration
for each test which is less likely to fail on most kernel. For instance
I may add a fix for a kernel bug and a test case for the bug. Since the
bug is not fixed on any released kernel autotest will start to fail,
so it may sense to disable the test in unittests.cfg meanwhile. Or I can
add a test that crashes kernel if it fails. It is probably not wise to
enable it for autotest, but running it manually will be still possible.

> If unittests.cfg is the canonical list of options, than the README needs to
> document different command lines for each test. If the README is canonical,
> then I don't understand why autotest is using their own options.
> 
I do not quite understand what do you mean here. README just have an
example on how to enable test device in QEMU. unittests.cfg does not
control that.

> My intent was to unify the two, so that autotest wouldn't reproduce an error
> that the kvm-unit-test docs might miss. And if there's ever a new unit test
> added that requires non-default qemu options, or is useful to run with a few
> permutations of cli options, unittests.cfg can be extended and x86-run will
> just do the right thing.
> 
> Does that make sense?
> 
> Thanks,
> Cole

--
			Gleb.

Re: [PATCH kvm-unit-tests 5/5] x86-run: Pull extra arguments from unittests.cfg

[Cole Robinson]

On 04/14/2013 09:40 AM, Gleb Natapov wrote:
> On Mon, Mar 25, 2013 at 02:30:27PM -0400, Cole Robinson wrote:
>> On 03/20/2013 03:06 PM, Marcelo Tosatti wrote:
>>> On Sun, Mar 17, 2013 at 07:58:56PM -0400, Cole Robinson wrote:
>>>> On 03/17/2013 11:25 AM, Gleb Natapov wrote:
>>>>> On Fri, Mar 15, 2013 at 08:09:09PM -0400, Cole Robinson wrote:
>>>>>> Some tests want extra arguments as enumerated in unittests.cfg,
>>>>>> use them.
>>>>>>
>>>>>> unittests.cfg also has a few sections about invoking certains tests
>>>>>> with different combinations of options, but x86-run doesn't do
>>>>>> anything with that.
>>>>> With this it will not be possible to use x86-run outside of autotest,
>>>>> no?
>>>>>
>>>>
>>>> Not true, x86-run is still meant to be the standalone helper script for
>>>> running unittests. autotest doesn't care about x86-run, and ConfigParser is a
>>>> standard python module.
>>>>
>>>> x86/unittests.cfg already exists in the kvm-unit-tests repo, I assumed it was
>>>> encoding required test options but maybe I'm wrong about that. It's still
>>>> useful to build off of if there's value in running some tests with different
>>>> combinations of parameters.
>>>
>>> I fail to see what is the point here.
>>>
>>> unittests.cfg has been originally (and continues to be, AFAIK), intended
>>> for autotest:
>>> http://kerneltrap.org/mailarchive/linux-kvm/2010/6/24/6264146
>>>
>>> Please don't remove manual execution from README.
>>
>> I guess I'm misunderstanding things here.
>>
>> I understood that unittests.cfg was only being used by autotest. What confused
>> me was that this config enumerates different command line options for some of
>> the tests. Why is autotest invoking these test cases with different options
>> than the README suggests? Which one is right?
>>
> Neither. README just have examples how to enable test device in various
> versions of QEMU. unittests.cfg suppose to have reasonable configuration
> for each test which is less likely to fail on most kernel. For instance
> I may add a fix for a kernel bug and a test case for the bug. Since the
> bug is not fixed on any released kernel autotest will start to fail,
> so it may sense to disable the test in unittests.cfg meanwhile. Or I can
> add a test that crashes kernel if it fails. It is probably not wise to
> enable it for autotest, but running it manually will be still possible.
> 
>> If unittests.cfg is the canonical list of options, than the README needs to
>> document different command lines for each test. If the README is canonical,
>> then I don't understand why autotest is using their own options.
>>
> I do not quite understand what do you mean here. README just have an
> example on how to enable test device in QEMU. unittests.cfg does not
> control that.
> 

Okay, sorry for the confusion. I revived the example invocations and reposted.

Thanks,
Cole

>> My intent was to unify the two, so that autotest wouldn't reproduce an error
>> that the kvm-unit-test docs might miss. And if there's ever a new unit test
>> added that requires non-default qemu options, or is useful to run with a few
>> permutations of cli options, unittests.cfg can be extended and x86-run will
>> just do the right thing.
>>
>> Does that make sense?
>>
>> Thanks,
>> Cole
> 
> --
> 			Gleb.
>