[DISCUSSION] Running one qtest test: how?

[DISCUSSION] Running one qtest test: how?

[Doug Evans]

[-- Attachment #1: Type: text/plain, Size: 1452 bytes --]

Hi.

I can run a subset of qtest tests with "make check-qtest-TARGET", but
that's the limit of the granularity that I can find. Why would one want
more granularity? When adding a test one wants the edit/test cycle as short
as possible and needlessly running other tests is a pain.

It'd be really nice to be able to specify one test via make check. I
realize I can pass V=1 and get some help to dig further. And if I grep for
my test in Makefile.mtest I can reduce the number of tests down to just the
test binary I want (e.g., "make run-test-229" to run qtest-i386/qos-test,
the 229 will vary tree to tree). I don't mind the 229, it's
machine generated but it's easy to find and will be reasonably stable in
one build (though ideally one could pass the test name to "make" instead of
NNN). But I still want more granularity. What I really want is:

$ make run-test-229 TEST_OPTS="-p /foo/bar"

so that only test /foo/bar is run (see the output of qos-test --help).
qos-test binaries can contain dozens of tests, I just want one of them.

Am I missing something? What do others do when adding a test?

I'm happy to work on a patch to let one pass additional parameters to the
test binary as in the above example. Guidance for what will be acceptable
appreciated. Different test binaries will take different parameters: A
general mechanism to pass arbitrary additional parameters to the test
binary (.test.cmd.NNN in mtest-speak) would be quite useful.

[-- Attachment #2: Type: text/html, Size: 2248 bytes --]

Re: [DISCUSSION] Running one qtest test: how?

[Alexander Bulekov]

On 201125 1539, Doug Evans wrote:
> Hi.
> 
> I can run a subset of qtest tests with "make check-qtest-TARGET", but
> that's the limit of the granularity that I can find. Why would one want
> more granularity? When adding a test one wants the edit/test cycle as short
> as possible and needlessly running other tests is a pain.
> 
> It'd be really nice to be able to specify one test via make check. I
> realize I can pass V=1 and get some help to dig further. And if I grep for
> my test in Makefile.mtest I can reduce the number of tests down to just the
> test binary I want (e.g., "make run-test-229" to run qtest-i386/qos-test,
> the 229 will vary tree to tree). I don't mind the 229, it's
> machine generated but it's easy to find and will be reasonably stable in
> one build (though ideally one could pass the test name to "make" instead of
> NNN). But I still want more granularity. What I really want is:
> 
> $ make run-test-229 TEST_OPTS="-p /foo/bar"
> 
> so that only test /foo/bar is run (see the output of qos-test --help).
> qos-test binaries can contain dozens of tests, I just want one of them.
> 
> Am I missing something? What do others do when adding a test?

I usually build only what I need and run it manually:
make qemu-system-TARGET
make tests/qtest/qos-test
QEMU_PATH=./qemu-system-TARGET ./tests/qtest/qos-test -p /foo/bar
?
That cuts down on the edit/test cycle, but it probably would be nicer if
there were some way to select a specific test for "make check-qtest".
-Alex

> 
> I'm happy to work on a patch to let one pass additional parameters to the
> test binary as in the above example. Guidance for what will be acceptable
> appreciated. Different test binaries will take different parameters: A
> general mechanism to pass arbitrary additional parameters to the test
> binary (.test.cmd.NNN in mtest-speak) would be quite useful.

Re: [DISCUSSION] Running one qtest test: how?

[Peter Maydell]

On Wed, 25 Nov 2020 at 23:41, Doug Evans <dje@google.com> wrote:
> I can run a subset of qtest tests with "make check-qtest-TARGET", but that's the limit of the granularity that I can find. Why would one want more granularity? When adding a test one wants the edit/test cycle as short as possible and needlessly running other tests is a pain.
>
> It'd be really nice to be able to specify one test via make check. I realize I can pass V=1 and get some help to dig further.

> Am I missing something? What do others do when adding a test?

I just follow the approach outlined in:
https://wiki.qemu.org/Features/QTest
 -- use V=1 to fish out the relevant commandline, and then
if necessary narrow it down further with -p.
I agree it might be nice if there were better support in
the build system for doing this more automatically.

thanks
-- PMM