Re: [Qemu-devel] [PATCH RFC 0/1] qtest: Generic PCI device test

[Markus Armbruster <armbru@redhat.com>]

Andreas Färber <afaerber@suse.de> writes:

> Hi Markus,
>
> Again thanks for digging into this.
>
> Am 29.01.2015 um 15:58 schrieb Markus Armbruster:
>> This test does everything a number of existing tests currently do:
>> 
>>     ac97-test.c e1000-test.c es1370-test.c eepro100-test.c
>>     ne2000-test.c nvme-test.c pcnet-test.c rtl8139-test.c
>>     tpci200-test.c virtio-balloon-test.c virtio-rng-test.c
>>     vmxnet3-test.c
>> 
>> They are all marked "TODO: Replace with functional tests".  Options
>> 
>> * Delete them now, undelete when we add functional tests
>> 
>> * Keep them, blacklist the devices in pci-devs-test.c
>> 
>> * Live with the duplicated testing
>> 
>> Andreas, I guess you got an opinion here.
>
> My preference would be to not remove device files. "Functional tests"
> refers to doing real device-specific MMIO or PIO, and the intent of
> contributing such stubs, beyond the basic -device init/realize testing,
> was lowering the hurdle so that maintainers can require bugfixers to
> contribute a matching test case where applicable.

Yes, asking people who aren't familiar with device tests to create one
from scratch is asking a lot.  Asking them to copy a skeleton and flesh
it out is much more practical.  Almost as good as fleshing out one of
your stubs, I think.

A non-skeleton test for a really simple PCI device could serve as
skeleton.  wdt_i6300esb.c is a possible candidate.

>> There's overlap with a few others:
>> 
>>     i82801b11-test.c usb-hcd-ehci-test.c usb-hcd-ohci-test.c
>>     usb-hcd-xhci-test.c virtio-blk-test.c virtio-net-test.c
>>     virtio-scsi-test.c virtio-serial-test.c
>> 
>> Options:
>> 
>> * Blacklist the devices in pci-devs-test.c
>> 
>> * Live with the duplicated testing
>> 
>> Andreas?
>
> Not having reviewed the respective test code yet, I would suggest to
> keep generic tests in pci-devs-test.c and to rather drop generic tests
> from device-specific files (de-duplication).

Makes sense to me.  However, some existing device-specific files won't
have any tests left then.  What do you want me to do with them?

> While I haven't benchmarked it, I assume that the bigger contributor to
> qtest runtime is the overhead of spawning qemu-system-* processes rather
> than running multiple tests per process. So living with duplication may
> be a convenience option.

My new test is a pig in that regard: it runs qemu-system-FOO once per
non-blacklisted PCI device.

I could pack tests into fewer runs by using more than just a single PCI
slot.  i440FX can do 30 without a pci-bridge (would probably a bad idea
for this test) and multifunction (certainly a bad idea).  However, when
a test fails, the culprit isn't as obvious then.  Do you want me to try
that?