Re: [Autotest] [KVM-AUTOTEST PATCH v2 5/5] [RFC] KVM test: add WHQL test definitions to tests_base.cfg.sample

[Amos Kong <akong@redhat.com>]

On Thu, Jul 22, 2010 at 01:14:19PM +0300, Michael Goldish wrote:
> The parameters that define submissions (dd_name_*, dd_data_*, desc_path_*) were
> collected from manually created submissions.
> I haven't yet collected the parameters for Windows 2003 and 2008, so for now
> WHQL tests are disabled for these versions.
> 
> This patch also adds a comment in tests.cfg.sample, encouraging the user to
> specify the DTM server's address and ports there.
> 
> Note that this patch renames WinVista.32sp1 to WinVista.32.sp1,
>                              WinVista.32sp2 to WinVista.32.sp2,
>                              WinVista.64sp1 to WinVista.64.sp1 and
>                              WinVista.64sp2 to WinVista.64.sp2.
> 
> Changes from v1:
> - Run all whql_submission tests with -snapshot
> - Split whql_submission.hdd into 2 variants: a full one and one that only runs
>   a single test (by setting job_filter).  Running all tests in a HDD submission
>   takes over 72 hours, so it probably can't be done routinely as part of our
>   regression testing.
> - Change test_device regex to "qemu harddisk" in whql_submission.hdd.
> - Update timeouts
> - Enable WHQL tests for Win2003
> 
> Signed-off-by: Michael Goldish <mgoldish@redhat.com>
> ---
>  client/tests/kvm/tests.cfg.sample      |    8 +
>  client/tests/kvm/tests_base.cfg.sample |  251 +++++++++++++++++++++++++-------
>  2 files changed, 205 insertions(+), 54 deletions(-)
> 
> diff --git a/client/tests/kvm/tests.cfg.sample b/client/tests/kvm/tests.cfg.sample
> index e01406e..cdb7b0a 100644
> --- a/client/tests/kvm/tests.cfg.sample
> +++ b/client/tests/kvm/tests.cfg.sample
> @@ -73,6 +73,14 @@ variants:
>          only Fedora.13.64
>          only unattended_install.cdrom boot shutdown
>  
> +# You may provide information about the DTM server for WHQL tests here:
> +#whql:
> +#    server_address = 10.20.30.40
> +#    server_shell_port = 10022
> +#    server_file_transfer_port = 10023
> +# Note that the DTM server must run rss.exe (available under deps/),
> +# preferably with administrator privileges.
> +
>  # Uncomment the following lines to enable abort-on-error mode:
>  #abort_on_error = yes
>  #kill_vm.* ?= no
> diff --git a/client/tests/kvm/tests_base.cfg.sample b/client/tests/kvm/tests_base.cfg.sample
> index d0b8acb..f631184 100644
> --- a/client/tests/kvm/tests_base.cfg.sample
> +++ b/client/tests/kvm/tests_base.cfg.sample
> @@ -285,6 +285,90 @@ variants:
>          iozone_cmd = "D:\IOzone\iozone.exe -a"
>          iozone_timeout = 3600
>  
> +    - @whql:         install setup unattended_install.cdrom
> +        nic_mode = tap
> +        # Replace this with the address of an installed DTM server
> +        server_address = 10.20.30.40
> +        # The server should run rss.exe like a regular Windows VM, preferably
> +        # with administrator privileges (or at least with permission to write
> +        # to the DTM studio directory)
> +        server_shell_port = 10022
> +        server_file_transfer_port = 10023
> +        server_studio_path = %programfiles%\Microsoft Driver Test Manager\Studio
> +        wtt_services = wttsvc
> +        variants:
> +            - whql_client_install:
> +                type = whql_client_install
> +                dsso_delete_machine_binary = deps/whql_delete_machine_15.exe
> +                # The username and password are required for accessing the DTM client
> +                # installer binary shared by the server
> +                server_username = administrator
> +                server_password = 1q2w3eP
> +                # This path refers to a shared directory on the server
> +                # (the final cmd will be something like \\servername\DTMInstall\...)
> +                install_cmd = \DTMInstall\Client\Setup.exe /passive
> +                install_timeout = 3600
> +            - whql_submission:    whql_client_install
> +                type = whql_submission
> +                extra_params += " -snapshot"
> +                dsso_test_binary = deps/whql_submission_15.exe
> +                test_timeout = 3600
> +                device_data = cat0 cat1 cat2 cat3 logoarch logoos whqlos whqlqual prog desc filter virt
> +                descriptors = desc1 desc2 desc3
> +                # DeviceData names
> +                dd_name_cat0     = Category
> +                dd_name_cat1     = Category
> +                dd_name_cat2     = Category
> +                dd_name_cat3     = Category
> +                dd_name_logoarch = LogoProcessorArchitecture
> +                dd_name_logoos   = LogoOperatingSystem
> +                dd_name_whqlos   = WhqlOs
> +                dd_name_whqlqual = WhqlQualification
> +                dd_name_prog     = LogoProgramId
> +                dd_name_desc     = LogoProgramDescription
> +                dd_name_filter   = WDKFilterAttribute
> +                dd_name_virt     = ParaVirtualizationDriver
> +                # Common DeviceData data
> +                dd_data_filter   = FilterIfNoInf
> +                dd_data_virt     = True
> +                variants:
> +                    - keyboard:
> +                        # test_device is a regular expression that should match a device's
> +                        # name as it appears in device manager.  The first device that matches
> +                        # is used.
> +                        test_device = keyboard
> +                        # Set timeout to 10 hours
> +                        test_timeout = 36000
> +                        dd_data_cat0 = Input\Keyboard
> +                        dd_data_cat1 = Device Fundamentals
> +                        dd_data_cat2 = System Fundamentals\Dynamic Partitioning
> +                        dd_data_prog = InputKbd
> +                        dd_data_desc = Input > Keyboard


Hello Lucas and Michael,

We have tested whql related patches in our localhost, it seems the env preparation,
tasks allocation and result collection are all good.

As you know, whql tests always take too much time, so we need to classify all the tests,
and add some variants in configuration. Do you have some plan about add other tests ?
or reserve this work to user :) ?

  Which tests could not be executed ( manual tests / not related with kvm, virtualization, ...) ?
  Which tests are basic function test ?
  Which tests take more than $n hours ?
  ....

How about split the whql configuration to a singel file, whql_test.cfg.sample ?

Regards,
Amos

> +                    - hdd:
> +                        test_device = qemu harddisk
> +                        device_data += " ex0 ex1 ex2 ex3"
> +                        dd_data_cat0 = Storage\Device Class\Disk\Disk
> +                        dd_data_cat1 = Storage\Device Class\Disk\Fixed
> +                        dd_data_cat2 = Storage\Device Class\Disk\Bus\ATA
> +                        dd_data_cat3 = Device Fundamentals
> +                        dd_data_prog = StorHDD
> +                        dd_data_desc = Storage > Hard Disk Drive (HDD)
> +                        dd_name_ex0 = Storage_bus_type
> +                        dd_data_ex0 = ATA/ATAPI
> +                        dd_name_ex1 = Hybrid_HDD_Support
> +                        dd_data_ex1 = 0
> +                        dd_name_ex2 = Non_Rotating_Media
> +                        dd_data_ex2 = 0
> +                        dd_name_ex3 = Secure_Storage
> +                        dd_data_ex3 = 0
> +                        variants:
> +                            - full:
> +                                # Yes, 100 hours, this is not a mistake
> +                                test_timeout = 360000
> +                            - syscache_test:
> +                                job_filter = syscache test
> +                                test_timeout = 7200
> +