[kvm-unit-tests PATCH] x86: vmx: Break the 'vmx' monolith into multiple tests

[kvm-unit-tests PATCH] x86: vmx: Break the 'vmx' monolith into multiple tests

[Aaron Lewis]

Break 'vmx' into multiple tests to reduce the number of tests run at
once.  This will help get a better signal up front on failure with them
broken up into categories.  This will also do a better job of not
disguising new failures if one of the tests is already failing.

One side effect of breaking this test up is any new test added will
have to be manually added to a categorized test.  It will not
automatically be picked up by 'vmx'.

Signed-off-by: Aaron Lewis <aaronlewis@google.com>
---
 x86/unittests.cfg | 42 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 39 insertions(+), 3 deletions(-)

diff --git a/x86/unittests.cfg b/x86/unittests.cfg
index b48c98b..0d90413 100644
--- a/x86/unittests.cfg
+++ b/x86/unittests.cfg
@@ -254,13 +254,49 @@ extra_params = -cpu qemu64,+umip
 file = la57.flat
 arch = i386
 
-[vmx]
+[vmx_legacy]
 file = vmx.flat
-extra_params = -cpu host,+vmx -append "-exit_monitor_from_l2_test -ept_access* -vmx_smp* -vmx_vmcs_shadow_test -atomic_switch_overflow_msrs_test -vmx_init_signal_test -vmx_apic_passthrough_tpr_threshold_test"
+extra_params = -cpu max,+vmx -append "null vmenter preemption_timer control_field_PAT control_field_EFER CR_shadowing I/O_bitmap instruction_intercept EPT_A/D_disabled EPT_A/D_enabled PML interrupt nmi_hlt debug_controls MSR_switch vmmcall disable_RDTSCP int3 into invalid_msr"
 arch = x86_64
 groups = vmx
 
-[ept]
+[vmx_basic]
+file = vmx.flat
+extra_params = -cpu max,+vmx -append "v2_null_test v2_multiple_entries_test fixture_test_case1 fixture_test_case2"
+arch = x86_64
+groups = vmx
+
+[vmx_vm_entry]
+file = vmx.flat
+extra_params = -cpu max,+vmx -append "vmx_controls_test vmx_host_state_area_test vmx_guest_state_area_test vmentry_movss_shadow_test vmentry_unrestricted_guest_test"
+arch = x86_64
+groups = vmx
+
+[vmx_apic]
+file = vmx.flat
+extra_params = -cpu max,+vmx -append "vmx_eoi_bitmap_ioapic_scan_test vmx_hlt_with_rvi_test vmx_apic_passthrough_test vmx_apic_passthrough_thread_test vmx_sipi_signal_test"
+arch = x86_64
+groups = vmx
+
+[vmx_regression]
+file = vmx.flat
+extra_params = -cpu max,+vmx -append "vmx_ldtr_test vmx_cr_load_test vmx_cr4_osxsave_test vmx_nm_test vmx_db_test vmx_nmi_window_test vmx_intr_window_test vmx_pending_event_test vmx_pending_event_hlt_test vmx_store_tsc_test"
+arch = x86_64
+groups = vmx
+
+[vmx_preemption_timer]
+file = vmx.flat
+extra_params = -cpu max,+vmx -append "vmx_preemption_timer_zero_test vmx_preemption_timer_tf_test vmx_preemption_timer_expiry_test"
+arch = x86_64
+groups = vmx
+
+[vmx_misc]
+file = vmx.flat
+extra_params = -cpu max,+vmx -append "invvpid_test atomic_switch_max_msrs_test rdtsc_vmexit_diff_test vmx_mtf_test vmx_mtf_pdpte_test vmx_exception_test"
+arch = x86_64
+groups = vmx
+
+[vmx_ept]
 file = vmx.flat
 extra_params = -cpu host,host-phys-bits,+vmx -m 2560 -append "ept_access*"
 arch = x86_64
-- 
2.36.0.464.gb9c8b46e94-goog

Re: [kvm-unit-tests PATCH] x86: vmx: Break the 'vmx' monolith into multiple tests

[Sean Christopherson]

On Mon, May 02, 2022, Aaron Lewis wrote:
> Break 'vmx' into multiple tests to reduce the number of tests run at
> once.  This will help get a better signal up front on failure with them
> broken up into categories.  This will also do a better job of not
> disguising new failures if one of the tests is already failing.
> 
> One side effect of breaking this test up is any new test added will
> have to be manually added to a categorized test.  It will not
> automatically be picked up by 'vmx'.
> 
> Signed-off-by: Aaron Lewis <aaronlewis@google.com>
> ---
>  x86/unittests.cfg | 42 +++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 39 insertions(+), 3 deletions(-)
> 
> diff --git a/x86/unittests.cfg b/x86/unittests.cfg
> index b48c98b..0d90413 100644
> --- a/x86/unittests.cfg
> +++ b/x86/unittests.cfg
> @@ -254,13 +254,49 @@ extra_params = -cpu qemu64,+umip
>  file = la57.flat
>  arch = i386
>  
> -[vmx]
> +[vmx_legacy]
>  file = vmx.flat
> -extra_params = -cpu host,+vmx -append "-exit_monitor_from_l2_test -ept_access* -vmx_smp* -vmx_vmcs_shadow_test -atomic_switch_overflow_msrs_test -vmx_init_signal_test -vmx_apic_passthrough_tpr_threshold_test"
> +extra_params = -cpu max,+vmx -append "null vmenter preemption_timer control_field_PAT control_field_EFER CR_shadowing I/O_bitmap instruction_intercept EPT_A/D_disabled EPT_A/D_enabled PML interrupt nmi_hlt debug_controls MSR_switch vmmcall disable_RDTSCP int3 into invalid_msr"
>  arch = x86_64
>  groups = vmx
>  
> -[ept]
> +[vmx_basic]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "v2_null_test v2_multiple_entries_test fixture_test_case1 fixture_test_case2"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_vm_entry]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "vmx_controls_test vmx_host_state_area_test vmx_guest_state_area_test vmentry_movss_shadow_test vmentry_unrestricted_guest_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_apic]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "vmx_eoi_bitmap_ioapic_scan_test vmx_hlt_with_rvi_test vmx_apic_passthrough_test vmx_apic_passthrough_thread_test vmx_sipi_signal_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_regression]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "vmx_ldtr_test vmx_cr_load_test vmx_cr4_osxsave_test vmx_nm_test vmx_db_test vmx_nmi_window_test vmx_intr_window_test vmx_pending_event_test vmx_pending_event_hlt_test vmx_store_tsc_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_preemption_timer]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "vmx_preemption_timer_zero_test vmx_preemption_timer_tf_test vmx_preemption_timer_expiry_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_misc]
> +file = vmx.flat
> +extra_params = -cpu max,+vmx -append "invvpid_test atomic_switch_max_msrs_test rdtsc_vmexit_diff_test vmx_mtf_test vmx_mtf_pdpte_test vmx_exception_test"
> +arch = x86_64
> +groups = vmx
> +
> +[vmx_ept]
>  file = vmx.flat
>  extra_params = -cpu host,host-phys-bits,+vmx -m 2560 -append "ept_access*"
>  arch = x86_64

hmm, I don't like that there's no common criteria for the buckets.  E.g.
"regression" is a bucket for how a test came to exist, whereas "apic" is a bucket
for what a test exercises.  And the granularity ends up being odd too, e.g. the
preemption timer gets its own bucket, but "regression" has a massive variety of
tests.

My preference would be to separate tests by what they cover, which I think will be
most useful.  E.g. if I'm making changes to EPT/paging related stuff, it would be
nice to be able to run just the paging tests early on.  And when a bucket fails,
it'd also give the user a hint as to want might have gone wrong.  I'm sure there
are tests that straddle multiple buckets, so it won't be perfect, but I think we
can get something useful.

 vmx_cpu 		(don't like the name, want it to be for tests that verify CPU arch behavior)
 vmx_consistency_checks
 vmx_events 		(non-vAPIC APIC tests, timers, NMIs, IRQs, preemption timer, etc...)
 vmx_paging		(ept A/D, PML, invvpid, vmx_pf_exception_test, etc...)
 vmx_vapic		(vAPIC, VID, PI, etc...)
 ept_access 		(because they probably needs their own config, but they aren't the only EPT tests)