[kvm-unit-test PATCH 0/5] Improvements for the Travis CI

[kvm-unit-test PATCH 0/5] Improvements for the Travis CI

[Thomas Huth]

The first two patches make the test matrix a little bit more flexible,
and the fourth patch enables the 32-bit builds on x86.

But the most important patch is likely the third one: It is possible to
test with KVM on Travis now, so we can run the tests within a real KVM
environment, without TCG! The only caveat is that qemu-system-x86_64
has to run as root ... fixing only the permissions of /dev/kvm did
not help here, I still got a "Permission denied" in that case.

Thomas Huth (5):
  travis.yml: Re-arrange the test matrix
  travis.yml: Install only the required packages for each entry in the
    matrix
  travis.yml: Test with KVM instead of TCG (on x86)
  travis.yml: Test the i386 build, too
  travis.yml: Expect that at least one test succeeds

 .travis.yml | 155 +++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 104 insertions(+), 51 deletions(-)

-- 
2.23.0

[kvm-unit-test PATCH 1/5] travis.yml: Re-arrange the test matrix

[Thomas Huth]

We will soon need more control over the individual test matrix
entries, so we should not limit the matrix to "env" sections,
i.e. put the "matrix:" keyword on the top, not the "env:".

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .travis.yml | 92 ++++++++++++++++++++++++++++++++---------------------
 1 file changed, 56 insertions(+), 36 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 6c14953..611bbdc 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -16,44 +16,64 @@ addons:
       - qemu-system
 git:
   submodules: false
-env:
-  matrix:
-    - CONFIG=""
-      BUILD_DIR="."
-      TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
+
+matrix:
+  include:
+    - env:
+      - CONFIG=""
+      - BUILD_DIR="."
+      - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
              vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
-    - CONFIG=""
-      BUILD_DIR="x86-builddir"
-      TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
+
+    - env:
+      - CONFIG=""
+      - BUILD_DIR="x86-builddir"
+      - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
              tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
-    - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
-      BUILD_DIR="."
-      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
-    - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
-      BUILD_DIR="arm-buildir"
-      TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
-    - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
-      BUILD_DIR="."
-      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
-    - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
-      BUILD_DIR="arm64-buildir"
-      TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
-    - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
-      BUILD_DIR="."
-      TESTS="spapr_hcall emulator rtas-set-time-of-day"
-      ACCEL="tcg,cap-htm=off"
-    - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
-      BUILD_DIR="ppc64le-buildir"
-      TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
-      ACCEL="tcg,cap-htm=off"
-    - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
-      BUILD_DIR="."
-      TESTS="diag10 diag308"
-      ACCEL="tcg,firmware=s390x/run"
-    - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
-      BUILD_DIR="s390x-builddir"
-      TESTS="sieve"
-      ACCEL="tcg,firmware=s390x/run"
+
+    - env:
+      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
+      - BUILD_DIR="."
+      - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
+
+    - env:
+      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
+      - BUILD_DIR="arm-buildir"
+      - TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
+
+    - env:
+      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
+      - BUILD_DIR="."
+      - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
+
+    - env:
+      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
+      - BUILD_DIR="arm64-buildir"
+      - TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
+
+    - env:
+      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
+      - BUILD_DIR="."
+      - TESTS="spapr_hcall emulator rtas-set-time-of-day"
+      - ACCEL="tcg,cap-htm=off"
+
+    - env:
+      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
+      - BUILD_DIR="ppc64le-buildir"
+      - TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
+      - ACCEL="tcg,cap-htm=off"
+
+    - env:
+      - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
+      - BUILD_DIR="."
+      - TESTS="diag10 diag308"
+      - ACCEL="tcg,firmware=s390x/run"
+
+    - env:
+      - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
+      - BUILD_DIR="s390x-builddir"
+      - TESTS="sieve"
+      - ACCEL="tcg,firmware=s390x/run"
 
 before_script:
   - mkdir -p $BUILD_DIR && cd $BUILD_DIR
-- 
2.23.0

Re: [kvm-unit-test PATCH 1/5] travis.yml: Re-arrange the test matrix

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> We will soon need more control over the individual test matrix
> entries, so we should not limit the matrix to "env" sections,
> i.e. put the "matrix:" keyword on the top, not the "env:".
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .travis.yml | 92 ++++++++++++++++++++++++++++++++---------------------
>  1 file changed, 56 insertions(+), 36 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 6c14953..611bbdc 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -16,44 +16,64 @@ addons:
>        - qemu-system
>  git:
>    submodules: false
> -env:
> -  matrix:
> -    - CONFIG=""
> -      BUILD_DIR="."
> -      TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
> +
> +matrix:
> +  include:
> +    - env:
> +      - CONFIG=""
> +      - BUILD_DIR="."
> +      - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
>               vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
> -    - CONFIG=""
> -      BUILD_DIR="x86-builddir"
> -      TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
> +
> +    - env:
> +      - CONFIG=""
> +      - BUILD_DIR="x86-builddir"
> +      - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
>               tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
> -    - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
> -      BUILD_DIR="."
> -      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
> -    - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
> -      BUILD_DIR="arm-buildir"
> -      TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
> -    - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
> -      BUILD_DIR="."
> -      TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
> -    - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
> -      BUILD_DIR="arm64-buildir"
> -      TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
> -    - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
> -      BUILD_DIR="."
> -      TESTS="spapr_hcall emulator rtas-set-time-of-day"
> -      ACCEL="tcg,cap-htm=off"
> -    - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
> -      BUILD_DIR="ppc64le-buildir"
> -      TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
> -      ACCEL="tcg,cap-htm=off"
> -    - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
> -      BUILD_DIR="."
> -      TESTS="diag10 diag308"
> -      ACCEL="tcg,firmware=s390x/run"
> -    - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
> -      BUILD_DIR="s390x-builddir"
> -      TESTS="sieve"
> -      ACCEL="tcg,firmware=s390x/run"
> +
> +    - env:
> +      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
> +      - BUILD_DIR="."
> +      - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
> +
> +    - env:
> +      - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
> +      - BUILD_DIR="arm-buildir"
> +      - TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
> +
> +    - env:
> +      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
> +      - BUILD_DIR="."
> +      - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
> +
> +    - env:
> +      - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
> +      - BUILD_DIR="arm64-buildir"
> +      - TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
> +
> +    - env:
> +      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
> +      - BUILD_DIR="."
> +      - TESTS="spapr_hcall emulator rtas-set-time-of-day"
> +      - ACCEL="tcg,cap-htm=off"
> +
> +    - env:
> +      - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
> +      - BUILD_DIR="ppc64le-buildir"
> +      - TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
> +      - ACCEL="tcg,cap-htm=off"
> +
> +    - env:
> +      - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
> +      - BUILD_DIR="."
> +      - TESTS="diag10 diag308"
> +      - ACCEL="tcg,firmware=s390x/run"
> +
> +    - env:
> +      - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
> +      - BUILD_DIR="s390x-builddir"
> +      - TESTS="sieve"
> +      - ACCEL="tcg,firmware=s390x/run"
>
>  before_script:
>    - mkdir -p $BUILD_DIR && cd $BUILD_DIR


--
Alex Bennée

[kvm-unit-test PATCH 2/5] travis.yml: Install only the required packages for each entry in the matrix

[Thomas Huth]

We don't need all cross compiler and QEMU versions for each and every entry
in the test matrix, only the ones for the current target architecture.
So let's speed up the installation process a little bit by only installing
the packages that we really need.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .travis.yml | 53 +++++++++++++++++++++++++++++++----------------------
 1 file changed, 31 insertions(+), 22 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 611bbdc..3f5b5ee 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,75 +1,84 @@
 sudo: false
 dist: bionic
 language: c
-compiler:
-  - gcc
 cache: ccache
-addons:
-  apt:
-    packages:
-      # Cross Toolchains
-      - gcc-arm-linux-gnueabihf
-      - gcc-aarch64-linux-gnu
-      - gcc-powerpc64le-linux-gnu
-      - gcc-s390x-linux-gnu
-      # Run dependencies
-      - qemu-system
 git:
   submodules: false
 
 matrix:
   include:
-    - env:
+
+    - addons:
+        apt_packages: gcc qemu-system-x86
+      env:
       - CONFIG=""
       - BUILD_DIR="."
       - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
              vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
 
-    - env:
+    - addons:
+        apt_packages: gcc qemu-system-x86
+      env:
       - CONFIG=""
       - BUILD_DIR="x86-builddir"
       - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
              tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
 
-    - env:
+    - addons:
+        apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
+      env:
       - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
       - BUILD_DIR="."
       - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
 
-    - env:
+    - addons:
+        apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
+      env:
       - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
       - BUILD_DIR="arm-buildir"
       - TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
 
-    - env:
+    - addons:
+        apt_packages: gcc-aarch64-linux-gnu qemu-system-aarch64
+      env:
       - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
       - BUILD_DIR="."
       - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
 
-    - env:
+    - addons:
+        apt_packages: gcc-aarch64-linux-gnu qemu-system-aarch64
+      env:
       - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
       - BUILD_DIR="arm64-buildir"
       - TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
 
-    - env:
+    - addons:
+        apt_packages: gcc-powerpc64le-linux-gnu qemu-system-ppc
+      env:
       - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
       - BUILD_DIR="."
       - TESTS="spapr_hcall emulator rtas-set-time-of-day"
       - ACCEL="tcg,cap-htm=off"
 
-    - env:
+    - addons:
+        apt_packages: gcc-powerpc64le-linux-gnu qemu-system-ppc
+      env:
       - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
       - BUILD_DIR="ppc64le-buildir"
       - TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
       - ACCEL="tcg,cap-htm=off"
 
-    - env:
+    - addons:
+        apt_packages: gcc-s390x-linux-gnu qemu-system-s390x
+      env:
       - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
       - BUILD_DIR="."
       - TESTS="diag10 diag308"
       - ACCEL="tcg,firmware=s390x/run"
 
-    - env:
+    - addons:
+        apt_packages: gcc-s390x-linux-gnu qemu-system-s390x
+      env:
       - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
       - BUILD_DIR="s390x-builddir"
       - TESTS="sieve"
-- 
2.23.0

Re: [kvm-unit-test PATCH 2/5] travis.yml: Install only the required packages for each entry in the matrix

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> We don't need all cross compiler and QEMU versions for each and every entry
> in the test matrix, only the ones for the current target architecture.
> So let's speed up the installation process a little bit by only installing
> the packages that we really need.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .travis.yml | 53 +++++++++++++++++++++++++++++++----------------------
>  1 file changed, 31 insertions(+), 22 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 611bbdc..3f5b5ee 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,75 +1,84 @@
>  sudo: false
>  dist: bionic
>  language: c
> -compiler:
> -  - gcc
>  cache: ccache
> -addons:
> -  apt:
> -    packages:
> -      # Cross Toolchains
> -      - gcc-arm-linux-gnueabihf
> -      - gcc-aarch64-linux-gnu
> -      - gcc-powerpc64le-linux-gnu
> -      - gcc-s390x-linux-gnu
> -      # Run dependencies
> -      - qemu-system
>  git:
>    submodules: false
>
>  matrix:
>    include:
> -    - env:
> +
> +    - addons:
> +        apt_packages: gcc qemu-system-x86
> +      env:
>        - CONFIG=""
>        - BUILD_DIR="."
>        - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
>               vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
>
> -    - env:
> +    - addons:
> +        apt_packages: gcc qemu-system-x86
> +      env:
>        - CONFIG=""
>        - BUILD_DIR="x86-builddir"
>        - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
>               tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
>
> -    - env:
> +    - addons:
> +        apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
> +      env:
>        - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
>        - BUILD_DIR="."
>        - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
>
> -    - env:
> +    - addons:
> +        apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
> +      env:
>        - CONFIG="--arch=arm --cross-prefix=arm-linux-gnueabihf-"
>        - BUILD_DIR="arm-buildir"
>        - TESTS="pci-test pmu gicv2-active gicv3-active psci selftest-setup"
>
> -    - env:
> +    - addons:
> +        apt_packages: gcc-aarch64-linux-gnu qemu-system-aarch64
> +      env:
>        - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
>        - BUILD_DIR="."
>        - TESTS="selftest-vectors-kernel selftest-vectors-user selftest-smp"
>
> -    - env:
> +    - addons:
> +        apt_packages: gcc-aarch64-linux-gnu qemu-system-aarch64
> +      env:
>        - CONFIG="--arch=arm64 --cross-prefix=aarch64-linux-gnu-"
>        - BUILD_DIR="arm64-buildir"
>        - TESTS="pci-test pmu gicv2-active gicv3-active psci timer selftest-setup"
>
> -    - env:
> +    - addons:
> +        apt_packages: gcc-powerpc64le-linux-gnu qemu-system-ppc
> +      env:
>        - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
>        - BUILD_DIR="."
>        - TESTS="spapr_hcall emulator rtas-set-time-of-day"
>        - ACCEL="tcg,cap-htm=off"
>
> -    - env:
> +    - addons:
> +        apt_packages: gcc-powerpc64le-linux-gnu qemu-system-ppc
> +      env:
>        - CONFIG="--arch=ppc64 --endian=little --cross-prefix=powerpc64le-linux-gnu-"
>        - BUILD_DIR="ppc64le-buildir"
>        - TESTS="rtas-get-time-of-day rtas-get-time-of-day-base"
>        - ACCEL="tcg,cap-htm=off"
>
> -    - env:
> +    - addons:
> +        apt_packages: gcc-s390x-linux-gnu qemu-system-s390x
> +      env:
>        - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
>        - BUILD_DIR="."
>        - TESTS="diag10 diag308"
>        - ACCEL="tcg,firmware=s390x/run"
>
> -    - env:
> +    - addons:
> +        apt_packages: gcc-s390x-linux-gnu qemu-system-s390x
> +      env:
>        - CONFIG="--arch=s390x --cross-prefix=s390x-linux-gnu-"
>        - BUILD_DIR="s390x-builddir"
>        - TESTS="sieve"


--
Alex Bennée

[kvm-unit-test PATCH 3/5] travis.yml: Test with KVM instead of TCG (on x86)

[Thomas Huth]

Travis nowadays supports KVM in their CI pipelines, so we can finally
run the kvm-unit-tests with KVM instead of TCG here. Unfortunately, there
are some quirks: First, the QEMU binary has to be running as root, otherwise
you get an "permission denied" error here - even if you fix up the access
permissions to /dev/kvm first. Second, not all x86 tests are working in
this environment, so we still have to manually select the test set here
(but the amount of tests is definitely higher now than what we were able
to run with TCG before).

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .travis.yml | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 3f5b5ee..f91118c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,4 +1,4 @@
-sudo: false
+sudo: true
 dist: bionic
 language: c
 cache: ccache
@@ -13,16 +13,21 @@ matrix:
       env:
       - CONFIG=""
       - BUILD_DIR="."
-      - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
-             vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
+      - TESTS="access asyncpf debug emulator ept hypercall hyperv_stimer
+               hyperv_synic idt_test intel_iommu ioapic ioapic-split
+               kvmclock_test msr pcid rdpru realmode rmap_chain s3 umip"
+      - ACCEL="kvm"
 
     - addons:
         apt_packages: gcc qemu-system-x86
       env:
       - CONFIG=""
       - BUILD_DIR="x86-builddir"
-      - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
-             tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
+      - TESTS="smptest smptest3 tsc tsc_adjust xsave vmexit_cpuid vmexit_vmcall
+               sieve vmexit_inl_pmtimer vmexit_ipi_halt vmexit_mov_from_cr8
+               vmexit_mov_to_cr8 vmexit_ple_round_robin vmexit_tscdeadline
+               vmexit_tscdeadline_immed  vmx_apic_passthrough_thread syscall"
+      - ACCEL="kvm"
 
     - addons:
         apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
@@ -85,6 +90,9 @@ matrix:
       - ACCEL="tcg,firmware=s390x/run"
 
 before_script:
+  - if [ "$ACCEL" = "kvm" ]; then
+      sudo chmod u+s /usr/bin/qemu-system-* ;
+    fi
   - mkdir -p $BUILD_DIR && cd $BUILD_DIR
   - if [ -e ./configure ]; then ./configure $CONFIG ; fi
   - if [ -e ../configure ]; then ../configure $CONFIG ; fi
-- 
2.23.0

Re: [kvm-unit-test PATCH 3/5] travis.yml: Test with KVM instead of TCG (on x86)

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> Travis nowadays supports KVM in their CI pipelines, so we can finally
> run the kvm-unit-tests with KVM instead of TCG here. Unfortunately, there
> are some quirks: First, the QEMU binary has to be running as root, otherwise
> you get an "permission denied" error here - even if you fix up the access
> permissions to /dev/kvm first.

Could it be another resource it's trying to access?

> Second, not all x86 tests are working in
> this environment, so we still have to manually select the test set here
> (but the amount of tests is definitely higher now than what we were able
> to run with TCG before).
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .travis.yml | 18 +++++++++++++-----
>  1 file changed, 13 insertions(+), 5 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 3f5b5ee..f91118c 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,4 +1,4 @@
> -sudo: false
> +sudo: true
>  dist: bionic
>  language: c
>  cache: ccache
> @@ -13,16 +13,21 @@ matrix:
>        env:
>        - CONFIG=""
>        - BUILD_DIR="."
> -      - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
> -             vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
> +      - TESTS="access asyncpf debug emulator ept hypercall hyperv_stimer
> +               hyperv_synic idt_test intel_iommu ioapic ioapic-split
> +               kvmclock_test msr pcid rdpru realmode rmap_chain s3 umip"
> +      - ACCEL="kvm"
>
>      - addons:
>          apt_packages: gcc qemu-system-x86
>        env:
>        - CONFIG=""
>        - BUILD_DIR="x86-builddir"
> -      - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
> -             tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
> +      - TESTS="smptest smptest3 tsc tsc_adjust xsave vmexit_cpuid vmexit_vmcall
> +               sieve vmexit_inl_pmtimer vmexit_ipi_halt vmexit_mov_from_cr8
> +               vmexit_mov_to_cr8 vmexit_ple_round_robin vmexit_tscdeadline
> +               vmexit_tscdeadline_immed  vmx_apic_passthrough_thread syscall"
> +      - ACCEL="kvm"
>
>      - addons:
>          apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
> @@ -85,6 +90,9 @@ matrix:
>        - ACCEL="tcg,firmware=s390x/run"
>
>  before_script:
> +  - if [ "$ACCEL" = "kvm" ]; then
> +      sudo chmod u+s /usr/bin/qemu-system-* ;
> +    fi
>    - mkdir -p $BUILD_DIR && cd $BUILD_DIR
>    - if [ -e ./configure ]; then ./configure $CONFIG ; fi
>    - if [ -e ../configure ]; then ../configure $CONFIG ; fi


--
Alex Bennée

Re: [kvm-unit-test PATCH 3/5] travis.yml: Test with KVM instead of TCG (on x86)

[Thomas Huth]

On 13/11/2019 14.49, Alex Bennée wrote:
> 
> Thomas Huth <thuth@redhat.com> writes:
> 
>> Travis nowadays supports KVM in their CI pipelines, so we can finally
>> run the kvm-unit-tests with KVM instead of TCG here. Unfortunately, there
>> are some quirks: First, the QEMU binary has to be running as root, otherwise
>> you get an "permission denied" error here - even if you fix up the access
>> permissions to /dev/kvm first.
> 
> Could it be another resource it's trying to access?

I did some more tests with strace and some "ls -l /dev/kvm" spread
around the scripts, and it seems like the permissions get reset on
Debian when a process tries to open /dev/kvm ? I.e. after doing a
"chmod", I correctly see:

crw-rw-rw- 1 root kvm 10, 232 Nov 13 15:03 /dev/kvm

... but after the run script tried to launch qemu, it's back to:

crw-rw---- 1 root kvm 10, 232 Nov 13 15:03 /dev/kvm

I assume some udev magic is enforcing this?

So it's as Paolo said in his mail, you need to be root or in the kvm
group to be able to use KVM on Debian / Ubuntu. Unfortunately a simple
"usermod -a -G kvm $USER" also does not seem to work here, since you
need a new login shell to take the change into account ... well, I could
then run the run_test.sh script through "sudo -E su $USER -c ..." [1]
but that's ugly too.

But instead of doing "chmod u+s ...", this seems to be working, too:

  sudo chgrp kvm /usr/bin/qemu-system-*
  sudo chmod g+s /usr/bin/qemu-system-*

... which sounds like the least ugliest hack to me currently, so I think
I'll go with that version.

 Thomas


[1] That's also what Travis suggests here:
    https://docs.travis-ci.com/user/reference/trusty/#group-membership

[kvm-unit-tests PATCH v2] travis.yml: Test with KVM instead of TCG (on x86)

[Thomas Huth]

Travis nowadays supports KVM in their CI pipelines, so we can finally
run the kvm-unit-tests with KVM instead of TCG here. Unfortunately, there
are some quirks:

First, /dev/kvm is not accessible on Ubuntu by default. You have to be
"root" or in the "kvm" group to access it. But changing the group of the
current user is not taking into account for the current shell process, so
that would need some indirections in the yml file. Thus the yml script now
rather changes the group and "g+s" permission of the qemu binary instead.

Second, not all x86 tests are working in this environment, so we still
have to manually select the test set here (but the amount of tests is
definitely higher now than what we were able to run with TCG before).

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 v2: Use chgrp + "chmod g+s" instead "chmod u+s" to get it running

 .travis.yml | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 3f5b5ee..89c50fe 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,4 +1,4 @@
-sudo: false
+sudo: true
 dist: bionic
 language: c
 cache: ccache
@@ -13,16 +13,21 @@ matrix:
       env:
       - CONFIG=""
       - BUILD_DIR="."
-      - TESTS="vmexit_cpuid vmexit_mov_from_cr8 vmexit_mov_to_cr8 vmexit_ipi
-             vmexit_ple_round_robin vmexit_tscdeadline vmexit_tscdeadline_immed"
+      - TESTS="access asyncpf debug emulator ept hypercall hyperv_stimer
+               hyperv_synic idt_test intel_iommu ioapic ioapic-split
+               kvmclock_test msr pcid rdpru realmode rmap_chain s3 umip"
+      - ACCEL="kvm"
 
     - addons:
         apt_packages: gcc qemu-system-x86
       env:
       - CONFIG=""
       - BUILD_DIR="x86-builddir"
-      - TESTS="ioapic-split ioapic smptest smptest3 eventinj msr port80 syscall
-             tsc rmap_chain umip intel_iommu vmexit_inl_pmtimer vmexit_ipi_halt"
+      - TESTS="smptest smptest3 tsc tsc_adjust xsave vmexit_cpuid vmexit_vmcall
+               sieve vmexit_inl_pmtimer vmexit_ipi_halt vmexit_mov_from_cr8
+               vmexit_mov_to_cr8 vmexit_ple_round_robin vmexit_tscdeadline
+               vmexit_tscdeadline_immed  vmx_apic_passthrough_thread syscall"
+      - ACCEL="kvm"
 
     - addons:
         apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
@@ -85,6 +90,10 @@ matrix:
       - ACCEL="tcg,firmware=s390x/run"
 
 before_script:
+  - if [ "$ACCEL" = "kvm" ]; then
+      sudo chgrp kvm /usr/bin/qemu-system-* ;
+      sudo chmod g+s /usr/bin/qemu-system-* ;
+    fi
   - mkdir -p $BUILD_DIR && cd $BUILD_DIR
   - if [ -e ./configure ]; then ./configure $CONFIG ; fi
   - if [ -e ../configure ]; then ../configure $CONFIG ; fi
-- 
2.23.0

[kvm-unit-test PATCH 4/5] travis.yml: Test the i386 build, too

[Thomas Huth]

After installing gcc-multilib, we can also test the 32-bit builds
on Travis.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .travis.yml | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index f91118c..9ceb04d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -29,6 +29,21 @@ matrix:
                vmexit_tscdeadline_immed  vmx_apic_passthrough_thread syscall"
       - ACCEL="kvm"
 
+    - addons:
+        apt_packages: gcc gcc-multilib qemu-system-x86
+      env:
+      - CONFIG="--arch=i386"
+      - BUILD_DIR="."
+      - TESTS="eventinj port80 sieve tsc taskswitch umip vmexit_ple_round_robin"
+
+    - addons:
+        apt_packages: gcc gcc-multilib qemu-system-x86
+      env:
+      - CONFIG="--arch=i386"
+      - BUILD_DIR="i386-builddir"
+      - TESTS="vmexit_mov_from_cr8 vmexit_ipi vmexit_ipi_halt vmexit_mov_to_cr8
+               vmexit_cpuid vmexit_tscdeadline vmexit_tscdeadline_immed"
+
     - addons:
         apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
       env:
-- 
2.23.0

Re: [kvm-unit-test PATCH 4/5] travis.yml: Test the i386 build, too

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> After installing gcc-multilib, we can also test the 32-bit builds
> on Travis.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  .travis.yml | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
>
> diff --git a/.travis.yml b/.travis.yml
> index f91118c..9ceb04d 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -29,6 +29,21 @@ matrix:
>                 vmexit_tscdeadline_immed  vmx_apic_passthrough_thread syscall"
>        - ACCEL="kvm"
>
> +    - addons:
> +        apt_packages: gcc gcc-multilib qemu-system-x86
> +      env:
> +      - CONFIG="--arch=i386"
> +      - BUILD_DIR="."
> +      - TESTS="eventinj port80 sieve tsc taskswitch umip vmexit_ple_round_robin"
> +
> +    - addons:
> +        apt_packages: gcc gcc-multilib qemu-system-x86
> +      env:
> +      - CONFIG="--arch=i386"
> +      - BUILD_DIR="i386-builddir"
> +      - TESTS="vmexit_mov_from_cr8 vmexit_ipi vmexit_ipi_halt vmexit_mov_to_cr8
> +               vmexit_cpuid vmexit_tscdeadline vmexit_tscdeadline_immed"
> +
>      - addons:
>          apt_packages: gcc-arm-linux-gnueabihf qemu-system-arm
>        env:


--
Alex Bennée

[kvm-unit-test PATCH 5/5] travis.yml: Expect that at least one test succeeds

[Thomas Huth]

While working on the travis.yml file, I've run into cases where
all tests are reported as "SKIP" by the run_test.sh script (e.g.
when QEMU could not be started). This should not result in a
successful test run, so mark it as failed if not at least one
test passed.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.travis.yml b/.travis.yml
index 9ceb04d..aacf7d2 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -115,3 +115,4 @@ script:
   - make -j3
   - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
   - if grep -q FAIL results.txt ; then exit 1 ; fi
+  - if ! grep -q PASS results.txt ; then exit 1 ; fi
-- 
2.23.0

Re: [kvm-unit-test PATCH 5/5] travis.yml: Expect that at least one test succeeds

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> While working on the travis.yml file, I've run into cases where
> all tests are reported as "SKIP" by the run_test.sh script (e.g.
> when QEMU could not be started). This should not result in a
> successful test run, so mark it as failed if not at least one
> test passed.

But doesn't this mean you could have everything fail except one pass and
still report success?

>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .travis.yml | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/.travis.yml b/.travis.yml
> index 9ceb04d..aacf7d2 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -115,3 +115,4 @@ script:
>    - make -j3
>    - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
>    - if grep -q FAIL results.txt ; then exit 1 ; fi
> +  - if ! grep -q PASS results.txt ; then exit 1 ; fi


--
Alex Bennée

Re: [kvm-unit-test PATCH 5/5] travis.yml: Expect that at least one test succeeds

[Thomas Huth]

On 13/11/2019 16.06, Alex Bennée wrote:
> 
> Thomas Huth <thuth@redhat.com> writes:
> 
>> While working on the travis.yml file, I've run into cases where
>> all tests are reported as "SKIP" by the run_test.sh script (e.g.
>> when QEMU could not be started). This should not result in a
>> successful test run, so mark it as failed if not at least one
>> test passed.
> 
> But doesn't this mean you could have everything fail except one pass and
> still report success?

The FAILs are already handled one line earlier...

>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  .travis.yml | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 9ceb04d..aacf7d2 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -115,3 +115,4 @@ script:
>>    - make -j3
>>    - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
>>    - if grep -q FAIL results.txt ; then exit 1 ; fi

... here -----------^

>> +  - if ! grep -q PASS results.txt ; then exit 1 ; fi

Maybe it would also be nicer to provide proper exit values in the
run_tests.sh script, but the logic there gives me a bad headache...
grep'ing for FAIL and PASS in the yml script is way more easy.

 Thomas

Re: [kvm-unit-test PATCH 5/5] travis.yml: Expect that at least one test succeeds

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 13/11/2019 16.06, Alex Bennée wrote:
>>
>> Thomas Huth <thuth@redhat.com> writes:
>>
>>> While working on the travis.yml file, I've run into cases where
>>> all tests are reported as "SKIP" by the run_test.sh script (e.g.
>>> when QEMU could not be started). This should not result in a
>>> successful test run, so mark it as failed if not at least one
>>> test passed.
>>
>> But doesn't this mean you could have everything fail except one pass and
>> still report success?
>
> The FAILs are already handled one line earlier...

Oops I should have noticed that:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

>
>>>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>  .travis.yml | 1 +
>>>  1 file changed, 1 insertion(+)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index 9ceb04d..aacf7d2 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -115,3 +115,4 @@ script:
>>>    - make -j3
>>>    - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
>>>    - if grep -q FAIL results.txt ; then exit 1 ; fi
>
> ... here -----------^
>
>>> +  - if ! grep -q PASS results.txt ; then exit 1 ; fi
>
> Maybe it would also be nicer to provide proper exit values in the
> run_tests.sh script, but the logic there gives me a bad headache...
> grep'ing for FAIL and PASS in the yml script is way more easy.
>
>  Thomas


--
Alex Bennée

Re: [kvm-unit-test PATCH 0/5] Improvements for the Travis CI

[Paolo Bonzini]

On 13/11/19 12:26, Thomas Huth wrote:
> The first two patches make the test matrix a little bit more flexible,
> and the fourth patch enables the 32-bit builds on x86.
> 
> But the most important patch is likely the third one: It is possible to
> test with KVM on Travis now, so we can run the tests within a real KVM
> environment, without TCG! The only caveat is that qemu-system-x86_64
> has to run as root ... fixing only the permissions of /dev/kvm did
> not help here, I still got a "Permission denied" in that case.

Ah, that's Debian.  You need to be in group kvm or root to run KVM.

Looks good, can you include it in the next pull request?

Paolo

> Thomas Huth (5):
>   travis.yml: Re-arrange the test matrix
>   travis.yml: Install only the required packages for each entry in the
>     matrix
>   travis.yml: Test with KVM instead of TCG (on x86)
>   travis.yml: Test the i386 build, too
>   travis.yml: Expect that at least one test succeeds
> 
>  .travis.yml | 155 +++++++++++++++++++++++++++++++++++-----------------
>  1 file changed, 104 insertions(+), 51 deletions(-)
> 

Re: [kvm-unit-test PATCH 0/5] Improvements for the Travis CI

[David Hildenbrand]

On 13.11.19 12:26, Thomas Huth wrote:
> The first two patches make the test matrix a little bit more flexible,
> and the fourth patch enables the 32-bit builds on x86.
> 
> But the most important patch is likely the third one: It is possible to
> test with KVM on Travis now, so we can run the tests within a real KVM
> environment, without TCG! The only caveat is that qemu-system-x86_64
> has to run as root ... fixing only the permissions of /dev/kvm did
> not help here, I still got a "Permission denied" in that case.
> 
> Thomas Huth (5):
>    travis.yml: Re-arrange the test matrix
>    travis.yml: Install only the required packages for each entry in the
>      matrix
>    travis.yml: Test with KVM instead of TCG (on x86)
>    travis.yml: Test the i386 build, too
>    travis.yml: Expect that at least one test succeeds
> 
>   .travis.yml | 155 +++++++++++++++++++++++++++++++++++-----------------
>   1 file changed, 104 insertions(+), 51 deletions(-)
> 

As discussed, queued to

https://github.com/davidhildenbrand/kvm-unit-tests.git s390x-next

for now. PULL request early next week.

-- 

Thanks,

David / dhildenb