[PATCH 0/6] Move remaining x86 Travis jobs to the gitlab-CI

[PATCH 0/6] Move remaining x86 Travis jobs to the gitlab-CI

[Thomas Huth]

Since Travis changed their policies, travis-ci.org will soon become
completely useless for the QEMU project. We should now really make sure
that we move the remaining tests as good as possible to the gitlab-CI
instead. Since the gitlab-CI has already quite a lot of jobs, I tried
to squeeze the missing bits as good as possible into the existing jobs
instead of adding a separate job for each and ever test that we had
in the Travis-CI - I hope that will help to avoid increasing the stress
on the CI system too much. 

Philippe Mathieu-Daudé (1):
  travis.yml: Move gprof/gcov test across to gitlab

Thomas Huth (5):
  travis.yml: Move the -fsanitize=undefined test to the gitlab-CI
  travis.yml: Move the --enable-modules test to the gitlab-CI
  travis.yml: Remove the --enable-debug jobs
  target/s390x/arch_dump: Fixes for the name field in the PT_NOTE
    section
  travis.yml: Move the -fsanitize=thread compile-testing to the
    gitlab-CI

 .gitlab-ci.yml                             |  27 ++++-
 .travis.yml                                | 110 ---------------------
 MAINTAINERS                                |   2 +-
 scripts/{travis => ci}/coverage-summary.sh |   2 +-
 target/s390x/arch_dump.c                   |   6 +-
 tests/docker/dockerfiles/ubuntu2004.docker |   2 +
 6 files changed, 33 insertions(+), 116 deletions(-)
 rename scripts/{travis => ci}/coverage-summary.sh (92%)

-- 
2.27.0

[PATCH 1/6] travis.yml: Move gprof/gcov test across to gitlab

[Thomas Huth]

From: Philippe Mathieu-Daudé <philmd@redhat.com>

Similarly to commit 8cdb2cef3f1, move the gprof/gcov test to GitLab.

The coverage-summary.sh script is not Travis-CI specific, make it
generic.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201108204535.2319870-10-philmd@redhat.com>
[thuth: Add gcovr and bsdmainutils which are required for the
        overage-summary.sh script to the ubuntu docker file]
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml                             | 12 ++++++++++++
 .travis.yml                                | 14 --------------
 MAINTAINERS                                |  2 +-
 scripts/{travis => ci}/coverage-summary.sh |  2 +-
 tests/docker/dockerfiles/ubuntu2004.docker |  2 ++
 5 files changed, 16 insertions(+), 16 deletions(-)
 rename scripts/{travis => ci}/coverage-summary.sh (92%)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 7c0db64710..8b97b512bb 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -468,6 +468,18 @@ check-deprecated:
     MAKE_CHECK_ARGS: check-tcg
   allow_failure: true
 
+# gprof/gcov are GCC features
+build-gprof-gcov:
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --enable-gprof --enable-gcov
+    MAKE_CHECK_ARGS: build-tcg
+    TARGETS: aarch64-softmmu mips64-softmmu ppc64-softmmu
+             riscv64-softmmu s390x-softmmu x86_64-softmmu
+  after_script:
+    - ${CI_PROJECT_DIR}/scripts/ci/coverage-summary.sh
+
 build-oss-fuzz:
   <<: *native_build_job_definition
   variables:
diff --git a/.travis.yml b/.travis.yml
index 5f1dea873e..76b69f6de1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -166,20 +166,6 @@ jobs:
       compiler: clang
 
 
-    # gprof/gcov are GCC features
-    - name: "GCC gprof/gcov"
-      dist: bionic
-      addons:
-        apt:
-          packages:
-            - ninja-build
-      env:
-        - CONFIG="--enable-gprof --enable-gcov --disable-libssh
-                  --target-list=${MAIN_SOFTMMU_TARGETS}"
-      after_success:
-        - ${SRC_DIR}/scripts/travis/coverage-summary.sh
-
-
     # Using newer GCC with sanitizers
     - name: "GCC9 with sanitizers (softmmu)"
       dist: bionic
diff --git a/MAINTAINERS b/MAINTAINERS
index bcd88668bc..f14a2e6eb5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3193,7 +3193,7 @@ R: Philippe Mathieu-Daudé <philmd@redhat.com>
 S: Maintained
 F: .github/lockdown.yml
 F: .travis.yml
-F: scripts/travis/
+F: scripts/ci/
 F: .shippable.yml
 F: tests/docker/
 F: tests/vm/
diff --git a/scripts/travis/coverage-summary.sh b/scripts/ci/coverage-summary.sh
similarity index 92%
rename from scripts/travis/coverage-summary.sh
rename to scripts/ci/coverage-summary.sh
index d7086cf9ca..8d9fb4de40 100755
--- a/scripts/travis/coverage-summary.sh
+++ b/scripts/ci/coverage-summary.sh
@@ -3,7 +3,7 @@
 # Author: Alex Bennée <alex.bennee@linaro.org>
 #
 # Summerise the state of code coverage with gcovr and tweak the output
-# to be more sane on Travis hosts. As we expect to be executed on a
+# to be more sane on CI runner. As we expect to be executed on a
 # throw away CI instance we do spam temp files all over the shop. You
 # most likely don't want to execute this script but just call gcovr
 # directly. See also "make coverage-report"
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 8519584d2b..9750016e51 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -1,8 +1,10 @@
 FROM ubuntu:20.04
 ENV PACKAGES flex bison \
+    bsdmainutils \
     ccache \
     clang-10\
     gcc \
+    gcovr \
     genisoimage \
     gettext \
     git \
-- 
2.27.0

[PATCH 2/6] travis.yml: Move the -fsanitize=undefined test to the gitlab-CI

[Thomas Huth]

Add it to the existing Clang job and also add a job that covers the
linux-user code with this compiler flag. To make sure that the detected
problems are not simply ignored, let's also use "-fno-sanitize-recover=..."
now instead.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 13 +++++++++++--
 .travis.yml    | 27 ---------------------------
 2 files changed, 11 insertions(+), 29 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8b97b512bb..41e11b41e4 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -433,13 +433,22 @@ build-some-softmmu-plugins:
     TARGETS: xtensa-softmmu arm-softmmu aarch64-softmmu alpha-softmmu
     MAKE_CHECK_ARGS: check-tcg
 
-build-clang:
+clang-system:
   <<: *native_build_job_definition
   variables:
     IMAGE: fedora
     CONFIGURE_ARGS: --cc=clang --cxx=clang++
+      --extra-cflags=-fno-sanitize-recover=undefined
     TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
-      ppc-softmmu s390x-softmmu arm-linux-user
+      ppc-softmmu s390x-softmmu
+    MAKE_CHECK_ARGS: check-qtest check-block check-tcg
+
+clang-user:
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
+      --extra-cflags=-fno-sanitize-recover=undefined
     MAKE_CHECK_ARGS: check
 
 # These targets are on the way out
diff --git a/.travis.yml b/.travis.yml
index 76b69f6de1..d1e9016da5 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -139,33 +139,6 @@ jobs:
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
-    # Test with Clang for compile portability (Travis uses clang-5.0)
-    - name: "Clang (user)"
-      env:
-        - CONFIG="--disable-system --host-cc=clang --cxx=clang++"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
-      compiler: clang
-
-
-    - name: "Clang (main-softmmu)"
-      env:
-        - CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS}
-                  --host-cc=clang --cxx=clang++"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-sanitize"
-      compiler: clang
-      before_script:
-        - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
-        - ${SRC_DIR}/configure ${CONFIG} --extra-cflags="-fsanitize=undefined -Werror" || { cat config.log meson-logs/meson-log.txt && exit 1; }
-
-
-    - name: "Clang (other-softmmu)"
-      env:
-        - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}
-                  --host-cc=clang --cxx=clang++"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
-      compiler: clang
-
-
     # Using newer GCC with sanitizers
     - name: "GCC9 with sanitizers (softmmu)"
       dist: bionic
-- 
2.27.0

[PATCH 3/6] travis.yml: Move the --enable-modules test to the gitlab-CI

[Thomas Huth]

Simply add the flag to an existing job, no need for yet another
job here.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 1 +
 .travis.yml    | 6 ------
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 41e11b41e4..4654798523 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -223,6 +223,7 @@ build-system-centos:
   variables:
     IMAGE: centos8
     CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
+                    --enable-modules
     TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
     MAKE_CHECK_ARGS: check-build
diff --git a/.travis.yml b/.travis.yml
index d1e9016da5..45dd017420 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -132,12 +132,6 @@ jobs:
         - CONFIG="--enable-debug-tcg --disable-system"
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
 
-    # Module builds are mostly of interest to major distros
-    - name: "GCC modules (main-softmmu)"
-      env:
-        - CONFIG="--enable-modules --target-list=${MAIN_SOFTMMU_TARGETS}"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
-
 
     # Using newer GCC with sanitizers
     - name: "GCC9 with sanitizers (softmmu)"
-- 
2.27.0

[PATCH 4/6] travis.yml: Remove the --enable-debug jobs

[Thomas Huth]

We already have such jobs in the gitlab-CI ("build-some-softmmu" and
"build-user-plugins"), so we can simply drop these from the Travis-CI.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .travis.yml | 12 ------------
 1 file changed, 12 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 45dd017420..b3fc72f561 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -119,18 +119,6 @@ after_script:
 
 jobs:
   include:
-    # --enable-debug implies --enable-debug-tcg, also runs quite a bit slower
-    - name: "GCC debug (main-softmmu)"
-      env:
-        - CONFIG="--enable-debug --target-list=${MAIN_SOFTMMU_TARGETS}"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
-
-
-    # TCG debug can be run just on its own and is mostly agnostic to user/softmmu distinctions
-    - name: "GCC debug (user)"
-      env:
-        - CONFIG="--enable-debug-tcg --disable-system"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
 
 
     # Using newer GCC with sanitizers
-- 
2.27.0

[PATCH 5/6] target/s390x/arch_dump: Fixes for the name field in the PT_NOTE section

[Thomas Huth]

According to the "ELF-64 Object File Format" specification:

"The first word in the entry, namesz, identifies the length, in
 bytes, of a name identifying the entry’s owner or originator. The name field
 contains a null-terminated string, with padding as necessary to ensure 8-
 byte alignment for the descriptor field. The length does not include the
 terminating null or the padding."

So we should not include the terminating NUL in the length field here.

Also there is a compiler warning with GCC 9.3 when compiling with
the -fsanitize=thread compiler flag:

 In function 'strncpy',
    inlined from 's390x_write_elf64_notes' at ../target/s390x/arch_dump.c:219:9:
 /usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: error:
  '__builtin_strncpy' specified bound 8 equals destination size
  [-Werror=stringop-truncation]

Since the name should always be NUL-terminated, we can simply decrease
the size of the strncpy by one here to silence this warning. And while
we're at it, also add an assert() to make sure that the provided names
always fit the size field (which is fine for the current callers, the
function is called once with "CORE" and once with "LINUX" as a name).

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 target/s390x/arch_dump.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/target/s390x/arch_dump.c b/target/s390x/arch_dump.c
index 50fa0ae4b6..20c3a09707 100644
--- a/target/s390x/arch_dump.c
+++ b/target/s390x/arch_dump.c
@@ -212,11 +212,13 @@ static int s390x_write_elf64_notes(const char *note_name,
     int note_size;
     int ret = -1;
 
+    assert(strlen(note_name) < sizeof(note.name));
+
     for (nf = funcs; nf->note_contents_func; nf++) {
         memset(&note, 0, sizeof(note));
-        note.hdr.n_namesz = cpu_to_be32(strlen(note_name) + 1);
+        note.hdr.n_namesz = cpu_to_be32(strlen(note_name));
         note.hdr.n_descsz = cpu_to_be32(nf->contents_size);
-        strncpy(note.name, note_name, sizeof(note.name));
+        strncpy(note.name, note_name, sizeof(note.name) - 1);
         (*nf->note_contents_func)(&note, cpu, id);
 
         note_size = sizeof(note) - sizeof(note.contents) + nf->contents_size;
-- 
2.27.0

[PATCH 6/6] travis.yml: Move the -fsanitize=thread compile-testing to the gitlab-CI

[Thomas Huth]

It's only about compile-testing (there is too much noise when running
the tests), so let's simply add the -fsanitize=thread flag to a job that
only compiles the sources. The "build-gprof-gcov" seems to be a good
candidate.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml |  1 +
 .travis.yml    | 51 --------------------------------------------------
 2 files changed, 1 insertion(+), 51 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 4654798523..e5c86e38c4 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -484,6 +484,7 @@ build-gprof-gcov:
   variables:
     IMAGE: ubuntu2004
     CONFIGURE_ARGS: --enable-gprof --enable-gcov
+                    --extra-cflags=-fsanitize=thread
     MAKE_CHECK_ARGS: build-tcg
     TARGETS: aarch64-softmmu mips64-softmmu ppc64-softmmu
              riscv64-softmmu s390x-softmmu x86_64-softmmu
diff --git a/.travis.yml b/.travis.yml
index b3fc72f561..18e62f282f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -120,57 +120,6 @@ after_script:
 jobs:
   include:
 
-
-    # Using newer GCC with sanitizers
-    - name: "GCC9 with sanitizers (softmmu)"
-      dist: bionic
-      addons:
-        apt:
-          update: true
-          sources:
-            # PPAs for newer toolchains
-            - ubuntu-toolchain-r-test
-          packages:
-            # Extra toolchains
-            - gcc-9
-            - g++-9
-            # Build dependencies
-            - libaio-dev
-            - libattr1-dev
-            - libbrlapi-dev
-            - libcap-ng-dev
-            - libgnutls28-dev
-            - libgtk-3-dev
-            - libiscsi-dev
-            - liblttng-ust-dev
-            - libnfs-dev
-            - libncurses5-dev
-            - libnss3-dev
-            - libpixman-1-dev
-            - libpng-dev
-            - librados-dev
-            - libsdl2-dev
-            - libsdl2-image-dev
-            - libseccomp-dev
-            - libspice-protocol-dev
-            - libspice-server-dev
-            - liburcu-dev
-            - libusb-1.0-0-dev
-            - libvte-2.91-dev
-            - ninja-build
-            - sparse
-            - uuid-dev
-      language: generic
-      compiler: none
-      env:
-        - COMPILER_NAME=gcc CXX=g++-9 CC=gcc-9
-        - CONFIG="--cc=gcc-9 --cxx=g++-9 --disable-linux-user"
-        - TEST_CMD=""
-      before_script:
-        - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
-        - ${SRC_DIR}/configure ${CONFIG} --extra-cflags="-g3 -O0 -fsanitize=thread" || { cat config.log meson-logs/meson-log.txt && exit 1; }
-
-
     - name: "[aarch64] GCC check-tcg"
       arch: arm64
       dist: focal
-- 
2.27.0

Re: [PATCH 3/6] travis.yml: Move the --enable-modules test to the gitlab-CI

[Wainer dos Santos Moschetta]

On 2/3/21 8:32 AM, Thomas Huth wrote:
> Simply add the flag to an existing job, no need for yet another
> job here.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .gitlab-ci.yml | 1 +
>   .travis.yml    | 6 ------
>   2 files changed, 1 insertion(+), 6 deletions(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 41e11b41e4..4654798523 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -223,6 +223,7 @@ build-system-centos:
>     variables:
>       IMAGE: centos8
>       CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
> +                    --enable-modules
>       TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
>         x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
>       MAKE_CHECK_ARGS: check-build
> diff --git a/.travis.yml b/.travis.yml
> index d1e9016da5..45dd017420 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -132,12 +132,6 @@ jobs:
>           - CONFIG="--enable-debug-tcg --disable-system"
>           - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
>   
> -    # Module builds are mostly of interest to major distros
> -    - name: "GCC modules (main-softmmu)"
> -      env:
> -        - CONFIG="--enable-modules --target-list=${MAIN_SOFTMMU_TARGETS}"
> -        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
> -
>   
>       # Using newer GCC with sanitizers
>       - name: "GCC9 with sanitizers (softmmu)"

Re: [PATCH 4/6] travis.yml: Remove the --enable-debug jobs

[Wainer dos Santos Moschetta]

Hi,

On 2/3/21 8:32 AM, Thomas Huth wrote:
> We already have such jobs in the gitlab-CI ("build-some-softmmu" and
> "build-user-plugins"), so we can simply drop these from the Travis-CI.

Those jobs --enable-debug-tcg. Shouldn't at least one --enable-debug?

- Wainer

>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .travis.yml | 12 ------------
>   1 file changed, 12 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 45dd017420..b3fc72f561 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -119,18 +119,6 @@ after_script:
>   
>   jobs:
>     include:
> -    # --enable-debug implies --enable-debug-tcg, also runs quite a bit slower
> -    - name: "GCC debug (main-softmmu)"
> -      env:
> -        - CONFIG="--enable-debug --target-list=${MAIN_SOFTMMU_TARGETS}"
> -        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
> -
> -
> -    # TCG debug can be run just on its own and is mostly agnostic to user/softmmu distinctions
> -    - name: "GCC debug (user)"
> -      env:
> -        - CONFIG="--enable-debug-tcg --disable-system"
> -        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
>   
>   
>       # Using newer GCC with sanitizers

Re: [PATCH 6/6] travis.yml: Move the -fsanitize=thread compile-testing to the gitlab-CI

[Wainer dos Santos Moschetta]

Hi,

On 2/3/21 8:32 AM, Thomas Huth wrote:
> It's only about compile-testing (there is too much noise when running
> the tests), so let's simply add the -fsanitize=thread flag to a job that
> only compiles the sources. The "build-gprof-gcov" seems to be a good
> candidate.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .gitlab-ci.yml |  1 +
>   .travis.yml    | 51 --------------------------------------------------
>   2 files changed, 1 insertion(+), 51 deletions(-)

It will be a little weird to see a gprof/gcov failing with a sanitize 
error, but it seems indeed the best job.

Allow me to double-check: doesn't it need -g to show the line numbers?

- Wainer

>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 4654798523..e5c86e38c4 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -484,6 +484,7 @@ build-gprof-gcov:
>     variables:
>       IMAGE: ubuntu2004
>       CONFIGURE_ARGS: --enable-gprof --enable-gcov
> +                    --extra-cflags=-fsanitize=thread
>       MAKE_CHECK_ARGS: build-tcg
>       TARGETS: aarch64-softmmu mips64-softmmu ppc64-softmmu
>                riscv64-softmmu s390x-softmmu x86_64-softmmu
> diff --git a/.travis.yml b/.travis.yml
> index b3fc72f561..18e62f282f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -120,57 +120,6 @@ after_script:
>   jobs:
>     include:
>   
> -
> -    # Using newer GCC with sanitizers
> -    - name: "GCC9 with sanitizers (softmmu)"
> -      dist: bionic
> -      addons:
> -        apt:
> -          update: true
> -          sources:
> -            # PPAs for newer toolchains
> -            - ubuntu-toolchain-r-test
> -          packages:
> -            # Extra toolchains
> -            - gcc-9
> -            - g++-9
> -            # Build dependencies
> -            - libaio-dev
> -            - libattr1-dev
> -            - libbrlapi-dev
> -            - libcap-ng-dev
> -            - libgnutls28-dev
> -            - libgtk-3-dev
> -            - libiscsi-dev
> -            - liblttng-ust-dev
> -            - libnfs-dev
> -            - libncurses5-dev
> -            - libnss3-dev
> -            - libpixman-1-dev
> -            - libpng-dev
> -            - librados-dev
> -            - libsdl2-dev
> -            - libsdl2-image-dev
> -            - libseccomp-dev
> -            - libspice-protocol-dev
> -            - libspice-server-dev
> -            - liburcu-dev
> -            - libusb-1.0-0-dev
> -            - libvte-2.91-dev
> -            - ninja-build
> -            - sparse
> -            - uuid-dev
> -      language: generic
> -      compiler: none
> -      env:
> -        - COMPILER_NAME=gcc CXX=g++-9 CC=gcc-9
> -        - CONFIG="--cc=gcc-9 --cxx=g++-9 --disable-linux-user"
> -        - TEST_CMD=""
> -      before_script:
> -        - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
> -        - ${SRC_DIR}/configure ${CONFIG} --extra-cflags="-g3 -O0 -fsanitize=thread" || { cat config.log meson-logs/meson-log.txt && exit 1; }
> -
> -
>       - name: "[aarch64] GCC check-tcg"
>         arch: arm64
>         dist: focal

Re: [PATCH 1/6] travis.yml: Move gprof/gcov test across to gitlab

[Wainer dos Santos Moschetta]

Hi,

On 2/3/21 8:32 AM, Thomas Huth wrote:
> From: Philippe Mathieu-Daudé <philmd@redhat.com>
>
> Similarly to commit 8cdb2cef3f1, move the gprof/gcov test to GitLab.
>
> The coverage-summary.sh script is not Travis-CI specific, make it
> generic.
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Message-Id: <20201108204535.2319870-10-philmd@redhat.com>
> [thuth: Add gcovr and bsdmainutils which are required for the
>          overage-summary.sh script to the ubuntu docker file]
s/overage/coverage/
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .gitlab-ci.yml                             | 12 ++++++++++++
>   .travis.yml                                | 14 --------------
>   MAINTAINERS                                |  2 +-
>   scripts/{travis => ci}/coverage-summary.sh |  2 +-
>   tests/docker/dockerfiles/ubuntu2004.docker |  2 ++
>   5 files changed, 16 insertions(+), 16 deletions(-)
>   rename scripts/{travis => ci}/coverage-summary.sh (92%)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 7c0db64710..8b97b512bb 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -468,6 +468,18 @@ check-deprecated:
>       MAKE_CHECK_ARGS: check-tcg
>     allow_failure: true
>   
> +# gprof/gcov are GCC features
> +build-gprof-gcov:
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: ubuntu2004
> +    CONFIGURE_ARGS: --enable-gprof --enable-gcov
> +    MAKE_CHECK_ARGS: build-tcg

With build-tcg it generates an empty report, e.g., 
https://gitlab.com/wainersm/qemu/-/jobs/1005923421

Shouldn't it run `make check`?

- Wainer

> +    TARGETS: aarch64-softmmu mips64-softmmu ppc64-softmmu
> +             riscv64-softmmu s390x-softmmu x86_64-softmmu
> +  after_script:
> +    - ${CI_PROJECT_DIR}/scripts/ci/coverage-summary.sh
> +
>   build-oss-fuzz:
>     <<: *native_build_job_definition
>     variables:
> diff --git a/.travis.yml b/.travis.yml
> index 5f1dea873e..76b69f6de1 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -166,20 +166,6 @@ jobs:
>         compiler: clang
>   
>   
> -    # gprof/gcov are GCC features
> -    - name: "GCC gprof/gcov"
> -      dist: bionic
> -      addons:
> -        apt:
> -          packages:
> -            - ninja-build
> -      env:
> -        - CONFIG="--enable-gprof --enable-gcov --disable-libssh
> -                  --target-list=${MAIN_SOFTMMU_TARGETS}"
> -      after_success:
> -        - ${SRC_DIR}/scripts/travis/coverage-summary.sh
> -
> -
>       # Using newer GCC with sanitizers
>       - name: "GCC9 with sanitizers (softmmu)"
>         dist: bionic
> diff --git a/MAINTAINERS b/MAINTAINERS
> index bcd88668bc..f14a2e6eb5 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3193,7 +3193,7 @@ R: Philippe Mathieu-Daudé <philmd@redhat.com>
>   S: Maintained
>   F: .github/lockdown.yml
>   F: .travis.yml
> -F: scripts/travis/
> +F: scripts/ci/
>   F: .shippable.yml
>   F: tests/docker/
>   F: tests/vm/
> diff --git a/scripts/travis/coverage-summary.sh b/scripts/ci/coverage-summary.sh
> similarity index 92%
> rename from scripts/travis/coverage-summary.sh
> rename to scripts/ci/coverage-summary.sh
> index d7086cf9ca..8d9fb4de40 100755
> --- a/scripts/travis/coverage-summary.sh
> +++ b/scripts/ci/coverage-summary.sh
> @@ -3,7 +3,7 @@
>   # Author: Alex Bennée <alex.bennee@linaro.org>
>   #
>   # Summerise the state of code coverage with gcovr and tweak the output
> -# to be more sane on Travis hosts. As we expect to be executed on a
> +# to be more sane on CI runner. As we expect to be executed on a
>   # throw away CI instance we do spam temp files all over the shop. You
>   # most likely don't want to execute this script but just call gcovr
>   # directly. See also "make coverage-report"
> diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
> index 8519584d2b..9750016e51 100644
> --- a/tests/docker/dockerfiles/ubuntu2004.docker
> +++ b/tests/docker/dockerfiles/ubuntu2004.docker
> @@ -1,8 +1,10 @@
>   FROM ubuntu:20.04
>   ENV PACKAGES flex bison \
> +    bsdmainutils \
>       ccache \
>       clang-10\
>       gcc \
> +    gcovr \
>       genisoimage \
>       gettext \
>       git \

Re: [PATCH 6/6] travis.yml: Move the -fsanitize=thread compile-testing to the gitlab-CI

[Wainer dos Santos Moschetta]

On 2/3/21 4:23 PM, Wainer dos Santos Moschetta wrote:
> Hi,
>
> On 2/3/21 8:32 AM, Thomas Huth wrote:
>> It's only about compile-testing (there is too much noise when running
>> the tests), so let's simply add the -fsanitize=thread flag to a job that
>> only compiles the sources. The "build-gprof-gcov" seems to be a good
>> candidate.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   .gitlab-ci.yml |  1 +
>>   .travis.yml    | 51 --------------------------------------------------
>>   2 files changed, 1 insertion(+), 51 deletions(-)
>
> It will be a little weird to see a gprof/gcov failing with a sanitize 
> error, but it seems indeed the best job.
>
> Allow me to double-check: doesn't it need -g to show the line numbers?

Never mind. It compiles with "-O2 -g". So:

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

>
>
> - Wainer
>
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 4654798523..e5c86e38c4 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -484,6 +484,7 @@ build-gprof-gcov:
>>     variables:
>>       IMAGE: ubuntu2004
>>       CONFIGURE_ARGS: --enable-gprof --enable-gcov
>> +                    --extra-cflags=-fsanitize=thread
>>       MAKE_CHECK_ARGS: build-tcg
>>       TARGETS: aarch64-softmmu mips64-softmmu ppc64-softmmu
>>                riscv64-softmmu s390x-softmmu x86_64-softmmu
>> diff --git a/.travis.yml b/.travis.yml
>> index b3fc72f561..18e62f282f 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -120,57 +120,6 @@ after_script:
>>   jobs:
>>     include:
>>   -
>> -    # Using newer GCC with sanitizers
>> -    - name: "GCC9 with sanitizers (softmmu)"
>> -      dist: bionic
>> -      addons:
>> -        apt:
>> -          update: true
>> -          sources:
>> -            # PPAs for newer toolchains
>> -            - ubuntu-toolchain-r-test
>> -          packages:
>> -            # Extra toolchains
>> -            - gcc-9
>> -            - g++-9
>> -            # Build dependencies
>> -            - libaio-dev
>> -            - libattr1-dev
>> -            - libbrlapi-dev
>> -            - libcap-ng-dev
>> -            - libgnutls28-dev
>> -            - libgtk-3-dev
>> -            - libiscsi-dev
>> -            - liblttng-ust-dev
>> -            - libnfs-dev
>> -            - libncurses5-dev
>> -            - libnss3-dev
>> -            - libpixman-1-dev
>> -            - libpng-dev
>> -            - librados-dev
>> -            - libsdl2-dev
>> -            - libsdl2-image-dev
>> -            - libseccomp-dev
>> -            - libspice-protocol-dev
>> -            - libspice-server-dev
>> -            - liburcu-dev
>> -            - libusb-1.0-0-dev
>> -            - libvte-2.91-dev
>> -            - ninja-build
>> -            - sparse
>> -            - uuid-dev
>> -      language: generic
>> -      compiler: none
>> -      env:
>> -        - COMPILER_NAME=gcc CXX=g++-9 CC=gcc-9
>> -        - CONFIG="--cc=gcc-9 --cxx=g++-9 --disable-linux-user"
>> -        - TEST_CMD=""
>> -      before_script:
>> -        - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
>> -        - ${SRC_DIR}/configure ${CONFIG} --extra-cflags="-g3 -O0 
>> -fsanitize=thread" || { cat config.log meson-logs/meson-log.txt && 
>> exit 1; }
>> -
>> -
>>       - name: "[aarch64] GCC check-tcg"
>>         arch: arm64
>>         dist: focal
>
>

Re: [PATCH 1/6] travis.yml: Move gprof/gcov test across to gitlab

[Thomas Huth]

On 03/02/2021 20.32, Wainer dos Santos Moschetta wrote:
> Hi,
> 
> On 2/3/21 8:32 AM, Thomas Huth wrote:
>> From: Philippe Mathieu-Daudé <philmd@redhat.com>
>>
>> Similarly to commit 8cdb2cef3f1, move the gprof/gcov test to GitLab.
>>
>> The coverage-summary.sh script is not Travis-CI specific, make it
>> generic.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> Message-Id: <20201108204535.2319870-10-philmd@redhat.com>
>> [thuth: Add gcovr and bsdmainutils which are required for the
>>          overage-summary.sh script to the ubuntu docker file]
> s/overage/coverage/
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   .gitlab-ci.yml                             | 12 ++++++++++++
>>   .travis.yml                                | 14 --------------
>>   MAINTAINERS                                |  2 +-
>>   scripts/{travis => ci}/coverage-summary.sh |  2 +-
>>   tests/docker/dockerfiles/ubuntu2004.docker |  2 ++
>>   5 files changed, 16 insertions(+), 16 deletions(-)
>>   rename scripts/{travis => ci}/coverage-summary.sh (92%)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 7c0db64710..8b97b512bb 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -468,6 +468,18 @@ check-deprecated:
>>       MAKE_CHECK_ARGS: check-tcg
>>     allow_failure: true
>> +# gprof/gcov are GCC features
>> +build-gprof-gcov:
>> +  <<: *native_build_job_definition
>> +  variables:
>> +    IMAGE: ubuntu2004
>> +    CONFIGURE_ARGS: --enable-gprof --enable-gcov
>> +    MAKE_CHECK_ARGS: build-tcg
> 
> With build-tcg it generates an empty report, e.g., 
> https://gitlab.com/wainersm/qemu/-/jobs/1005923421
> 
> Shouldn't it run `make check`?

D'oh, you're right. I think we need to run at least a "make check-unit" 
here. I'll rework my patch accordingly...

By the way, it's broken on Travis since a long time, e.g. with version 5.0 
there is already only a stack trace:

https://travis-ci.org/github/qemu/qemu/jobs/680661167#L8411

Seems like nobody noticed this for almost a year now...

  Thomas

Re: [PATCH 1/6] travis.yml: Move gprof/gcov test across to gitlab

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 03/02/2021 20.32, Wainer dos Santos Moschetta wrote:
>> Hi,
>> 
>> On 2/3/21 8:32 AM, Thomas Huth wrote:
>>> From: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>
>>> Similarly to commit 8cdb2cef3f1, move the gprof/gcov test to GitLab.
>>>
>>> The coverage-summary.sh script is not Travis-CI specific, make it
>>> generic.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>> Message-Id: <20201108204535.2319870-10-philmd@redhat.com>
>>> [thuth: Add gcovr and bsdmainutils which are required for the
>>>          overage-summary.sh script to the ubuntu docker file]
>> s/overage/coverage/
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>   .gitlab-ci.yml                             | 12 ++++++++++++
>>>   .travis.yml                                | 14 --------------
>>>   MAINTAINERS                                |  2 +-
>>>   scripts/{travis => ci}/coverage-summary.sh |  2 +-
>>>   tests/docker/dockerfiles/ubuntu2004.docker |  2 ++
>>>   5 files changed, 16 insertions(+), 16 deletions(-)
>>>   rename scripts/{travis => ci}/coverage-summary.sh (92%)
>>>
>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>> index 7c0db64710..8b97b512bb 100644
>>> --- a/.gitlab-ci.yml
>>> +++ b/.gitlab-ci.yml
>>> @@ -468,6 +468,18 @@ check-deprecated:
>>>       MAKE_CHECK_ARGS: check-tcg
>>>     allow_failure: true
>>> +# gprof/gcov are GCC features
>>> +build-gprof-gcov:
>>> +  <<: *native_build_job_definition
>>> +  variables:
>>> +    IMAGE: ubuntu2004
>>> +    CONFIGURE_ARGS: --enable-gprof --enable-gcov
>>> +    MAKE_CHECK_ARGS: build-tcg
>> 
>> With build-tcg it generates an empty report, e.g., 
>> https://gitlab.com/wainersm/qemu/-/jobs/1005923421
>> 
>> Shouldn't it run `make check`?
>
> D'oh, you're right. I think we need to run at least a "make check-unit" 
> here. I'll rework my patch accordingly...
>
> By the way, it's broken on Travis since a long time, e.g. with version 5.0 
> there is already only a stack trace:
>
> https://travis-ci.org/github/qemu/qemu/jobs/680661167#L8411
>
> Seems like nobody noticed this for almost a year now...

doh - the check_coverage was an after_success step so never influenced
the result. It was only a band aid really though - it would be better if
we published the html coverage report on gitlab's pages (like we now do
for annotated gtags source: https://qemu-project.gitlab.io/qemu/src/

>
>   Thomas


-- 
Alex Bennée