[PULL 0/7] Gitlab CI fixes and improvements

[PULL 0/7] Gitlab CI fixes and improvements

[Thomas Huth]

 Hi Peter,
 
the following changes since commit 06539ebc76b8625587aa78d646a9d8d5fddf84f3:

  Merge remote-tracking branch 'remotes/philmd-gitlab/tags/mips-hw-next-20200526' into staging (2020-05-26 20:25:06 +0100)

are available in the Git repository at:

  https://gitlab.com/huth/qemu.git tags/pull-request-2020-05-28

for you to fetch changes up to 0016afa250d251a1b3fbc90c6d618e7f00e02f14:

  gitlab-ci: Determine the number of jobs dynamically (2020-05-28 11:01:38 +0200)

----------------------------------------------------------------
- Build with other images instead of the broken Debian containers
- Fix building with the latest version of Clang (at least wrt. to
  the gitlab-CI pipeline)
- Add Philippe, Alex and Wainer to the Gitlab-CI section in MAINTAINERS
----------------------------------------------------------------

Alex Bennée (1):
      linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS

Cleber Rosa (1):
      GitLab CI: avoid calling before_scripts on unintended jobs

Philippe Mathieu-Daudé (1):
      gitlab-ci: Remove flex/bison packages

Thomas Huth (4):
      MAINTAINERS: Add Philippe, Alex and Wainer to the Gitlab-CI section
      gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder
      gitlab-ci: Do not use the standard container images from gitlab
      gitlab-ci: Determine the number of jobs dynamically

 .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml       |  0
 .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml |  0
 .gitlab-ci.yml                                     | 67 +++++++++++++++-------
 MAINTAINERS                                        |  5 +-
 linux-user/elfload.c                               |  2 +
 5 files changed, 51 insertions(+), 23 deletions(-)
 rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
 rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)

[PULL 1/7] linux-user: limit check to HOST_LONG_BITS < TARGET_ABI_BITS

[Thomas Huth]

From: Alex Bennée <alex.bennee@linaro.org>

Newer clangs rightly spot that you can never exceed the full address
space of 64 bit hosts with:

  linux-user/elfload.c:2076:41: error: result of comparison 'unsigned
  long' > 18446744073709551615 is always false
  [-Werror,-Wtautological-type-limit-compare]
  4685         if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
  4686             ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~
  4687 1 error generated.

So lets limit the check to 32 bit hosts only.

Fixes: ee94743034bf
Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200525131823.715-8-thuth@redhat.com>
[thuth: Use HOST_LONG_BITS < TARGET_ABI_BITS instead of HOST_LONG_BITS == 32]
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 linux-user/elfload.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/linux-user/elfload.c b/linux-user/elfload.c
index 01a9323a63..ebc663ea0b 100644
--- a/linux-user/elfload.c
+++ b/linux-user/elfload.c
@@ -2073,12 +2073,14 @@ static void pgb_have_guest_base(const char *image_name, abi_ulong guest_loaddr,
             exit(EXIT_FAILURE);
         }
     } else {
+#if HOST_LONG_BITS < TARGET_ABI_BITS
         if ((guest_hiaddr - guest_base) > ~(uintptr_t)0) {
             error_report("%s: requires more virtual address space "
                          "than the host can provide (0x%" PRIx64 ")",
                          image_name, (uint64_t)guest_hiaddr - guest_base);
             exit(EXIT_FAILURE);
         }
+#endif
     }
 
     /*
-- 
2.18.1

[PULL 2/7] MAINTAINERS: Add Philippe, Alex and Wainer to the Gitlab-CI section

[Thomas Huth]

Initially, I was the only one who was using Gitlab while most developers
had their git trees still on other systems, but that has changed nowadays.
There is now much more interest in the Gitlab-CI today, so it would be
good to have more than only one maintainer / reviewer for the gitlab-ci.yml
file. Alex, Wainer and Philippe kindly offered their help here, so let's
add them to the corresponding section in the MAINTAINERS file now.

Message-Id: <20200210155115.9371-1-thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 MAINTAINERS | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index a209b5d8ce..71a0438843 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2901,6 +2901,9 @@ W: https://cirrus-ci.com/github/qemu/qemu
 
 GitLab Continuous Integration
 M: Thomas Huth <thuth@redhat.com>
+M: Philippe Mathieu-Daudé <philmd@redhat.com>
+M: Alex Bennée <alex.bennee@linaro.org>
+R: Wainer dos Santos Moschetta <wainersm@redhat.com>
 S: Maintained
 F: .gitlab-ci.yml
 
-- 
2.18.1

[PULL 3/7] gitlab-ci: Remove flex/bison packages

[Thomas Huth]

From: Philippe Mathieu-Daudé <philmd@redhat.com>

QEMU does not use flex/bison packages.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200515163029.12917-4-philmd@redhat.com>
Message-Id: <20200525131823.715-3-thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b889fb96b6..994774250f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -4,7 +4,7 @@ include:
 
 before_script:
  - apt-get update -qq
- - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
+ - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
 
 build-system1:
  script:
-- 
2.18.1

[PULL 4/7] GitLab CI: avoid calling before_scripts on unintended jobs

[Thomas Huth]

From: Cleber Rosa <crosa@redhat.com>

At this point it seems that all jobs depend on those steps, with
maybe the EDK2 jobs as exceptions.

The jobs that will be added later will not want those scripts to be
run, so let's move these steps to the appropriate jobs, while
still trying to avoid repetition.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200525131823.715-4-thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
[thuth: Rebased to current master branch, use separate template]
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 994774250f..bc6aee6aba 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -2,11 +2,13 @@ include:
   - local: '/.gitlab-ci-edk2.yml'
   - local: '/.gitlab-ci-opensbi.yml'
 
-before_script:
- - apt-get update -qq
- - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
+.update_apt_template: &before_script_apt
+ before_script:
+  - apt-get update -qq
+  - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
 
 build-system1:
+ <<: *before_script_apt
  script:
  - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
       libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev
@@ -19,6 +21,7 @@ build-system1:
  - make -j2 check
 
 build-system2:
+ <<: *before_script_apt
  script:
  - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev
       libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev
@@ -32,6 +35,7 @@ build-system2:
  - make -j2 check
 
 build-disabled:
+ <<: *before_script_apt
  script:
  - mkdir build
  - cd build
@@ -46,6 +50,7 @@ build-disabled:
  - make -j2 check-qtest SPEED=slow
 
 build-tcg-disabled:
+ <<: *before_script_apt
  script:
  - apt-get install -y -qq clang libgtk-3-dev libusb-dev
  - mkdir build
@@ -64,6 +69,7 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
+ <<: *before_script_apt
  script:
  - mkdir build
  - cd build
@@ -73,6 +79,7 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
+ <<: *before_script_apt
  script:
  - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev
       xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev
@@ -85,6 +92,7 @@ build-clang:
  - make -j2 check
 
 build-tci:
+ <<: *before_script_apt
  script:
  - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
  - mkdir build
-- 
2.18.1

[PULL 5/7] gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder

[Thomas Huth]

We have a dedicated folder for the gitlab-ci - so there is no need
to clutter the top directory with these .yml files.

Message-Id: <20200525131823.715-5-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml       | 0
 .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0
 .gitlab-ci.yml                                     | 4 ++--
 MAINTAINERS                                        | 2 +-
 4 files changed, 3 insertions(+), 3 deletions(-)
 rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
 rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)

diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci.d/edk2.yml
similarity index 100%
rename from .gitlab-ci-edk2.yml
rename to .gitlab-ci.d/edk2.yml
diff --git a/.gitlab-ci-opensbi.yml b/.gitlab-ci.d/opensbi.yml
similarity index 100%
rename from .gitlab-ci-opensbi.yml
rename to .gitlab-ci.d/opensbi.yml
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index bc6aee6aba..5208d93ff8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,6 +1,6 @@
 include:
-  - local: '/.gitlab-ci-edk2.yml'
-  - local: '/.gitlab-ci-opensbi.yml'
+  - local: '/.gitlab-ci.d/edk2.yml'
+  - local: '/.gitlab-ci.d/opensbi.yml'
 
 .update_apt_template: &before_script_apt
  before_script:
diff --git a/MAINTAINERS b/MAINTAINERS
index 71a0438843..0944d9c731 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2542,7 +2542,7 @@ F: roms/edk2
 F: roms/edk2-*
 F: tests/data/uefi-boot-images/
 F: tests/uefi-test-tools/
-F: .gitlab-ci-edk2.yml
+F: .gitlab-ci.d/edk2.yml
 F: .gitlab-ci.d/edk2/
 
 Usermode Emulation
-- 
2.18.1

[PULL 6/7] gitlab-ci: Do not use the standard container images from gitlab

[Thomas Huth]

Currently all pipelines of the gitlab CI are failing, except for the
"build-user" pipeline. There is an issue with the default container
image (likely Debian stable) where they imported something bad in one
of the system headers:

 /usr/include/linux/swab.h: In function '__swab':
 /builds/huth/qemu/include/qemu/bitops.h:20:34: error: "sizeof" is not
  defined, evaluates to 0 [-Werror=undef]
 #define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)

We could maybe work-around this issue or wait for the default containers
to get fixed, but considering that we use Ubuntu (and thus Debian-style)
CI in Travis already to a very large extent, we should consider to use
some RPM-based distros in our gitlab CI instead. Thus let's change the
failing pipelines to use Fedora and CentOS (and also one Ubuntu 19.10,
since 20.04 is broken, too) now.

Message-Id: <20200525131823.715-6-thuth@redhat.com>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 37 +++++++++++++++++++++++++------------
 1 file changed, 25 insertions(+), 12 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 5208d93ff8..559ec2ab4d 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -5,9 +5,17 @@ include:
 .update_apt_template: &before_script_apt
  before_script:
   - apt-get update -qq
-  - apt-get install -y -qq libglib2.0-dev libpixman-1-dev genisoimage
+  - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make
+        genisoimage
+
+.update_dnf_template: &before_script_dnf
+ before_script:
+  - dnf update -y
+  - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel
+        make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel
 
 build-system1:
+ image: ubuntu:19.10
  <<: *before_script_apt
  script:
  - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
@@ -21,11 +29,12 @@ build-system1:
  - make -j2 check
 
 build-system2:
- <<: *before_script_apt
+ image: fedora:latest
+ <<: *before_script_dnf
  script:
- - apt-get install -y -qq libsdl2-dev libgcrypt-dev libbrlapi-dev libaio-dev
-      libfdt-dev liblzo2-dev librdmacm-dev libibverbs-dev libibumad-dev
-      libzstd-dev
+ - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel
+       libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel
+       libzstd-devel
  - mkdir build
  - cd build
  - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
@@ -35,7 +44,8 @@ build-system2:
  - make -j2 check
 
 build-disabled:
- <<: *before_script_apt
+ image: fedora:latest
+ <<: *before_script_dnf
  script:
  - mkdir build
  - cd build
@@ -50,9 +60,10 @@ build-disabled:
  - make -j2 check-qtest SPEED=slow
 
 build-tcg-disabled:
- <<: *before_script_apt
+ image: centos:8
+ <<: *before_script_dnf
  script:
- - apt-get install -y -qq clang libgtk-3-dev libusb-dev
+ - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel
  - mkdir build
  - cd build
  - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
@@ -79,10 +90,11 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
- <<: *before_script_apt
+ image: fedora:latest
+ <<: *before_script_dnf
  script:
- - apt-get install -y -qq clang libsdl2-dev libattr1-dev libcap-ng-dev
-      xfslibs-dev libiscsi-dev libnfs-dev libseccomp-dev gnutls-dev librbd-dev
+ - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel
+       libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel
  - mkdir build
  - cd build
  - ../configure --cc=clang --cxx=clang++ --enable-werror
@@ -92,7 +104,8 @@ build-clang:
  - make -j2 check
 
 build-tci:
- <<: *before_script_apt
+ image: centos:8
+ <<: *before_script_dnf
  script:
  - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
  - mkdir build
-- 
2.18.1

[PULL 7/7] gitlab-ci: Determine the number of jobs dynamically

[Thomas Huth]

Some people might want to run the gitlab CI pipelines in an environment
where multiple CPUs are available to the runners, so let's rather get
the number for "-j" from the "nproc" program (increased by 1 to compensate
for jobs that wait for I/O) instead of hard-coding it.

Message-Id: <20200525131823.715-7-thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 559ec2ab4d..349c77aa58 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -7,12 +7,14 @@ include:
   - apt-get update -qq
   - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make
         genisoimage
+  - JOBS=$(expr $(nproc) + 1)
 
 .update_dnf_template: &before_script_dnf
  before_script:
   - dnf update -y
   - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel
         make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel
+  - JOBS=$(expr $(nproc) + 1)
 
 build-system1:
  image: ubuntu:19.10
@@ -25,8 +27,8 @@ build-system1:
  - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu
       cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu
       mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
 build-system2:
  image: fedora:latest
@@ -40,8 +42,8 @@ build-system2:
  - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
       microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu
       sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
 build-disabled:
  image: fedora:latest
@@ -56,8 +58,8 @@ build-disabled:
       --disable-qom-cast-debug --disable-spice --disable-vhost-vsock
       --disable-vhost-net --disable-vhost-crypto --disable-vhost-user
       --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user"
- - make -j2
- - make -j2 check-qtest SPEED=slow
+ - make -j"$JOBS"
+ - make -j"$JOBS" check-qtest SPEED=slow
 
 build-tcg-disabled:
  image: centos:8
@@ -67,7 +69,7 @@ build-tcg-disabled:
  - mkdir build
  - cd build
  - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
- - make -j2
+ - make -j"$JOBS"
  - make check-unit
  - make check-qapi-schema
  - cd tests/qemu-iotests/
@@ -86,7 +88,7 @@ build-user:
  - cd build
  - ../configure --enable-werror --disable-system --disable-guest-agent
                --disable-capstone --disable-slirp --disable-fdt
- - make -j2
+ - make -j"$JOBS"
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
@@ -100,8 +102,8 @@ build-clang:
  - ../configure --cc=clang --cxx=clang++ --enable-werror
       --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
                      ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user"
- - make -j2
- - make -j2 check
+ - make -j"$JOBS"
+ - make -j"$JOBS" check
 
 build-tci:
  image: centos:8
@@ -112,7 +114,7 @@ build-tci:
  - cd build
  - ../configure --enable-tcg-interpreter
       --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
- - make -j2
+ - make -j"$JOBS"
  - make run-tcg-tests-x86_64-softmmu
  - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
  - for tg in $TARGETS ; do
-- 
2.18.1

Re: [PULL 0/7] Gitlab CI fixes and improvements

[Peter Maydell]

On Thu, 28 May 2020 at 11:10, Thomas Huth <thuth@redhat.com> wrote:
>
>  Hi Peter,
>
> the following changes since commit 06539ebc76b8625587aa78d646a9d8d5fddf84f3:
>
>   Merge remote-tracking branch 'remotes/philmd-gitlab/tags/mips-hw-next-20200526' into staging (2020-05-26 20:25:06 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/huth/qemu.git tags/pull-request-2020-05-28
>
> for you to fetch changes up to 0016afa250d251a1b3fbc90c6d618e7f00e02f14:
>
>   gitlab-ci: Determine the number of jobs dynamically (2020-05-28 11:01:38 +0200)
>
> ----------------------------------------------------------------
> - Build with other images instead of the broken Debian containers
> - Fix building with the latest version of Clang (at least wrt. to
>   the gitlab-CI pipeline)
> - Add Philippe, Alex and Wainer to the Gitlab-CI section in MAINTAINERS
> ----------------------------------------------------------------


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.1
for any user-visible changes.

-- PMM

gitlab-ci: Do not use the standard container images from gitlab

[Sam Eiderman]

Hi,

I am using debian 10 container to compile qemu too.

I think that what happens here is that

  /usr/include/linux/swab.h

Uses BITS_PER_LONG instead of __BITS_PER_LONG which is actually defined before
in qemu at:

  include/qemu/bitops.h:#define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)

which injects this definition into the linux swab.h header.

By changing BITS_PER_LONG to __BITS_PER_LONG in the linux headers, I managed to
successfully compile qemu.

A different approach would be to move the linux header includes
(#include <linux/cdrom.h>) in file-posix.c above all other includes - which in
some way makes more sense (since we probaly don't want qemu defines to control
linux headers) but it requires a more complex refactoring.

Re: gitlab-ci: Do not use the standard container images from gitlab

[Alex Bennée]

Sam Eiderman <sameid@google.com> writes:

> Hi,
>
> I am using debian 10 container to compile qemu too.
>
> I think that what happens here is that
>
>   /usr/include/linux/swab.h
>
> Uses BITS_PER_LONG instead of __BITS_PER_LONG which is actually defined before
> in qemu at:

That is indeed the error - we are just waiting for Debian to update
linux-libc-dev with the fix to the kernel headers:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960271

>
>   include/qemu/bitops.h:#define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)
>
> which injects this definition into the linux swab.h header.
>
> By changing BITS_PER_LONG to __BITS_PER_LONG in the linux headers, I managed to
> successfully compile qemu.
>
> A different approach would be to move the linux header includes
> (#include <linux/cdrom.h>) in file-posix.c above all other includes - which in
> some way makes more sense (since we probaly don't want qemu defines to control
> linux headers) but it requires a more complex refactoring.


-- 
Alex Bennée

Re: gitlab-ci: Do not use the standard container images from gitlab

[Sam Eiderman]

Thanks for the link

I do believe that the correct approach for me is to rename
BITS_PER_LONG to __BITS_PER_LONG (I just added a sed command in my
Dockerfile) and move on with my particular usage, however I am just
wondering whether dropping debian10/ubuntu20 in the official qemu ci/
pipeline until it's fixed is the correct approach instead of keep
failing it until the error resolves, in a way we want to always know
on which OSs the compilation fails for visibility, no?

Thanks again!


On Sat, Jun 6, 2020 at 2:49 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Sam Eiderman <sameid@google.com> writes:
>
> > Hi,
> >
> > I am using debian 10 container to compile qemu too.
> >
> > I think that what happens here is that
> >
> >   /usr/include/linux/swab.h
> >
> > Uses BITS_PER_LONG instead of __BITS_PER_LONG which is actually defined before
> > in qemu at:
>
> That is indeed the error - we are just waiting for Debian to update
> linux-libc-dev with the fix to the kernel headers:
>
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960271
>
> >
> >   include/qemu/bitops.h:#define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)
> >
> > which injects this definition into the linux swab.h header.
> >
> > By changing BITS_PER_LONG to __BITS_PER_LONG in the linux headers, I managed to
> > successfully compile qemu.
> >
> > A different approach would be to move the linux header includes
> > (#include <linux/cdrom.h>) in file-posix.c above all other includes - which in
> > some way makes more sense (since we probaly don't want qemu defines to control
> > linux headers) but it requires a more complex refactoring.
>
>
> --
> Alex Bennée

Re: gitlab-ci: Do not use the standard container images from gitlab

[Thomas Huth]

On 06/06/2020 14.38, Sam Eiderman wrote:
> Thanks for the link
> 
> I do believe that the correct approach for me is to rename
> BITS_PER_LONG to __BITS_PER_LONG (I just added a sed command in my
> Dockerfile) and move on with my particular usage, however I am just
> wondering whether dropping debian10/ubuntu20 in the official qemu ci/
> pipeline until it's fixed is the correct approach instead of keep
> failing it until the error resolves, in a way we want to always know
> on which OSs the compilation fails for visibility, no?

 Hi,

that bug was only one reason to move the pipelines to another OS. The
other reason is that we are already extensively testing various Ubuntu
(and thus Debian-based) versions in the Travis CI - but did not test any
RPM-based distros in the CI yet. Since Travis is bound to Ubuntu, we can
not test Fedora/CentOS there, thus the Gitlab CI pipelines have now been
moved to RPM-based distros (except for the "build-user" pipeline which
is still using Debian, and the "build-system1" which is now using Ubuntu
19.04 instead, so I think we still have a good mix there).

Note that the problem with Ubuntu 20.04 is also something completely
different: It hangs in an interactive prompt during update and waits for
user input, so that the pipelines finally times out:

 https://gitlab.com/huth/qemu/-/jobs/584573287#L800

If you know a work-around for that, we can move the build-system1
pipeline from 19.04 to 20.04 ... or if Debian gets finally fixed, we can
also move that pipeline back to Debian. I'm fine either way, as long as
the pipelines do not fail due to non-QEMU bugs in the distros.

 Thomas

Re: gitlab-ci: Do not use the standard container images from gitlab

[Sam Eiderman]

I see, thanks for the clarification.

However sometimes builds usually do tend to work on Ubuntu but fail to
work on Debian since it's not always a 1-1 (as in this case) - so you
might want to consider to keep testing Debian together with Ubuntu.

Regarding the Ubuntu 20 problem - have you tried "export
DEBIAN_FRONTEND=noninteractive"? didn't see it in the logs

Sam


On Sun, Jun 7, 2020 at 8:39 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 06/06/2020 14.38, Sam Eiderman wrote:
> > Thanks for the link
> >
> > I do believe that the correct approach for me is to rename
> > BITS_PER_LONG to __BITS_PER_LONG (I just added a sed command in my
> > Dockerfile) and move on with my particular usage, however I am just
> > wondering whether dropping debian10/ubuntu20 in the official qemu ci/
> > pipeline until it's fixed is the correct approach instead of keep
> > failing it until the error resolves, in a way we want to always know
> > on which OSs the compilation fails for visibility, no?
>
>  Hi,
>
> that bug was only one reason to move the pipelines to another OS. The
> other reason is that we are already extensively testing various Ubuntu
> (and thus Debian-based) versions in the Travis CI - but did not test any
> RPM-based distros in the CI yet. Since Travis is bound to Ubuntu, we can
> not test Fedora/CentOS there, thus the Gitlab CI pipelines have now been
> moved to RPM-based distros (except for the "build-user" pipeline which
> is still using Debian, and the "build-system1" which is now using Ubuntu
> 19.04 instead, so I think we still have a good mix there).
>
> Note that the problem with Ubuntu 20.04 is also something completely
> different: It hangs in an interactive prompt during update and waits for
> user input, so that the pipelines finally times out:
>
>  https://gitlab.com/huth/qemu/-/jobs/584573287#L800
>
> If you know a work-around for that, we can move the build-system1
> pipeline from 19.04 to 20.04 ... or if Debian gets finally fixed, we can
> also move that pipeline back to Debian. I'm fine either way, as long as
> the pipelines do not fail due to non-QEMU bugs in the distros.
>
>  Thomas
>

Re: [PULL 5/7] gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder

[Philippe Mathieu-Daudé]

Hi Thomas, Alex,

+Daniel

On 5/28/20 12:10 PM, Thomas Huth wrote:
> We have a dedicated folder for the gitlab-ci - so there is no need
> to clutter the top directory with these .yml files.
> 
> Message-Id: <20200525131823.715-5-thuth@redhat.com>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml       | 0
>   .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0
>   .gitlab-ci.yml                                     | 4 ++--
>   MAINTAINERS                                        | 2 +-
>   4 files changed, 3 insertions(+), 3 deletions(-)
>   rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
>   rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
> 
> diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci.d/edk2.yml
> similarity index 100%
> rename from .gitlab-ci-edk2.yml
> rename to .gitlab-ci.d/edk2.yml
> diff --git a/.gitlab-ci-opensbi.yml b/.gitlab-ci.d/opensbi.yml
> similarity index 100%
> rename from .gitlab-ci-opensbi.yml
> rename to .gitlab-ci.d/opensbi.yml
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index bc6aee6aba..5208d93ff8 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,6 +1,6 @@
>   include:
> -  - local: '/.gitlab-ci-edk2.yml'
> -  - local: '/.gitlab-ci-opensbi.yml'
> +  - local: '/.gitlab-ci.d/edk2.yml'
> +  - local: '/.gitlab-ci.d/opensbi.yml'

It seems these jobs are now always run, the "rules:changes"
to restrict them is ignored.

I searched a bit and found a plausible explanation here:
https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges

"Caution: In pipelines with sources other than the three above changes 
can’t determine if a given file is new or old and always returns true. 
This includes pipelines triggered by pushing new tags. Configuring jobs 
to use only: changes with other only: refs keywords is possible, but not 
recommended."

I don't understand what they mean by "three", is a directory considered
a three? Then that would explain the change.

Regards,

Phil.

Re: [PULL 5/7] gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder

[Daniel P. Berrangé]

On Mon, Oct 12, 2020 at 03:44:00PM +0200, Philippe Mathieu-Daudé wrote:
> Hi Thomas, Alex,
> 
> +Daniel
> 
> On 5/28/20 12:10 PM, Thomas Huth wrote:
> > We have a dedicated folder for the gitlab-ci - so there is no need
> > to clutter the top directory with these .yml files.
> > 
> > Message-Id: <20200525131823.715-5-thuth@redhat.com>
> > Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> > Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > ---
> >   .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml       | 0
> >   .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0
> >   .gitlab-ci.yml                                     | 4 ++--
> >   MAINTAINERS                                        | 2 +-
> >   4 files changed, 3 insertions(+), 3 deletions(-)
> >   rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
> >   rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
> > 
> > diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci.d/edk2.yml
> > similarity index 100%
> > rename from .gitlab-ci-edk2.yml
> > rename to .gitlab-ci.d/edk2.yml
> > diff --git a/.gitlab-ci-opensbi.yml b/.gitlab-ci.d/opensbi.yml
> > similarity index 100%
> > rename from .gitlab-ci-opensbi.yml
> > rename to .gitlab-ci.d/opensbi.yml
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index bc6aee6aba..5208d93ff8 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -1,6 +1,6 @@
> >   include:
> > -  - local: '/.gitlab-ci-edk2.yml'
> > -  - local: '/.gitlab-ci-opensbi.yml'
> > +  - local: '/.gitlab-ci.d/edk2.yml'
> > +  - local: '/.gitlab-ci.d/opensbi.yml'
> 
> It seems these jobs are now always run, the "rules:changes"
> to restrict them is ignored.
> 
> I searched a bit and found a plausible explanation here:
> https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
> 
> "Caution: In pipelines with sources other than the three above changes can’t
> determine if a given file is new or old and always returns true. This
> includes pipelines triggered by pushing new tags. Configuring jobs to use
> only: changes with other only: refs keywords is possible, but not
> recommended."
> 
> I don't understand what they mean by "three", is a directory considered
> a three? Then that would explain the change.

The edk2.yml file as a "when: always" clause attached to every rule,
so surely this negates the point of filtering by making it always
run ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PULL 5/7] gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder

[Philippe Mathieu-Daudé]

On 10/12/20 4:01 PM, Daniel P. Berrangé wrote:
> On Mon, Oct 12, 2020 at 03:44:00PM +0200, Philippe Mathieu-Daudé wrote:
>> Hi Thomas, Alex,
>>
>> +Daniel
>>
>> On 5/28/20 12:10 PM, Thomas Huth wrote:
>>> We have a dedicated folder for the gitlab-ci - so there is no need
>>> to clutter the top directory with these .yml files.
>>>
>>> Message-Id: <20200525131823.715-5-thuth@redhat.com>
>>> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>    .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml       | 0
>>>    .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0
>>>    .gitlab-ci.yml                                     | 4 ++--
>>>    MAINTAINERS                                        | 2 +-
>>>    4 files changed, 3 insertions(+), 3 deletions(-)
>>>    rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
>>>    rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
>>>
>>> diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci.d/edk2.yml
>>> similarity index 100%
>>> rename from .gitlab-ci-edk2.yml
>>> rename to .gitlab-ci.d/edk2.yml
>>> diff --git a/.gitlab-ci-opensbi.yml b/.gitlab-ci.d/opensbi.yml
>>> similarity index 100%
>>> rename from .gitlab-ci-opensbi.yml
>>> rename to .gitlab-ci.d/opensbi.yml
>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>> index bc6aee6aba..5208d93ff8 100644
>>> --- a/.gitlab-ci.yml
>>> +++ b/.gitlab-ci.yml
>>> @@ -1,6 +1,6 @@
>>>    include:
>>> -  - local: '/.gitlab-ci-edk2.yml'
>>> -  - local: '/.gitlab-ci-opensbi.yml'
>>> +  - local: '/.gitlab-ci.d/edk2.yml'
>>> +  - local: '/.gitlab-ci.d/opensbi.yml'
>>
>> It seems these jobs are now always run, the "rules:changes"
>> to restrict them is ignored.
>>
>> I searched a bit and found a plausible explanation here:
>> https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
>>
>> "Caution: In pipelines with sources other than the three above changes can’t
>> determine if a given file is new or old and always returns true. This
>> includes pipelines triggered by pushing new tags. Configuring jobs to use
>> only: changes with other only: refs keywords is possible, but not
>> recommended."
>>
>> I don't understand what they mean by "three", is a directory considered
>> a three? Then that would explain the change.
> 
> The edk2.yml file as a "when: always" clause attached to every rule,
> so surely this negates the point of filtering by making it always
> run ?

I understand it runs regardless a job in any previous stage failed.

 From https://docs.gitlab.com/ee/ci/yaml/#when

* always - execute job regardless of the status of jobs from prior stages.
* on_success - execute job only when all jobs from prior stages succeed 
(or are considered succeeding because they have allow_failure: true). 
This is the default.

I think I used that because at the time I sent that patch
some docker image builds were failing. But we don't really
need it, I'll remove.

Anyhow this used to work with "when: always", see:
https://gitlab.com/qemu-project/qemu/-/pipelines/168158357

> 
> Regards,
> Daniel
> 

Re: [PULL 5/7] gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder

[Philippe Mathieu-Daudé]

On 10/13/20 4:18 PM, Philippe Mathieu-Daudé wrote:
> On 10/12/20 4:01 PM, Daniel P. Berrangé wrote:
>> On Mon, Oct 12, 2020 at 03:44:00PM +0200, Philippe Mathieu-Daudé wrote:
>>> Hi Thomas, Alex,
>>>
>>> +Daniel
>>>
>>> On 5/28/20 12:10 PM, Thomas Huth wrote:
>>>> We have a dedicated folder for the gitlab-ci - so there is no need
>>>> to clutter the top directory with these .yml files.
>>>>
>>>> Message-Id: <20200525131823.715-5-thuth@redhat.com>
>>>> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
>>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>>> ---
>>>>    .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml       | 0
>>>>    .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml | 0
>>>>    .gitlab-ci.yml                                     | 4 ++--
>>>>    MAINTAINERS                                        | 2 +-
>>>>    4 files changed, 3 insertions(+), 3 deletions(-)
>>>>    rename .gitlab-ci-edk2.yml => .gitlab-ci.d/edk2.yml (100%)
>>>>    rename .gitlab-ci-opensbi.yml => .gitlab-ci.d/opensbi.yml (100%)
>>>>
>>>> diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci.d/edk2.yml
>>>> similarity index 100%
>>>> rename from .gitlab-ci-edk2.yml
>>>> rename to .gitlab-ci.d/edk2.yml
>>>> diff --git a/.gitlab-ci-opensbi.yml b/.gitlab-ci.d/opensbi.yml
>>>> similarity index 100%
>>>> rename from .gitlab-ci-opensbi.yml
>>>> rename to .gitlab-ci.d/opensbi.yml
>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>> index bc6aee6aba..5208d93ff8 100644
>>>> --- a/.gitlab-ci.yml
>>>> +++ b/.gitlab-ci.yml
>>>> @@ -1,6 +1,6 @@
>>>>    include:
>>>> -  - local: '/.gitlab-ci-edk2.yml'
>>>> -  - local: '/.gitlab-ci-opensbi.yml'
>>>> +  - local: '/.gitlab-ci.d/edk2.yml'
>>>> +  - local: '/.gitlab-ci.d/opensbi.yml'
>>>
>>> It seems these jobs are now always run, the "rules:changes"
>>> to restrict them is ignored.
>>>
>>> I searched a bit and found a plausible explanation here:
>>> https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
>>>
>>> "Caution: In pipelines with sources other than the three above
>>> changes can’t
>>> determine if a given file is new or old and always returns true. This
>>> includes pipelines triggered by pushing new tags. Configuring jobs to
>>> use
>>> only: changes with other only: refs keywords is possible, but not
>>> recommended."
>>>
>>> I don't understand what they mean by "three", is a directory considered
>>> a three? Then that would explain the change.
>>
>> The edk2.yml file as a "when: always" clause attached to every rule,
>> so surely this negates the point of filtering by making it always
>> run ?

git show 71920809ceab
+docker-edk2:
+ stage: build
+ rules: # Only run this job when the Dockerfile is modified
+ - changes:
+   - .gitlab-ci-edk2.yml
+   - .gitlab-ci.d/edk2/Dockerfile
+   when: always

Per https://docs.gitlab.com/ee/ci/yaml/#rules-clauses

  Rules are evaluated in order until a match is found.

  The job is not added to the pipeline:

    If no rules match, and there is no standalone
    when: on_success, when: delayed or when: always.

For docker-edk2 the rule is "If .gitlab-ci-edk2.yml or
.gitlab-ci.d/edk2/Dockerfile changed, then always add;
else do not add".

> I understand it runs regardless a job in any previous stage failed.
> 
> From https://docs.gitlab.com/ee/ci/yaml/#when
> 
> * always - execute job regardless of the status of jobs from prior stages.
> * on_success - execute job only when all jobs from prior stages succeed
> (or are considered succeeding because they have allow_failure: true).
> This is the default.
> 
> I think I used that because at the time I sent that patch
> some docker image builds were failing. But we don't really
> need it, I'll remove.

Also because this is a long job burning free tier credits pointlessly.

(I'll also do it for the OpenSBI job).

> 
> Anyhow this used to work with "when: always", see:
> https://gitlab.com/qemu-project/qemu/-/pipelines/168158357
> 
>>
>> Regards,
>> Daniel
>>
>