[XEN PATCH 0/2] automation: Fix build with new qemu-xen version

[XEN PATCH 0/2] automation: Fix build with new qemu-xen version

[Anthony PERARD]

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.automation-add-ninja-v1

Adding ninja-build pkg when possible.

I'll push new containers soon.

Anthony PERARD (2):
  automation: Adding ninja-build to some docker images
  automation: Check if ninja is available before building QEMU

 automation/build/alpine/3.12.dockerfile        | 1 +
 automation/build/fedora/29.dockerfile          | 9 ++++++---
 automation/build/suse/opensuse-leap.dockerfile | 2 ++
 automation/build/ubuntu/bionic.dockerfile      | 2 ++
 automation/build/ubuntu/focal.dockerfile       | 2 ++
 automation/scripts/build                       | 4 +++-
 6 files changed, 16 insertions(+), 4 deletions(-)

-- 
Anthony PERARD

[XEN PATCH 1/2] automation: Adding ninja-build to some docker images

[Anthony PERARD]

This is to allow building the latest version of QEMU.

fedora/29:
    In addition to adding "ninja", I've add to make some other
    changes: some `go build` failed with `mkdir /.cache` no
    permission, so I've created a user.
    (this was discovered while testing the new container with the
    script containerize.)

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 automation/build/alpine/3.12.dockerfile        | 1 +
 automation/build/fedora/29.dockerfile          | 9 ++++++---
 automation/build/suse/opensuse-leap.dockerfile | 2 ++
 automation/build/ubuntu/bionic.dockerfile      | 2 ++
 automation/build/ubuntu/focal.dockerfile       | 2 ++
 5 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/automation/build/alpine/3.12.dockerfile b/automation/build/alpine/3.12.dockerfile
index bb8e5ece7414..4ee3ddc12e52 100644
--- a/automation/build/alpine/3.12.dockerfile
+++ b/automation/build/alpine/3.12.dockerfile
@@ -47,6 +47,7 @@ RUN \
   apk add glib-dev && \
   apk add libattr && \
   apk add libcap-ng-dev && \
+  apk add ninja && \
   apk add pixman-dev && \
   \
   # cleanup
diff --git a/automation/build/fedora/29.dockerfile b/automation/build/fedora/29.dockerfile
index 027b93ceaf4e..60a5d722668f 100644
--- a/automation/build/fedora/29.dockerfile
+++ b/automation/build/fedora/29.dockerfile
@@ -2,9 +2,6 @@ FROM fedora:29
 LABEL maintainer.name="The Xen Project" \
       maintainer.email="xen-devel@lists.xenproject.org"
 
-RUN mkdir /build
-WORKDIR /build
-
 # install Xen depends
 RUN dnf -y install \
         clang \
@@ -43,5 +40,11 @@ RUN dnf -y install \
         ocaml \
         ocaml-findlib \
         golang \
+        # QEMU
+        ninja-build \
     && dnf clean all && \
     rm -rf /var/cache/dnf
+
+RUN useradd --create-home user
+USER user
+WORKDIR /build
diff --git a/automation/build/suse/opensuse-leap.dockerfile b/automation/build/suse/opensuse-leap.dockerfile
index 573fbd8ae47e..03f8a7acad4f 100644
--- a/automation/build/suse/opensuse-leap.dockerfile
+++ b/automation/build/suse/opensuse-leap.dockerfile
@@ -67,5 +67,7 @@ RUN zypper install -y --no-recommends \
         which \
         xz-devel \
         zlib-devel \
+        # QEMU
+        ninja \
         && \
         zypper clean -a
diff --git a/automation/build/ubuntu/bionic.dockerfile b/automation/build/ubuntu/bionic.dockerfile
index 406a97494caf..1e5a27c70f15 100644
--- a/automation/build/ubuntu/bionic.dockerfile
+++ b/automation/build/ubuntu/bionic.dockerfile
@@ -45,6 +45,8 @@ RUN apt-get update && \
         wget \
         git \
         nasm \
+        # QEMU
+        ninja-build \
         && \
         apt-get autoremove -y && \
         apt-get clean && \
diff --git a/automation/build/ubuntu/focal.dockerfile b/automation/build/ubuntu/focal.dockerfile
index 952a3172aa64..74655b9876d2 100644
--- a/automation/build/ubuntu/focal.dockerfile
+++ b/automation/build/ubuntu/focal.dockerfile
@@ -44,6 +44,8 @@ RUN apt-get update && \
         wget \
         git \
         nasm \
+        # QEMU
+        ninja-build \
         && \
         apt-get autoremove -y && \
         apt-get clean && \
-- 
Anthony PERARD

[XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Anthony PERARD]

ninja is now required to build the latest version of QEMU, some
container still don't have ninja and attempting to add it breaks the
build for different reasons, so QEMU will be skip on those containers.

Failures:
- ubuntu/xenial:
    fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
- debian/unstable-i386: (debian-unstable-32)
    /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'.  Stop.
- suse/opensuse-tumbleweed:
    failed to build iPXE
- debian/unstable:
    update of the container fails with:
        The following packages have unmet dependencies:
         clang-8 : Depends: libstdc++-8-dev but it is not installable
                   Depends: libgcc-8-dev but it is not installable
                   Depends: libobjc-8-dev but it is not installable
                   Recommends: llvm-8-dev but it is not going to be installed
                   Recommends: libomp-8-dev but it is not going to be installed

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 automation/scripts/build | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/automation/scripts/build b/automation/scripts/build
index 46b6903d2922..d8c77e244cda 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -46,7 +46,9 @@ if ! test -z "$(ldd /bin/ls|grep musl|head -1)"; then
 fi
 
 # Qemu requires Python 3.5 or later
-if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))"; then
+# QEMU requires ninja
+if ! type python3 || python3 -c "import sys; res = sys.version_info < (3, 5); exit(not(res))" \
+        || ! type ninja; then
     cfgargs+=("--with-system-qemu=/bin/false")
 fi
 
-- 
Anthony PERARD

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Andrew Cooper]

On 07/07/2021 17:40, Anthony PERARD wrote:
> ninja is now required to build the latest version of QEMU, some
> container still don't have ninja and attempting to add it breaks the
> build for different reasons, so QEMU will be skip on those containers.
>
> Failures:
> - ubuntu/xenial:
>     fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
> - debian/unstable-i386: (debian-unstable-32)
>     /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'.  Stop.
> - suse/opensuse-tumbleweed:
>     failed to build iPXE
> - debian/unstable:
>     update of the container fails with:
>         The following packages have unmet dependencies:
>          clang-8 : Depends: libstdc++-8-dev but it is not installable
>                    Depends: libgcc-8-dev but it is not installable
>                    Depends: libobjc-8-dev but it is not installable
>                    Recommends: llvm-8-dev but it is not going to be installed
>                    Recommends: libomp-8-dev but it is not going to be installed

Only the first failure seems to be related to ninja.

The second is a known bug which still needs addressing.  Our build
system's logic to cope with `checkpolicy` being absent doesn't work.

Tumbleweed is generally broken and fails at ./configure due to missing
compression libraries.  It is quite possible that the iPXE failure is
related.

The final one is probably because unstable has dropped clang-8.  The
container hasn't been rebuilt in a while because we still haven't got
auto-rebuild sorted.

~Andrew

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 197 bytes --]

Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> iPXE failure

it just needs to be updated to ipxe.git#master to make it compatible with gcc11.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Andrew Cooper]

On 07/07/2021 20:52, Olaf Hering wrote:
> Am Wed, 7 Jul 2021 18:46:03 +0100
> schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
>
>> iPXE failure
> it just needs to be updated to ipxe.git#master to make it compatible with gcc11.

Ah great.  I presume that is your "[PATCH v2] tools: ipxe: update for
fixing build with GCC11" patch which has been sitting neglected on the list?

~Andrew

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

Am Wed, 7 Jul 2021 23:44:33 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> On 07/07/2021 20:52, Olaf Hering wrote:
> > Am Wed, 7 Jul 2021 18:46:03 +0100
> > schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
> >> iPXE failure  
> > it just needs to be updated to ipxe.git#master to make it compatible with gcc11.  
> Ah great.  I presume that is your "[PATCH v2] tools: ipxe: update for
> fixing build with GCC11" patch which has been sitting neglected on the list?

Yes, but the build logs indicate gcc10 is used in this container. So it might be something else.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 468 bytes --]

Am Wed, 7 Jul 2021 18:46:03 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> Tumbleweed is generally broken and fails at ./configure due to missing
> compression libraries.

Something requests zlib-devel to be installed.

I suggest to provide all config.logs, not just the one from the top directory.

Also a "test -f /etc/os-release && cat /etc/os-release" should be added after "$CC --version", just to know how stale the container is.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Andrew Cooper]

On 07/07/2021 21:35, Olaf Hering wrote:
> Am Wed, 7 Jul 2021 18:46:03 +0100
> schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
>
>> Tumbleweed is generally broken and fails at ./configure due to missing
>> compression libraries.
> Something requests zlib-devel to be installed.

Yeah - it shouldn't be hard to fix.

> I suggest to provide all config.logs, not just the one from the top directory.
>
> Also a "test -f /etc/os-release && cat /etc/os-release" should be added after "$CC --version", just to know how stale the container is.

We ought to yes, although I think at the moment we probably don't want
to realise how stale all the containers are.

~Andrew

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

Am Wed, 7 Jul 2021 23:47:02 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> We ought to yes, although I think at the moment we probably don't want
> to realise how stale all the containers are.

At least for Tumbleweed it is useful information.

opensuse-tumbleweed.dockerfile requests zlib-devel already.

tools/config.log is required to see what the failure actually is.
The toplevel config.log is not helpful in this particular case.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 354 bytes --]

Am Thu, 8 Jul 2021 02:32:24 +0200
schrieb Olaf Hering <olaf@aepfle.de>:

> At least for Tumbleweed it is useful information.

CPE_NAME="cpe:/o:opensuse:tumbleweed:20190730"

Two years old for a rolling release.

Is there any documentation how to refresh the used containers? I think for Tumbleweed this should be done every other week.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Andrew Cooper]

On 08/07/2021 13:23, Olaf Hering wrote:
> Am Thu, 8 Jul 2021 02:32:24 +0200
> schrieb Olaf Hering <olaf@aepfle.de>:
>
>> At least for Tumbleweed it is useful information.
> CPE_NAME="cpe:/o:opensuse:tumbleweed:20190730"
>
> Two years old for a rolling release.
>
> Is there any documentation how to refresh the used containers? I think for Tumbleweed this should be done every other week.

Automatically rebuilding the containers is still a task on the CI
working group's todo list.

That said, the tumbleweed containers were rebuilt far more recently than
that, so something is wonky...


In a xen tree, see automation/build/Makefile for details, but tl;dr to
rebuild:

$ make -C automation/build/ tumbleweed

and to push:

$ docker login registry.gitlab.com/xen-project/xen
$ make -C automation/build/ tumbleweed PUSH=1

Bear in mind that the final line will update the live CI system.  What I
normally do to test new containers is to give it a test suffix, push to
the live system, and push a branch to xen.git with the container names
edited in automation/gitlab-ci/*.yaml.

~Andrew

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

Am Thu, 8 Jul 2021 13:51:49 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> That said, the tumbleweed containers were rebuilt far more recently than
> that, so something is wonky...

After playing with 'automation/scripts/containerize', docker image ls shows:
registry.gitlab.com/xen-project/xen/suse   opensuse-tumbleweed   f9b84e2346f0   7 weeks ago         2.27GB

No idea how that created a container with broken content.
zlib-devel.rpm exists, but libz.so is a broken symlink.

I will update the README to list the required steps for updating.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Anthony PERARD]

On Wed, Jul 07, 2021 at 06:46:03PM +0100, Andrew Cooper wrote:
> On 07/07/2021 17:40, Anthony PERARD wrote:
> > ninja is now required to build the latest version of QEMU, some
> > container still don't have ninja and attempting to add it breaks the
> > build for different reasons, so QEMU will be skip on those containers.
> >
> > Failures:
> > - ubuntu/xenial:
> >     fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
> > - debian/unstable-i386: (debian-unstable-32)
> >     /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'.  Stop.
> > - suse/opensuse-tumbleweed:
> >     failed to build iPXE
> > - debian/unstable:
> >     update of the container fails with:
> >         The following packages have unmet dependencies:
> >          clang-8 : Depends: libstdc++-8-dev but it is not installable
> >                    Depends: libgcc-8-dev but it is not installable
> >                    Depends: libobjc-8-dev but it is not installable
> >                    Recommends: llvm-8-dev but it is not going to be installed
> >                    Recommends: libomp-8-dev but it is not going to be installed
> 
> Only the first failure seems to be related to ninja.
> 
> The second is a known bug which still needs addressing.  Our build
> system's logic to cope with `checkpolicy` being absent doesn't work.
> 
> Tumbleweed is generally broken and fails at ./configure due to missing
> compression libraries.  It is quite possible that the iPXE failure is
> related.
> 
> The final one is probably because unstable has dropped clang-8.  The
> container hasn't been rebuilt in a while because we still haven't got
> auto-rebuild sorted.


So those containers are ones that I've noticed needed ninja (as found
out by looking at a single pipeline), but one which trying to install
ninja and thus rebuilding the container mean that the build was broken.
I just didn't want to spend time trying to fix issues that should have
been found out by a regular rebuild of the containers. So I've just
listed the reason there and moved on.

-- 
Anthony PERARD

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Andrew Cooper]

On 08/07/2021 11:52, Anthony PERARD wrote:
> On Wed, Jul 07, 2021 at 06:46:03PM +0100, Andrew Cooper wrote:
>> On 07/07/2021 17:40, Anthony PERARD wrote:
>>> ninja is now required to build the latest version of QEMU, some
>>> container still don't have ninja and attempting to add it breaks the
>>> build for different reasons, so QEMU will be skip on those containers.
>>>
>>> Failures:
>>> - ubuntu/xenial:
>>>     fatal: ninja version (1.5.1) incompatible with build file ninja_required_version version (1.7.1).
>>> - debian/unstable-i386: (debian-unstable-32)
>>>     /build/tools/flask/policy/Makefile.common:115: *** target pattern contains no '%'.  Stop.
>>> - suse/opensuse-tumbleweed:
>>>     failed to build iPXE
>>> - debian/unstable:
>>>     update of the container fails with:
>>>         The following packages have unmet dependencies:
>>>          clang-8 : Depends: libstdc++-8-dev but it is not installable
>>>                    Depends: libgcc-8-dev but it is not installable
>>>                    Depends: libobjc-8-dev but it is not installable
>>>                    Recommends: llvm-8-dev but it is not going to be installed
>>>                    Recommends: libomp-8-dev but it is not going to be installed
>> Only the first failure seems to be related to ninja.
>>
>> The second is a known bug which still needs addressing.  Our build
>> system's logic to cope with `checkpolicy` being absent doesn't work.
>>
>> Tumbleweed is generally broken and fails at ./configure due to missing
>> compression libraries.  It is quite possible that the iPXE failure is
>> related.
>>
>> The final one is probably because unstable has dropped clang-8.  The
>> container hasn't been rebuilt in a while because we still haven't got
>> auto-rebuild sorted.
>
> So those containers are ones that I've noticed needed ninja (as found
> out by looking at a single pipeline), but one which trying to install
> ninja and thus rebuilding the container mean that the build was broken.
> I just didn't want to spend time trying to fix issues that should have
> been found out by a regular rebuild of the containers.

That's fine, but as these containers weren't rebuilt, the details of any
knock-on error aren't relevant here.

I'd go with simply:

"ninja is now required to build the latest version of QEMU, and not all
distros have a suitable version.  Skip the QEMU build when ninja is not
available".

Can fix on commit if you're happy with this.

~Andrew

Re: [XEN PATCH 2/2] automation: Check if ninja is available before building QEMU

[Anthony PERARD]

On Thu, Jul 08, 2021 at 12:23:38PM +0100, Andrew Cooper wrote:
> I'd go with simply:
> 
> "ninja is now required to build the latest version of QEMU, and not all
> distros have a suitable version.  Skip the QEMU build when ninja is not
> available".
> 
> Can fix on commit if you're happy with this.

Sounds good, thanks.

-- 
Anthony PERARD

Re: [XEN PATCH 0/2] automation: Fix build with new qemu-xen version

[Anthony PERARD]

On Wed, Jul 07, 2021 at 05:39:59PM +0100, Anthony PERARD wrote:
> Adding ninja-build pkg when possible.
> 
> I'll push new containers soon.

I've pushed:
    registry.gitlab.com/xen-project/xen/fedora:29
    registry.gitlab.com/xen-project/xen/ubuntu:bionic
    registry.gitlab.com/xen-project/xen/ubuntu:focal
    registry.gitlab.com/xen-project/xen/alpine:3.12
    registry.gitlab.com/xen-project/xen/suse:opensuse-leap

> 
> Anthony PERARD (2):
>   automation: Adding ninja-build to some docker images
>   automation: Check if ninja is available before building QEMU
> 
>  automation/build/alpine/3.12.dockerfile        | 1 +
>  automation/build/fedora/29.dockerfile          | 9 ++++++---
>  automation/build/suse/opensuse-leap.dockerfile | 2 ++
>  automation/build/ubuntu/bionic.dockerfile      | 2 ++
>  automation/build/ubuntu/focal.dockerfile       | 2 ++
>  automation/scripts/build                       | 4 +++-
>  6 files changed, 16 insertions(+), 4 deletions(-)

-- 
Anthony PERARD