[PATCH v2 0/3] gitlab-ci: Add jobs to build the EDK2 firmware binaries

[PATCH v2 0/3] gitlab-ci: Add jobs to build the EDK2 firmware binaries

[Philippe Mathieu-Daudé]

We provide the EDK2 firmware binaries in pc-bios/. When we
update the roms/edk2/ submodule, we need to rebuild the
firmware binaries.
To avoid the burden on a single developer, this series add
a GitLab job to generate reproducible EDK2 firmware binaries.

The job is only on specific conditions:
- roms/edk2/ is updated
- or the branch or tag start with 'edk2'
- or 'edk2' appears in last commit description

We won't use ccache to build the EDK2 firmwares, but the patch
was ready/tested so I'm including it.

The proposed procedure to update the EDK2 submodule is:

- add a commit to update roms/edk2/ submodule
- push to gitlab
- download generated artifacts (only available if job succeed)
- unzip the firmware binaries from the artifacts archive
- test the binaries
- add a commit with the binaries, referencing the ci job url.

Example of pipeline that built the edk2-stable201905 firmwares:
https://gitlab.com/philmd/qemu/pipelines/107553178
(the job is https://gitlab.com/philmd/qemu/-/jobs/395823729)

The first patch is already reviewed, but is a prerequisite to
use the Ubuntu docker image to build, so I included it.

Since v1 [*]
- split in 2 jobs: build docker image, build firmwares
  (this way we build the docker image once for all the
   subsequent jobs)
- addressed Laszlo comment (minor the 'git-submodule-update)

Regards,

Phil.

[*] https://www.mail-archive.com/qemu-devel@nongnu.org/msg668432.html

Philippe Mathieu-Daudé (3):
  roms/edk2-funcs: Force softfloat ARM toolchain prefix on Debian
  gitlab-ci.yml: Add jobs to build EDK2 firmware binaries
  gitlab-ci-edk2.yml: Use ccache

 .gitlab-ci-edk2.yml          | 58 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.d/edk2/Dockerfile | 30 +++++++++++++++++++
 .gitlab-ci.yml               |  3 ++
 MAINTAINERS                  |  4 ++-
 roms/edk2-funcs.sh           |  3 ++
 5 files changed, 97 insertions(+), 1 deletion(-)
 create mode 100644 .gitlab-ci-edk2.yml
 create mode 100644 .gitlab-ci.d/edk2/Dockerfile

-- 
2.21.1

[PATCH v2 1/3] roms/edk2-funcs: Force softfloat ARM toolchain prefix on Debian

[Philippe Mathieu-Daudé]

The Debian (based) distributions currently provides 2 ARM
toolchains, documented as [1]:

* The ARM EABI (armel) port targets a range of older 32-bit ARM
  devices, particularly those used in NAS hardware and a variety
  of *plug computers.
* The newer ARM hard-float (armhf) port supports newer, more
  powerful 32-bit devices using version 7 of the ARM architecture
  specification.

For various reasons documented in [2], the EDK2 project suggests
to use the softfloat toolchain (named 'armel' by Debian).

Force the softfloat cross toolchain prefix on Debian distributions.

[1] https://www.debian.org/ports/arm/#status
[2] https://github.com/tianocore/edk2/commit/41203b9a

Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
v3: fixed s/float/soft/ typo (Laszlo)
---
 roms/edk2-funcs.sh | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/roms/edk2-funcs.sh b/roms/edk2-funcs.sh
index 3f4485b201..cd6e4f2c82 100644
--- a/roms/edk2-funcs.sh
+++ b/roms/edk2-funcs.sh
@@ -112,6 +112,9 @@ qemu_edk2_get_cross_prefix()
      ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then
     # no cross-compiler needed
     :
+  elif ( [ -e /etc/debian_version ] && [ "$gcc_arch" == arm ] ); then
+    # force soft-float cross-compiler on Debian
+    printf 'arm-linux-gnueabi-'
   else
     printf '%s-linux-gnu-\n' "$gcc_arch"
   fi
-- 
2.21.1

[PATCH v2 2/3] gitlab-ci.yml: Add jobs to build the EDK2 firmware binaries

[Philippe Mathieu-Daudé]

Add two GitLab job to build the EDK2 firmware binaries.

The first job build a Docker image with the packages requisite
to build EDK2, and store this image in the GitLab registry.
The second job pull the image from the registry and build the
EDK2 firmware binaries.

The docker image is only rebuilt if the GitLab YAML or the
Dockerfile is updated.
The second job is only built when the roms/edk2/ submodule is
updated, when a git-ref starts with 'edk2' or when the last
commit contains 'EDK2'. The files generated are archived in
the artifacts.zip file.

With edk2-stable201905, it took 2 minutes 52 seconds to build
the docker image, and 36 minutes 28 seconds to generate the
artifacts.zip with the firmware binaries (filesize: 10MiB).

See: https://gitlab.com/philmd/qemu/pipelines/107553178

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
v2:
- Split job, build docker image first
- addressed Laszlo comments but kept the git-submodule-update call
---
 .gitlab-ci-edk2.yml          | 49 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.d/edk2/Dockerfile | 27 ++++++++++++++++++++
 .gitlab-ci.yml               |  3 +++
 MAINTAINERS                  |  4 ++-
 4 files changed, 82 insertions(+), 1 deletion(-)
 create mode 100644 .gitlab-ci-edk2.yml
 create mode 100644 .gitlab-ci.d/edk2/Dockerfile

diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci-edk2.yml
new file mode 100644
index 0000000000..088ba4b43a
--- /dev/null
+++ b/.gitlab-ci-edk2.yml
@@ -0,0 +1,49 @@
+docker-edk2:
+ stage: build
+ rules: # Only run this job when the Dockerfile is modified
+ - changes:
+   - .gitlab-ci-edk2.yml
+   - .gitlab-ci.d/edk2/Dockerfile
+   when: always
+ image: docker:19.03.1
+ services:
+ - docker:19.03.1-dind
+ variables:
+  GIT_DEPTH: 3
+  IMAGE_TAG: $CI_REGISTRY_IMAGE:edk2-cross-build
+  # We don't use TLS
+  DOCKER_HOST: tcp://docker:2375
+  DOCKER_TLS_CERTDIR: ""
+ before_script:
+ - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
+ script:
+ - docker pull $IMAGE_TAG || true
+ - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
+                                        --tag $IMAGE_TAG .gitlab-ci.d/edk2
+ - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
+ - docker push $IMAGE_TAG
+
+build-edk2:
+ rules: # Only run this job when ...
+ - changes: # ... roms/edk2/ is modified (submodule updated)
+   - roms/edk2/*
+   when: always
+ - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
+   when: always
+ - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
+   when: always
+ artifacts:
+   paths: # 'artifacts.zip' will contains the following files:
+   - pc-bios/edk2*bz2
+   - pc-bios/edk2-licenses.txt
+   - edk2-stdout.log
+   - edk2-stderr.log
+ image: $CI_REGISTRY_IMAGE:edk2-cross-build
+ variables:
+   GIT_DEPTH: 3
+ script: # Clone the required submodules and build EDK2
+ - git submodule update --init roms/edk2
+ - git -C roms/edk2 submodule update --init
+ - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
+ - echo "=== Using ${JOBS} simultaneous jobs ==="
+ - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a edk2-stderr.log >&2
diff --git a/.gitlab-ci.d/edk2/Dockerfile b/.gitlab-ci.d/edk2/Dockerfile
new file mode 100644
index 0000000000..b4584d1cf6
--- /dev/null
+++ b/.gitlab-ci.d/edk2/Dockerfile
@@ -0,0 +1,27 @@
+#
+# Docker image to cross-compile EDK2 firmware binaries
+#
+FROM ubuntu:16.04
+
+MAINTAINER Philippe Mathieu-Daudé <philmd@redhat.com>
+
+# Install packages required to build EDK2
+RUN apt update \
+    && \
+    \
+    DEBIAN_FRONTEND=noninteractive \
+    apt install --assume-yes --no-install-recommends \
+        build-essential \
+        ca-certificates \
+        dos2unix \
+        gcc-aarch64-linux-gnu \
+        gcc-arm-linux-gnueabi \
+        git \
+        iasl \
+        make \
+        nasm \
+        python \
+        uuid-dev \
+    && \
+    \
+    rm -rf /var/lib/apt/lists/*
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ebcef0ebe9..f799246047 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,3 +1,6 @@
+include:
+  - local: '/.gitlab-ci-edk2.yml'
+
 before_script:
  - apt-get update -qq
  - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
diff --git a/MAINTAINERS b/MAINTAINERS
index cd2dc137a3..93620ed406 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2358,6 +2358,8 @@ F: roms/edk2
 F: roms/edk2-*
 F: tests/data/uefi-boot-images/
 F: tests/uefi-test-tools/
+F: .gitlab-ci-edk2.yml
+F: .gitlab-ci.d/edk2/
 
 Usermode Emulation
 ------------------
@@ -2701,7 +2703,7 @@ W: https://cirrus-ci.com/github/qemu/qemu
 GitLab Continuous Integration
 M: Thomas Huth <thuth@redhat.com>
 S: Maintained
-F: .gitlab-ci.yml
+F: .gitlab-ci*.yml
 
 Guest Test Compilation Support
 M: Alex Bennée <alex.bennee@linaro.org>
-- 
2.21.1

[NOTFORMERGE PATCH v2 3/3] gitlab-ci-edk2.yml: Use ccache

[Philippe Mathieu-Daudé]

By using ccache we reduce the job duration from
  36 minutes 28 seconds: https://gitlab.com/philmd/qemu/-/jobs/395823729
to
  29 minutes 15 seconds: https://gitlab.com/philmd/qemu/-/jobs/395840851

  Running after script...
  $ ccache --show-stats
  cache hit (direct)                  6604
  files in cache                     12090
  cache size                         335.5 MB

For now downloading this cache takes 16 seconds, archiving
it 44 seconds.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
We prefer to not use ccache to build the EDK2 firmware binaries, see:

https://www.mail-archive.com/qemu-devel@nongnu.org/msg668616.html

 .gitlab-ci-edk2.yml          | 13 +++++++++++--
 .gitlab-ci.d/edk2/Dockerfile |  3 +++
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci-edk2.yml
index 088ba4b43a..50b8aee6b7 100644
--- a/.gitlab-ci-edk2.yml
+++ b/.gitlab-ci-edk2.yml
@@ -10,7 +10,7 @@ docker-edk2:
  - docker:19.03.1-dind
  variables:
   GIT_DEPTH: 3
-  IMAGE_TAG: $CI_REGISTRY_IMAGE:edk2-cross-build
+  IMAGE_TAG: $CI_REGISTRY_IMAGE:edk2-cross-build-ccache
   # We don't use TLS
   DOCKER_HOST: tcp://docker:2375
   DOCKER_TLS_CERTDIR: ""
@@ -38,12 +38,21 @@ build-edk2:
    - pc-bios/edk2-licenses.txt
    - edk2-stdout.log
    - edk2-stderr.log
- image: $CI_REGISTRY_IMAGE:edk2-cross-build
+ image: $CI_REGISTRY_IMAGE:edk2-cross-build-ccache
  variables:
    GIT_DEPTH: 3
+   CCACHE_DIR: ${CI_PROJECT_DIR}/.ccache
+ cache: # Use the same cache for all EDK2 jobs
+   key: edk2-cross-build-ccache
+   paths:
+   - ${CCACHE_DIR}
+ before_script:
+ - ccache --zero-stats
  script: # Clone the required submodules and build EDK2
  - git submodule update --init roms/edk2
  - git -C roms/edk2 submodule update --init
  - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
  - echo "=== Using ${JOBS} simultaneous jobs ==="
  - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a edk2-stderr.log >&2
+ after_script:
+ - ccache --show-stats
diff --git a/.gitlab-ci.d/edk2/Dockerfile b/.gitlab-ci.d/edk2/Dockerfile
index b4584d1cf6..350b4df240 100644
--- a/.gitlab-ci.d/edk2/Dockerfile
+++ b/.gitlab-ci.d/edk2/Dockerfile
@@ -13,6 +13,7 @@ RUN apt update \
     apt install --assume-yes --no-install-recommends \
         build-essential \
         ca-certificates \
+        ccache \
         dos2unix \
         gcc-aarch64-linux-gnu \
         gcc-arm-linux-gnueabi \
@@ -25,3 +26,5 @@ RUN apt update \
     && \
     \
     rm -rf /var/lib/apt/lists/*
+
+ENV PATH /usr/lib/ccache:$PATH
-- 
2.21.1

Re: [PATCH v2 2/3] gitlab-ci.yml: Add jobs to build the EDK2 firmware binaries

[Laszlo Ersek]

On 01/07/20 16:31, Philippe Mathieu-Daudé wrote:
> Add two GitLab job to build the EDK2 firmware binaries.
> 
> The first job build a Docker image with the packages requisite
> to build EDK2, and store this image in the GitLab registry.
> The second job pull the image from the registry and build the
> EDK2 firmware binaries.
> 
> The docker image is only rebuilt if the GitLab YAML or the
> Dockerfile is updated.
> The second job is only built when the roms/edk2/ submodule is
> updated, when a git-ref starts with 'edk2' or when the last
> commit contains 'EDK2'. The files generated are archived in
> the artifacts.zip file.
> 
> With edk2-stable201905, it took 2 minutes 52 seconds to build
> the docker image, and 36 minutes 28 seconds to generate the
> artifacts.zip with the firmware binaries (filesize: 10MiB).
> 
> See: https://gitlab.com/philmd/qemu/pipelines/107553178
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> v2:
> - Split job, build docker image first
> - addressed Laszlo comments but kept the git-submodule-update call
> ---
>  .gitlab-ci-edk2.yml          | 49 ++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.d/edk2/Dockerfile | 27 ++++++++++++++++++++
>  .gitlab-ci.yml               |  3 +++
>  MAINTAINERS                  |  4 ++-
>  4 files changed, 82 insertions(+), 1 deletion(-)
>  create mode 100644 .gitlab-ci-edk2.yml
>  create mode 100644 .gitlab-ci.d/edk2/Dockerfile
> 
> diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci-edk2.yml
> new file mode 100644
> index 0000000000..088ba4b43a
> --- /dev/null
> +++ b/.gitlab-ci-edk2.yml
> @@ -0,0 +1,49 @@
> +docker-edk2:
> + stage: build
> + rules: # Only run this job when the Dockerfile is modified
> + - changes:
> +   - .gitlab-ci-edk2.yml
> +   - .gitlab-ci.d/edk2/Dockerfile
> +   when: always
> + image: docker:19.03.1
> + services:
> + - docker:19.03.1-dind
> + variables:
> +  GIT_DEPTH: 3
> +  IMAGE_TAG: $CI_REGISTRY_IMAGE:edk2-cross-build
> +  # We don't use TLS
> +  DOCKER_HOST: tcp://docker:2375
> +  DOCKER_TLS_CERTDIR: ""
> + before_script:
> + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
> + script:
> + - docker pull $IMAGE_TAG || true
> + - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
> +                                        --tag $IMAGE_TAG .gitlab-ci.d/edk2
> + - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
> + - docker push $IMAGE_TAG
> +
> +build-edk2:
> + rules: # Only run this job when ...
> + - changes: # ... roms/edk2/ is modified (submodule updated)
> +   - roms/edk2/*
> +   when: always
> + - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
> +   when: always
> + - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
> +   when: always
> + artifacts:
> +   paths: # 'artifacts.zip' will contains the following files:
> +   - pc-bios/edk2*bz2
> +   - pc-bios/edk2-licenses.txt
> +   - edk2-stdout.log
> +   - edk2-stderr.log
> + image: $CI_REGISTRY_IMAGE:edk2-cross-build
> + variables:
> +   GIT_DEPTH: 3
> + script: # Clone the required submodules and build EDK2
> + - git submodule update --init roms/edk2
> + - git -C roms/edk2 submodule update --init
> + - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
> + - echo "=== Using ${JOBS} simultaneous jobs ==="
> + - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a edk2-stderr.log >&2
> diff --git a/.gitlab-ci.d/edk2/Dockerfile b/.gitlab-ci.d/edk2/Dockerfile
> new file mode 100644
> index 0000000000..b4584d1cf6
> --- /dev/null
> +++ b/.gitlab-ci.d/edk2/Dockerfile
> @@ -0,0 +1,27 @@
> +#
> +# Docker image to cross-compile EDK2 firmware binaries
> +#
> +FROM ubuntu:16.04
> +
> +MAINTAINER Philippe Mathieu-Daudé <philmd@redhat.com>
> +
> +# Install packages required to build EDK2
> +RUN apt update \
> +    && \
> +    \
> +    DEBIAN_FRONTEND=noninteractive \
> +    apt install --assume-yes --no-install-recommends \
> +        build-essential \
> +        ca-certificates \
> +        dos2unix \
> +        gcc-aarch64-linux-gnu \
> +        gcc-arm-linux-gnueabi \
> +        git \
> +        iasl \
> +        make \
> +        nasm \
> +        python \
> +        uuid-dev \
> +    && \
> +    \
> +    rm -rf /var/lib/apt/lists/*
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index ebcef0ebe9..f799246047 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,3 +1,6 @@
> +include:
> +  - local: '/.gitlab-ci-edk2.yml'
> +
>  before_script:
>   - apt-get update -qq
>   - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
> diff --git a/MAINTAINERS b/MAINTAINERS
> index cd2dc137a3..93620ed406 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2358,6 +2358,8 @@ F: roms/edk2
>  F: roms/edk2-*
>  F: tests/data/uefi-boot-images/
>  F: tests/uefi-test-tools/
> +F: .gitlab-ci-edk2.yml
> +F: .gitlab-ci.d/edk2/
>  
>  Usermode Emulation
>  ------------------
> @@ -2701,7 +2703,7 @@ W: https://cirrus-ci.com/github/qemu/qemu
>  GitLab Continuous Integration
>  M: Thomas Huth <thuth@redhat.com>
>  S: Maintained
> -F: .gitlab-ci.yml
> +F: .gitlab-ci*.yml
>  
>  Guest Test Compilation Support
>  M: Alex Bennée <alex.bennee@linaro.org>
> 

The Docker stuff is not my cup of tea, but the edk2 build stuff looks OK.

Reviewed-by: Laszlo Ersek <lersek@redhat.com>

Thanks
Laszlo

Re: [PATCH v2 2/3] gitlab-ci.yml: Add jobs to build the EDK2 firmware binaries

[Thomas Huth]

On 07/01/2020 16.31, Philippe Mathieu-Daudé wrote:
> Add two GitLab job to build the EDK2 firmware binaries.
> 
> The first job build a Docker image with the packages requisite
> to build EDK2, and store this image in the GitLab registry.
> The second job pull the image from the registry and build the
> EDK2 firmware binaries.
> 
> The docker image is only rebuilt if the GitLab YAML or the
> Dockerfile is updated.
> The second job is only built when the roms/edk2/ submodule is
> updated, when a git-ref starts with 'edk2' or when the last
> commit contains 'EDK2'. The files generated are archived in
> the artifacts.zip file.

Don't you build the EDK2 QEMU binaries in the tianocore CI yet? ...
Feels a little bit weird to build EDK2 as part of the QEMU CI, too...?

> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index ebcef0ebe9..f799246047 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,3 +1,6 @@
> +include:
> +  - local: '/.gitlab-ci-edk2.yml'
> +
>  before_script:
>   - apt-get update -qq
>   - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
> diff --git a/MAINTAINERS b/MAINTAINERS
> index cd2dc137a3..93620ed406 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2358,6 +2358,8 @@ F: roms/edk2
>  F: roms/edk2-*
>  F: tests/data/uefi-boot-images/
>  F: tests/uefi-test-tools/
> +F: .gitlab-ci-edk2.yml
> +F: .gitlab-ci.d/edk2/
>  
>  Usermode Emulation
>  ------------------
> @@ -2701,7 +2703,7 @@ W: https://cirrus-ci.com/github/qemu/qemu
>  GitLab Continuous Integration
>  M: Thomas Huth <thuth@redhat.com>
>  S: Maintained
> -F: .gitlab-ci.yml
> +F: .gitlab-ci*.yml

I don't think that I want to be responsible for the .gitlab-ci-edk2.yml
file, so could you please drop that hunk from the patch? Thanks.

With that hunk removed, FWIW:

Acked-by: Thomas Huth <thuth@redhat.com>