[PULL 0/2] EDK2 firmware patches

[PULL 0/2] EDK2 firmware patches

[Philippe Mathieu-Daudé]

The following changes since commit 43d1455cf84283466e5c22a217db5ef4b8197b14:

  qapi: Fix code generation with Python 3.5 (2020-01-20 12:17:38 +0000)

are available in the Git repository at:

  https://gitlab.com/philmd/qemu.git tags/edk2-next-20200121

for you to fetch changes up to 71920809ceabedf7dbec67dda4d649bcbb11f55c:

  gitlab-ci.yml: Add jobs to build EDK2 firmware binaries (2020-01-21 14:15:48 +0100)

----------------------------------------------------------------

Another set of build-sys patches, to help building the firmware
binaries we use for testing. We almost have reproducible builds.

----------------------------------------------------------------

Philippe Mathieu-Daudé (2):
  roms/edk2-funcs: Force softfloat ARM toolchain prefix on Debian
  gitlab-ci.yml: Add jobs to build EDK2 firmware binaries

 .gitlab-ci-edk2.yml          | 49 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.d/edk2/Dockerfile | 27 ++++++++++++++++++++
 .gitlab-ci.yml               |  3 +++
 MAINTAINERS                  |  2 ++
 roms/edk2-funcs.sh           |  3 +++
 5 files changed, 84 insertions(+)
 create mode 100644 .gitlab-ci-edk2.yml
 create mode 100644 .gitlab-ci.d/edk2/Dockerfile

-- 
2.21.1

[PULL 1/2] roms/edk2-funcs: Force softfloat ARM toolchain prefix on Debian

[Philippe Mathieu-Daudé]

The Debian (based) distributions currently provides 2 ARM
toolchains, documented as [1]:

* The ARM EABI (armel) port targets a range of older 32-bit ARM
  devices, particularly those used in NAS hardware and a variety
  of *plug computers.
* The newer ARM hard-float (armhf) port supports newer, more
  powerful 32-bit devices using version 7 of the ARM architecture
  specification.

For various reasons documented in [2], the EDK2 project suggests
to use the softfloat toolchain (named 'armel' by Debian).

Force the softfloat cross toolchain prefix on Debian distributions.

[1] https://www.debian.org/ports/arm/#status
[2] https://github.com/tianocore/edk2/commit/41203b9a

Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/edk2-funcs.sh | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/roms/edk2-funcs.sh b/roms/edk2-funcs.sh
index 3f4485b201..cd6e4f2c82 100644
--- a/roms/edk2-funcs.sh
+++ b/roms/edk2-funcs.sh
@@ -112,6 +112,9 @@ qemu_edk2_get_cross_prefix()
      ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then
     # no cross-compiler needed
     :
+  elif ( [ -e /etc/debian_version ] && [ "$gcc_arch" == arm ] ); then
+    # force soft-float cross-compiler on Debian
+    printf 'arm-linux-gnueabi-'
   else
     printf '%s-linux-gnu-\n' "$gcc_arch"
   fi
-- 
2.21.1

[PULL 2/2] gitlab-ci.yml: Add jobs to build EDK2 firmware binaries

[Philippe Mathieu-Daudé]

Add two GitLab job to build the EDK2 firmware binaries.

The first job build a Docker image with the packages requisite
to build EDK2, and store this image in the GitLab registry.
The second job pull the image from the registry and build the
EDK2 firmware binaries.

The docker image is only rebuilt if the GitLab YAML or the
Dockerfile is updated.
The second job is only built when the roms/edk2/ submodule is
updated, when a git-ref starts with 'edk2' or when the last
commit contains 'EDK2'. The files generated are archived in
the artifacts.zip file.

With edk2-stable201905, it took 2 minutes 52 seconds to build
the docker image, and 36 minutes 28 seconds to generate the
artifacts.zip with the firmware binaries (filesize: 10MiB).

See: https://gitlab.com/philmd/qemu/pipelines/107553178

Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci-edk2.yml          | 49 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.d/edk2/Dockerfile | 27 ++++++++++++++++++++
 .gitlab-ci.yml               |  3 +++
 MAINTAINERS                  |  2 ++
 4 files changed, 81 insertions(+)
 create mode 100644 .gitlab-ci-edk2.yml
 create mode 100644 .gitlab-ci.d/edk2/Dockerfile

diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci-edk2.yml
new file mode 100644
index 0000000000..088ba4b43a
--- /dev/null
+++ b/.gitlab-ci-edk2.yml
@@ -0,0 +1,49 @@
+docker-edk2:
+ stage: build
+ rules: # Only run this job when the Dockerfile is modified
+ - changes:
+   - .gitlab-ci-edk2.yml
+   - .gitlab-ci.d/edk2/Dockerfile
+   when: always
+ image: docker:19.03.1
+ services:
+ - docker:19.03.1-dind
+ variables:
+  GIT_DEPTH: 3
+  IMAGE_TAG: $CI_REGISTRY_IMAGE:edk2-cross-build
+  # We don't use TLS
+  DOCKER_HOST: tcp://docker:2375
+  DOCKER_TLS_CERTDIR: ""
+ before_script:
+ - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
+ script:
+ - docker pull $IMAGE_TAG || true
+ - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
+                                        --tag $IMAGE_TAG .gitlab-ci.d/edk2
+ - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
+ - docker push $IMAGE_TAG
+
+build-edk2:
+ rules: # Only run this job when ...
+ - changes: # ... roms/edk2/ is modified (submodule updated)
+   - roms/edk2/*
+   when: always
+ - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
+   when: always
+ - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
+   when: always
+ artifacts:
+   paths: # 'artifacts.zip' will contains the following files:
+   - pc-bios/edk2*bz2
+   - pc-bios/edk2-licenses.txt
+   - edk2-stdout.log
+   - edk2-stderr.log
+ image: $CI_REGISTRY_IMAGE:edk2-cross-build
+ variables:
+   GIT_DEPTH: 3
+ script: # Clone the required submodules and build EDK2
+ - git submodule update --init roms/edk2
+ - git -C roms/edk2 submodule update --init
+ - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
+ - echo "=== Using ${JOBS} simultaneous jobs ==="
+ - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a edk2-stderr.log >&2
diff --git a/.gitlab-ci.d/edk2/Dockerfile b/.gitlab-ci.d/edk2/Dockerfile
new file mode 100644
index 0000000000..b4584d1cf6
--- /dev/null
+++ b/.gitlab-ci.d/edk2/Dockerfile
@@ -0,0 +1,27 @@
+#
+# Docker image to cross-compile EDK2 firmware binaries
+#
+FROM ubuntu:16.04
+
+MAINTAINER Philippe Mathieu-Daudé <philmd@redhat.com>
+
+# Install packages required to build EDK2
+RUN apt update \
+    && \
+    \
+    DEBIAN_FRONTEND=noninteractive \
+    apt install --assume-yes --no-install-recommends \
+        build-essential \
+        ca-certificates \
+        dos2unix \
+        gcc-aarch64-linux-gnu \
+        gcc-arm-linux-gnueabi \
+        git \
+        iasl \
+        make \
+        nasm \
+        python \
+        uuid-dev \
+    && \
+    \
+    rm -rf /var/lib/apt/lists/*
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d9a90f795d..228783993e 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,3 +1,6 @@
+include:
+  - local: '/.gitlab-ci-edk2.yml'
+
 before_script:
  - apt-get update -qq
  - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev genisoimage
diff --git a/MAINTAINERS b/MAINTAINERS
index 55d3642e6c..2c768ed3d8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2368,6 +2368,8 @@ F: roms/edk2
 F: roms/edk2-*
 F: tests/data/uefi-boot-images/
 F: tests/uefi-test-tools/
+F: .gitlab-ci-edk2.yml
+F: .gitlab-ci.d/edk2/
 
 Usermode Emulation
 ------------------
-- 
2.21.1

Re: [PULL 0/2] EDK2 firmware patches

[Peter Maydell]

On Tue, 21 Jan 2020 at 15:19, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>
> The following changes since commit 43d1455cf84283466e5c22a217db5ef4b8197b14:
>
>   qapi: Fix code generation with Python 3.5 (2020-01-20 12:17:38 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/philmd/qemu.git tags/edk2-next-20200121
>
> for you to fetch changes up to 71920809ceabedf7dbec67dda4d649bcbb11f55c:
>
>   gitlab-ci.yml: Add jobs to build EDK2 firmware binaries (2020-01-21 14:15:48 +0100)
>
> ----------------------------------------------------------------
>
> Another set of build-sys patches, to help building the firmware
> binaries we use for testing. We almost have reproducible builds.
>
> ----------------------------------------------------------------


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.0
for any user-visible changes.

-- PMM

Re: [PULL 0/2] EDK2 firmware patches

[Laszlo Ersek]

On 01/21/20 16:15, Philippe Mathieu-Daudé wrote:

> Another set of build-sys patches, to help building the firmware
> binaries we use for testing. We almost have reproducible builds.

\o/