[PATCH v4 0/3] Yocto Gitlab CI

[PATCH v4 0/3] Yocto Gitlab CI

[Bertrand Marquis]

This patch series is a first attempt to check if we could use Yocto in
gitlab ci to build and run xen on qemu for arm, arm64 and x86.

The first patch is introducing a container template from which container
files can be generated for all combinations we want to support (qemu
arm, arm64 and x86 targets and hosts x86_64 and arm64).
It is also introducing a generic build script (build-yocto.sh) that is
used to create, build and run a Yocto project.

The second patch is adding a way to easily clean locally created
containers.

The third patch is introducing some gitlab CI templates and jobs so that
we can test a Xen source tree in gitlab-CI using Yocto.

This has been tested on a x86 host machine and on an arm host machine
(with mk_dsdt.c fix).

Changes in v4:
- rework the container generation to support multiple hosts
- rework the container generation to use a single template for all
  docker files (make process is generating the docker files).

Changes in v3:
- limit number of jobs in yocto
- do not copy build script inside container
- add patch from Michal to create gitlab jobs

Changes in v2:
- remove gitignore patch which was merged
- add a --dump-log support in build-yocto.sh script and use it during
  container creation to see the error logs.


Bertrand Marquis (2):
  automation: Create Yocto docker images
  automation: Add a clean rule for containers

Michal Orzel (1):
  automation: Add CI test jobs for Yocto

 automation/build/Makefile                  |  26 +-
 automation/build/yocto/build-yocto.sh      | 349 +++++++++++++++++++++
 automation/build/yocto/yocto.dockerfile.in | 114 +++++++
 automation/build/yocto/yocto.inc           |  41 +++
 automation/gitlab-ci/test.yaml             |  43 +++
 5 files changed, 571 insertions(+), 2 deletions(-)
 create mode 100755 automation/build/yocto/build-yocto.sh
 create mode 100644 automation/build/yocto/yocto.dockerfile.in
 create mode 100644 automation/build/yocto/yocto.inc

-- 
2.25.1

[PATCH v4 1/3] automation: Create Yocto docker images

[Bertrand Marquis]

Add containers suitable to run yocto kirkstone build based on ubuntu
22.04. It contains all packages required by Yocto and a checkout of the
layers required to build Xen with Yocto.

Add a generic docker image template to be used to automatically generate
docker files for different configurations:
- specific yocto version
- different targets (qemu arm, arm64 and x86)
- different host platforms (x86 or arm64)

During a call to 'make all', only the images for the current host
platform will be generated.
If needed, images for an other host platform can be generated manually
by calling the right make target (see make help).

Add a build script to build and run xen on qemu using Yocto.
The script supports arm32, arm64 and x86_64 and checks that dom0 is
properly booting. At this stage this does not run any guest on top of
dom0. The script is to be executed in one of the docker images to build
and run a system using a Xen source tree.

Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Changes in v4:
- Rework the system to have one dockerfile template from which make will
generate the required dockerfiles for the wanted configuration
- add support for different host architectures
- Merge the generation system into one single dockerfile
- Merge patches 1 and 2 in a single patch
- Introduce CONTAINERS_EXTRA to have extra containers not built by
default (for those not used by CI but useful to users)
Changes in v3:
- limit number of jobs in yocto by default to 8 and add --num-jobs
option to the script to set a custom number of jobs
- do not copy anymore the build-yocto.sh script inside the main image so
that the current one in the repository is used when running
Changes in v2:
- add a --dump-log command line option to build-yocto.sh script to dump
the logs if an error occurs.
Changes in v1:
- add --image command line argument to build-yocto.sh to allow building
something different than xen-image-minimal.
- modify dockerfile to have one layer per line and make it easier to add
other. I kept the for loop to keep the number of docker steps lower
- update commit message to warn that no guest are tested.
- fix build-yocto script to properly return with an error if expect
script ends up in timeout or EOF.
---
 automation/build/Makefile                  |  14 +-
 automation/build/yocto/build-yocto.sh      | 349 +++++++++++++++++++++
 automation/build/yocto/yocto.dockerfile.in | 114 +++++++
 automation/build/yocto/yocto.inc           |  41 +++
 4 files changed, 516 insertions(+), 2 deletions(-)
 create mode 100755 automation/build/yocto/build-yocto.sh
 create mode 100644 automation/build/yocto/yocto.dockerfile.in
 create mode 100644 automation/build/yocto/yocto.inc

diff --git a/automation/build/Makefile b/automation/build/Makefile
index a4b2b85178cf..72a5335baec1 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -1,13 +1,18 @@
 
 # the base of where these containers will appear
 REGISTRY := registry.gitlab.com/xen-project/xen
-CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
+CONTAINERS = $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard */*.dockerfile)))
+CONTAINERS_EXTRA =
 DOCKER_CMD ?= docker
 
+include yocto/yocto.inc
+
 help:
 	@echo "Builds containers for building Xen based on different distros"
 	@echo "To build one run 'make DISTRO/VERSION'. Available containers:"
 	@$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
+	@echo "Extra containers (not built using make all):"
+	@$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo ${file};)
 	@echo "To push container builds, set the env var PUSH"
 
 %: %.dockerfile ## Builds containers
@@ -16,5 +21,10 @@ help:
 		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
 	fi
 
-.PHONY: all
+.PHONY: all clean
 all: $(CONTAINERS)
+
+# Remove generated dockerfiles for yocto
+clean:
+	rm -f yocto/*.dockerfiles
+
diff --git a/automation/build/yocto/build-yocto.sh b/automation/build/yocto/build-yocto.sh
new file mode 100755
index 000000000000..d0c93dfaffe0
--- /dev/null
+++ b/automation/build/yocto/build-yocto.sh
@@ -0,0 +1,349 @@
+#!/bin/bash
+#
+# Yocto meta virtualization build and run script
+#
+# This script is building Yocto xen-image-minimal for qemu targets and run
+# them using runqemu inside yocto to check that dom0 is booting properly
+# The build is using a local xen source tree so that specific patches can be
+# tested.
+# In order to optimize the build time, a build cache is used so that only xen
+# packages and its dependencies are rebuilt (qemu and final image mainly).
+#
+
+# Directories
+YOCTODIR="$HOME/yocto-layers"
+CACHEDIR="$HOME/yocto-cache"
+LOGDIR="$HOME/logs"
+XENDIR="$HOME/xen"
+BUILDDIR="$HOME/build"
+
+# what yocto bsp we support
+TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
+VERBOSE="n"
+TARGETLIST=""
+BUILDJOBS="8"
+
+# actions to do
+do_clean="n"
+do_build="y"
+do_run="y"
+do_localsrc="n"
+do_dump="n"
+build_result=0
+
+# layers to include in the project
+build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
+                 meta-openembedded/meta-oe meta-openembedded/meta-python \
+                 meta-openembedded/meta-filesystems \
+                 meta-openembedded/meta-networking meta-virtualization"
+
+# yocto image to build
+build_image="xen-image-minimal"
+
+function print_progress() {
+    echo -n "$(date +%T) $*"
+}
+
+function run_task() {
+    local task_name="$1"
+    local task_target="$2"
+
+    task_log="${task_name//project_}-${task_target}"
+
+    mkdir -p "${LOGDIR}"
+    print_progress
+    echo -n "${task_name//project_} ${task_target}: "
+    if [ "${VERBOSE}" = "n" ]; then
+        $@ > "${LOGDIR}/${task_log}.log" 2>&1
+    else
+        $@ 2>&1 | tee "${LOGDIR}/${task_log}.log"
+    fi
+
+    if [ ${?} -ne 0 ]; then
+        echo "Error"
+        build_result=$((build_result+1))
+        if [ "${do_dump}" = "y" ]; then
+            echo
+            echo "############ LOGS-START ############"
+            cat "${LOGDIR}/${task_log}.log"
+            echo "############  LOGS-END  ############"
+            echo
+        fi
+        return 1
+    else
+        echo "OK"
+        return 0
+    fi
+}
+
+function project_create() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+
+    (
+        # init yocto project
+        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
+
+        # add needed layers
+        for layer in ${build_layerlist}; do
+            bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1
+        done
+    ) || return 1
+
+    # customize project configuration
+    cat <<EOF >> "${destdir}/conf/local.conf"
+# Yocto BSP
+MACHINE = "${target}"
+
+# Use local cache to reuse previous builds results
+SSTATE_DIR = "${CACHEDIR}/sstate-cache"
+DL_DIR = "${CACHEDIR}/downloads"
+
+# Enable xen and virtualization
+DISTRO_FEATURES = " virtualization xen ipv4"
+
+# Speed up run by not generating ssh host keys
+IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
+
+# Save some disk space
+INHERIT += "rm_work"
+
+# Reduce number of jobs
+BB_NUMBER_THREADS="${BUILDJOBS}"
+
+EOF
+
+    if [ "${do_localsrc}" = "y" ]; then
+        XENVERS=$(grep -e "^XEN_REL" \
+            "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
+            2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
+            | sed -e "s/.*=//" | sort -V | tail -n 1)
+
+        XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
+        XENSUB=$(basename "$(realpath -m "${XENDIR}")")
+
+        cat <<EOF >> "${destdir}/conf/local.conf"
+# Use local sources for xen and xen-tools
+FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
+FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
+
+SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
+SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
+
+PREFERRED_VERSION:pn-xen = "${XENVERS}%"
+PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
+
+S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
+S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
+
+SRCREV:pn-xen = "\${AUTOREV}"
+SRCREV:pn-xen-tools = "\${AUTOREV}"
+
+SRCPV:pn-xen = "1"
+SRCPV:pn-xen-tools = "1"
+
+# Disable all QA errors as the recipe is not up to date with changes in Xen
+# when we use local sources
+ERROR_QA:pn-xen = "arch"
+ERROR_QA:pn-xen-tools = "arch"
+
+EOF
+    fi
+}
+
+function project_build() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+
+    (
+        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
+
+        bitbake "${build_image}" || exit 1
+    ) || return 1
+}
+
+function project_clean() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+
+    rm -rf "${destdir}"
+}
+
+function project_run() {
+    target="${1:?}"
+    destdir="${BUILDDIR}/${target}"
+    (
+        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null 2>&1
+
+        /usr/bin/expect <<EOF
+set timeout 100
+spawn bash -c "runqemu serialstdio nographic slirp"
+
+expect_after {
+    -re "(.*)\r" {
+        exp_continue
+    }
+    timeout {send_user "ERROR-Timeout!\n"; exit 1}
+    eof {send_user "ERROR-EOF!\n"; exit 1}
+}
+
+# wait initial login
+expect -re ".* login: "
+send "root\r"
+expect -re "root@.*# "
+
+EOF
+    exit $?
+    ) || return 1
+}
+
+function help() {
+    cat <<EOF
+Usage: ${0} [TARGET1] [TARGET2]
+
+This script is build the yocto xen-image-minimal for different qemu targets
+and is running it after.
+Without any target specified, all supported targets are done.
+
+Options:
+  -h, --help       Print this help
+  -v, --verbose    Verbose build
+  --list-target    List supported targets
+  --clean          Clean existing project before starting
+  --no-build       Do not build (to run an already built project)
+  --no-run         Do not run
+  --num-jobs=NUM   Define the number of parallel jobs in Yocto.
+                   Default: ${BUILDJOBS}
+  --dump-log       On error, dump the logs on the console
+  --image=IMG      Yocto image or package to build
+                   Default: xen-image-minimal
+  --xen-dir=DIR    path to xen hypervisor source tree
+                   if not provide, normal yocto version of xen is built
+                   Default: ${XENDIR}
+  --out-dir=DIR    directory where to create the projectss
+                   Default: ${BUILDDIR}
+  --log-dir=DIR    directory to store logs
+                   Default: ${LOGDIR}
+  --cache-dir=DIR  directory where to take and store build cache
+                   Default: ${CACHEDIR}
+  --layer-dir=DIR  directory containing the checkout of yocto layers
+                   Default: ${YOCTODIR}
+EOF
+}
+
+for OPTION in "$@"
+do
+    case ${OPTION} in
+        -h|--help)
+            help
+            exit 0
+            ;;
+        -v|--verbose)
+            VERBOSE="y"
+            ;;
+        --list-targets)
+            echo "${TARGET_SUPPORTED}"
+            exit 0
+            ;;
+        --clean)
+            do_clean="y"
+            ;;
+        --no-build)
+            do_build="n"
+            ;;
+        --no-run)
+            do_run="n"
+            ;;
+        --dump-log)
+            do_dump="y"
+            ;;
+        --num-jobs=*)
+            BUILDJOBS="${OPTION#*=}"
+            ;;
+        --image=*)
+            build_image="${OPTION#*=}"
+            ;;
+        --xen-dir=*)
+            XENDIR="${OPTION#*=}"
+            if [ ! -e "${XENDIR}/xen/Makefile" ]; then
+                echo "No Xen source tree in ${XENDIR}"
+                exit 1
+            fi
+            do_localsrc="y"
+            ;;
+        --out-dir=*)
+            BUILDDIR="${OPTION#*=}"
+            ;;
+        --log-dir=*)
+            LOGDIR="${OPTION#*=}"
+            ;;
+        --cache-dir=*)
+            CACHEDIR="${OPTION#*=}"
+            ;;
+        --layer-dir=*)
+            YOCTODIR="${OPTION#*=}"
+            ;;
+        --*)
+            echo "Invalid option ${OPTION}"
+            help
+            exit 1
+            ;;
+        *)
+            if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
+                TARGETLIST="${TARGETLIST} ${OPTION}"
+            else
+                echo "Unsupported target ${OPTION}"
+                exit 1
+            fi
+            ;;
+    esac
+done
+
+# if no target is specified build all targets
+if [ -z "${TARGETLIST}" ]; then
+    TARGETLIST="${TARGET_SUPPORTED}"
+fi
+
+mkdir -p "${CACHEDIR}"
+mkdir -p "${LOGDIR}"
+mkdir -p "${BUILDDIR}"
+
+# Make sure we have an absolute path
+YOCTODIR=$(realpath -m "${YOCTODIR}")
+CACHEDIR=$(realpath -m "${CACHEDIR}")
+BUILDDIR=$(realpath -m "${BUILDDIR}")
+LOGDIR=$(realpath -m "${LOGDIR}")
+if [ "${do_localsrc}" = "y" ]; then
+    XENDIR=$(realpath -m "${XENDIR}")
+fi
+
+# Check that we have all the layers we need
+for f in ${build_layerlist}; do
+    if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
+        echo "Layer ${f} missing in ${YOCTODIR}"
+        exit 1
+    fi
+done
+
+for f in ${TARGETLIST}; do
+    if [ "${do_clean}" = "y" ]; then
+        run_task project_clean ${f}
+    fi
+    if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then
+        run_task project_create ${f}
+    fi
+    if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then
+        if [ "${do_build}" = "y" ]; then
+            run_task project_build ${f}
+        fi
+        if [ "${do_run}" = "y" ]; then
+            run_task project_run ${f}
+        fi
+
+    fi
+done
+
+print_progress "Build Complete (${build_result} errors)"
+echo
+exit ${build_result}
+
diff --git a/automation/build/yocto/yocto.dockerfile.in b/automation/build/yocto/yocto.dockerfile.in
new file mode 100644
index 000000000000..5350bb2b87b7
--- /dev/null
+++ b/automation/build/yocto/yocto.dockerfile.in
@@ -0,0 +1,114 @@
+# Docker file to create an environment to build yocto with virtualization
+#
+# Arguments that can be passed during image creation using --build-arg:
+# "host_uid=$(id -u)": to use current user uid for build user in the image
+# "host_gid=$(id -g)": to use current user gid for build user in the image
+# "ubuntu_version=VERS": to select the ubuntu version number
+
+# Use standard ubuntu minimal
+ARG ubuntu_version=22.04
+From ##DOCKERPLAT##ubuntu:$ubuntu_version AS base
+LABEL maintainer.name="The Xen Project " \
+      maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Install minimal ubuntu requirements for yocto and other tools we need
+# See https://docs.yoctoproject.org/4.0.1/brief-yoctoprojectqs/index.html#build-host-packages
+RUN apt-get update && \
+    apt-get --quiet --yes install \
+        gawk \
+        wget \
+        git \
+        diffstat \
+        unzip \
+        texinfo \
+        gcc \
+        build-essential \
+        chrpath \
+        socat \
+        cpio \
+        python3 \
+        python3-pip \
+        python3-pexpect \
+        xz-utils \
+        debianutils \
+        iputils-ping \
+        python3-git \
+        python3-jinja2 \
+        libegl1-mesa \
+        libsdl1.2-dev \
+        python3-subunit \
+        mesa-common-dev \
+        zstd \
+        liblz4-tool \
+        file \
+        vim \
+        bison \
+        expect \
+        locales \
+        liblz4-tool \
+        zstd \
+        openssl \
+        libssl3 \
+        ca-certificates \
+        && \
+        apt-get autoremove -y && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
+
+# Use bash as shell
+RUN rm /bin/sh && ln -s bash /bin/sh
+
+# Fix local for yocto
+RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
+    LANG=en_US.UTF-8
+ENV LANG en_US.UTF-8
+ENV LC_ALL en_US.UTF-8
+
+# Create a user for the build (we don't want to build as root)
+ENV USER_NAME docker-build
+ARG host_uid=1000
+ARG host_gid=1000
+RUN groupadd -g $host_gid $USER_NAME && \
+    useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
+
+# Switch to our user instead of root and start in its home
+USER $USER_NAME
+WORKDIR /home/$USER_NAME
+
+# Create needed directories
+RUN mkdir -p /home/$USER_NAME/yocto-layers \
+             /home/$USER_NAME/yocto-cache \
+             /home/$USER_NAME/logs \
+             /home/$USER_NAME/bin \
+             /home/$USER_NAME/xen && \
+    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
+
+# clone yocto repositories we need
+RUN for rep in \
+                https://github.com/openembedded/meta-openembedded \
+                https://git.yoctoproject.org/poky \
+                https://git.yoctoproject.org/meta-virtualization \
+            ; do \
+        git -C /home/$USER_NAME/yocto-layers \
+            clone -b ##YOCTOVERSION## --single-branch $rep; \
+    done
+
+# The builder stage is building an initial cache state that we include in the
+# final image
+From base AS builder
+
+# This step can take one to several hours depending on your download bandwith
+# and the speed of your computer
+COPY ./build-yocto.sh /
+RUN /build-yocto.sh --dump-log ##YOCTOTARGET##
+
+From base
+
+# Only copy the cache status
+COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
+
+LABEL maintainer.name="The Xen Project " \
+      maintainer.email="xen-devel@lists.xenproject.org"
+
diff --git a/automation/build/yocto/yocto.inc b/automation/build/yocto/yocto.inc
new file mode 100644
index 000000000000..04076bc8d174
--- /dev/null
+++ b/automation/build/yocto/yocto.inc
@@ -0,0 +1,41 @@
+# This makefile generates the docker files for Yocto builds
+# The containers for the current architecture are the one built using make all
+# To build containers for a different architecture, you need to call make for
+# the image you want explicitely
+# The containers are named this way:
+# YOCTOVERSION-TARGET for x86_64 hosts
+# YOCTOVERSION-TARGET-arm64v8 for arm64 hosts
+
+# Yocto versions we are currently using
+YOCTO_VERSION = kirkstone
+
+# Yocto BSPs we want to build for
+YOCTO_TARGETS = qemuarm64 qemuarm qemux86-64
+
+# Supported Host platforms (host architecture specific ones)
+YOCTO_HOSTS = amd64 arm64v8
+
+# Architecture we want to use in gitlab CI (depends on runners arch)
+CI_ARCH = arm64v8
+
+define GEN_DOCKER
+# Make all is generating what we want in the CI
+ifeq ($(CI_ARCH),$(3))
+CONTAINERS += yocto/$(1)-$(2)$(4)
+else
+CONTAINERS_EXTRA += yocto/$(1)-$(2)$(4)
+endif
+
+yocto/$(1)-$(2)$(4).dockerfile: yocto/yocto.dockerfile.in
+	echo > $$@
+	cat $$< | \
+	    sed -e "s,##YOCTOVERSION##,$(1),g" | \
+	    sed -e "s,##YOCTOTARGET##,$(2),g" | \
+	    sed -e "s,##DOCKERPLAT##,$(3)/,g" > $$@
+
+endef
+
+$(eval $(foreach vers,$(YOCTO_VERSION),\
+    $(foreach tar,$(YOCTO_TARGETS),\
+    $(foreach hst,$(YOCTO_HOSTS),\
+    $(call GEN_DOCKER,$(vers),$(tar),$(hst),$(if $(filter amd64,$(hst)),,-$(hst)))))))
-- 
2.25.1

Re: [PATCH v4 1/3] automation: Create Yocto docker images

[Michal Orzel]

Hi Bertrand,

Just, some minor comments.

On 17/11/2022 10:39, Bertrand Marquis wrote:
> 
> 
> Add containers suitable to run yocto kirkstone build based on ubuntu
> 22.04. It contains all packages required by Yocto and a checkout of the
> layers required to build Xen with Yocto.
> 
> Add a generic docker image template to be used to automatically generate
> docker files for different configurations:
> - specific yocto version
> - different targets (qemu arm, arm64 and x86)
> - different host platforms (x86 or arm64)
> 
> During a call to 'make all', only the images for the current host
> platform will be generated.
This looks like a stale comment. The reason being, in the new version, by default
we build the images for the architectures expected by the CI and its runners
to match the current workflow.

> If needed, images for an other host platform can be generated manually
> by calling the right make target (see make help).
> 
> Add a build script to build and run xen on qemu using Yocto.
> The script supports arm32, arm64 and x86_64 and checks that dom0 is
> properly booting. At this stage this does not run any guest on top of
> dom0. The script is to be executed in one of the docker images to build
> and run a system using a Xen source tree.
> 
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
> ---
> Changes in v4:
> - Rework the system to have one dockerfile template from which make will
> generate the required dockerfiles for the wanted configuration
That is great.

> - add support for different host architectures
> - Merge the generation system into one single dockerfile
> - Merge patches 1 and 2 in a single patch
> - Introduce CONTAINERS_EXTRA to have extra containers not built by
> default (for those not used by CI but useful to users)
> Changes in v3:
> - limit number of jobs in yocto by default to 8 and add --num-jobs
> option to the script to set a custom number of jobs
> - do not copy anymore the build-yocto.sh script inside the main image so
> that the current one in the repository is used when running
> Changes in v2:
> - add a --dump-log command line option to build-yocto.sh script to dump
> the logs if an error occurs.
> Changes in v1:
> - add --image command line argument to build-yocto.sh to allow building
> something different than xen-image-minimal.
> - modify dockerfile to have one layer per line and make it easier to add
> other. I kept the for loop to keep the number of docker steps lower
> - update commit message to warn that no guest are tested.
> - fix build-yocto script to properly return with an error if expect
> script ends up in timeout or EOF.
> ---
>  automation/build/Makefile                  |  14 +-
>  automation/build/yocto/build-yocto.sh      | 349 +++++++++++++++++++++
>  automation/build/yocto/yocto.dockerfile.in | 114 +++++++
>  automation/build/yocto/yocto.inc           |  41 +++
>  4 files changed, 516 insertions(+), 2 deletions(-)
>  create mode 100755 automation/build/yocto/build-yocto.sh
>  create mode 100644 automation/build/yocto/yocto.dockerfile.in
>  create mode 100644 automation/build/yocto/yocto.inc
> 
> diff --git a/automation/build/Makefile b/automation/build/Makefile
> index a4b2b85178cf..72a5335baec1 100644
> --- a/automation/build/Makefile
> +++ b/automation/build/Makefile
> @@ -1,13 +1,18 @@
> 
>  # the base of where these containers will appear
>  REGISTRY := registry.gitlab.com/xen-project/xen
> -CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
> +CONTAINERS = $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard */*.dockerfile)))
> +CONTAINERS_EXTRA =
>  DOCKER_CMD ?= docker
> 
> +include yocto/yocto.inc
> +
>  help:
>         @echo "Builds containers for building Xen based on different distros"
>         @echo "To build one run 'make DISTRO/VERSION'. Available containers:"
>         @$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
> +       @echo "Extra containers (not built using make all):"
> +       @$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo ${file};)
>         @echo "To push container builds, set the env var PUSH"
> 
>  %: %.dockerfile ## Builds containers
> @@ -16,5 +21,10 @@ help:
>                 $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
>         fi
> 
> -.PHONY: all
> +.PHONY: all clean
>  all: $(CONTAINERS)
> +
> +# Remove generated dockerfiles for yocto
> +clean:
> +       rm -f yocto/*.dockerfiles
Are these files needed after make is completed?
If not, maybe to avoid having some untracked files in the tree, they could be removed after make is done?

> +
> diff --git a/automation/build/yocto/build-yocto.sh b/automation/build/yocto/build-yocto.sh
> new file mode 100755
> index 000000000000..d0c93dfaffe0
> --- /dev/null
> +++ b/automation/build/yocto/build-yocto.sh
> @@ -0,0 +1,349 @@
> +#!/bin/bash
> +#
> +# Yocto meta virtualization build and run script
> +#
> +# This script is building Yocto xen-image-minimal for qemu targets and run
> +# them using runqemu inside yocto to check that dom0 is booting properly
Missing dot at the end of a sentence.

> +# The build is using a local xen source tree so that specific patches can be
> +# tested.
> +# In order to optimize the build time, a build cache is used so that only xen
> +# packages and its dependencies are rebuilt (qemu and final image mainly).
> +#
> +
> +# Directories
> +YOCTODIR="$HOME/yocto-layers"
> +CACHEDIR="$HOME/yocto-cache"
> +LOGDIR="$HOME/logs"
> +XENDIR="$HOME/xen"
> +BUILDDIR="$HOME/build"
> +
> +# what yocto bsp we support
> +TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
> +VERBOSE="n"
> +TARGETLIST=""
> +BUILDJOBS="8"
> +
> +# actions to do
> +do_clean="n"
> +do_build="y"
> +do_run="y"
> +do_localsrc="n"
> +do_dump="n"
> +build_result=0
> +
> +# layers to include in the project
> +build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
> +                 meta-openembedded/meta-oe meta-openembedded/meta-python \
> +                 meta-openembedded/meta-filesystems \
> +                 meta-openembedded/meta-networking meta-virtualization"
> +
> +# yocto image to build
> +build_image="xen-image-minimal"
> +
> +function print_progress() {
> +    echo -n "$(date +%T) $*"
> +}
> +
> +function run_task() {
> +    local task_name="$1"
> +    local task_target="$2"
> +
> +    task_log="${task_name//project_}-${task_target}"
> +
> +    mkdir -p "${LOGDIR}"
> +    print_progress
> +    echo -n "${task_name//project_} ${task_target}: "
> +    if [ "${VERBOSE}" = "n" ]; then
> +        $@ > "${LOGDIR}/${task_log}.log" 2>&1
> +    else
> +        $@ 2>&1 | tee "${LOGDIR}/${task_log}.log"
> +    fi
> +
> +    if [ ${?} -ne 0 ]; then
> +        echo "Error"
> +        build_result=$((build_result+1))
> +        if [ "${do_dump}" = "y" ]; then
> +            echo
> +            echo "############ LOGS-START ############"
> +            cat "${LOGDIR}/${task_log}.log"
> +            echo "############  LOGS-END  ############"
> +            echo
> +        fi
> +        return 1
> +    else
> +        echo "OK"
> +        return 0
> +    fi
> +}
> +
> +function project_create() {
> +    target="${1:?}"
> +    destdir="${BUILDDIR}/${target}"
> +
> +    (
> +        # init yocto project
> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
> +
> +        # add needed layers
> +        for layer in ${build_layerlist}; do
> +            bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1
> +        done
> +    ) || return 1
> +
> +    # customize project configuration
> +    cat <<EOF >> "${destdir}/conf/local.conf"
> +# Yocto BSP
> +MACHINE = "${target}"
> +
> +# Use local cache to reuse previous builds results
> +SSTATE_DIR = "${CACHEDIR}/sstate-cache"
> +DL_DIR = "${CACHEDIR}/downloads"
> +
> +# Enable xen and virtualization
> +DISTRO_FEATURES = " virtualization xen ipv4"
> +
> +# Speed up run by not generating ssh host keys
> +IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
> +
> +# Save some disk space
> +INHERIT += "rm_work"
> +
> +# Reduce number of jobs
> +BB_NUMBER_THREADS="${BUILDJOBS}"
> +
> +EOF
> +
> +    if [ "${do_localsrc}" = "y" ]; then
> +        XENVERS=$(grep -e "^XEN_REL" \
> +            "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
> +            2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
> +            | sed -e "s/.*=//" | sort -V | tail -n 1)
> +
> +        XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
> +        XENSUB=$(basename "$(realpath -m "${XENDIR}")")
> +
> +        cat <<EOF >> "${destdir}/conf/local.conf"
> +# Use local sources for xen and xen-tools
> +FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
> +FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
> +
> +SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
> +SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
> +
> +PREFERRED_VERSION:pn-xen = "${XENVERS}%"
> +PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
> +
> +S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
> +S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
> +
> +SRCREV:pn-xen = "\${AUTOREV}"
> +SRCREV:pn-xen-tools = "\${AUTOREV}"
> +
> +SRCPV:pn-xen = "1"
> +SRCPV:pn-xen-tools = "1"
> +
> +# Disable all QA errors as the recipe is not up to date with changes in Xen
> +# when we use local sources
> +ERROR_QA:pn-xen = "arch"
> +ERROR_QA:pn-xen-tools = "arch"
> +
> +EOF
> +    fi
> +}
> +
> +function project_build() {
> +    target="${1:?}"
> +    destdir="${BUILDDIR}/${target}"
> +
> +    (
> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
> +
> +        bitbake "${build_image}" || exit 1
> +    ) || return 1
> +}
> +
> +function project_clean() {
> +    target="${1:?}"
> +    destdir="${BUILDDIR}/${target}"
> +
> +    rm -rf "${destdir}"
> +}
> +
> +function project_run() {
> +    target="${1:?}"
> +    destdir="${BUILDDIR}/${target}"
> +    (
> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null 2>&1
> +
> +        /usr/bin/expect <<EOF
> +set timeout 100
> +spawn bash -c "runqemu serialstdio nographic slirp"
> +
> +expect_after {
> +    -re "(.*)\r" {
> +        exp_continue
> +    }
> +    timeout {send_user "ERROR-Timeout!\n"; exit 1}
> +    eof {send_user "ERROR-EOF!\n"; exit 1}
> +}
> +
> +# wait initial login
> +expect -re ".* login: "
> +send "root\r"
> +expect -re "root@.*# "
> +
> +EOF
> +    exit $?
> +    ) || return 1
> +}
> +
> +function help() {
> +    cat <<EOF
> +Usage: ${0} [TARGET1] [TARGET2]
> +
> +This script is build the yocto xen-image-minimal for different qemu targets
> +and is running it after.
> +Without any target specified, all supported targets are done.
> +
> +Options:
> +  -h, --help       Print this help
> +  -v, --verbose    Verbose build
> +  --list-target    List supported targets
> +  --clean          Clean existing project before starting
> +  --no-build       Do not build (to run an already built project)
> +  --no-run         Do not run
> +  --num-jobs=NUM   Define the number of parallel jobs in Yocto.
> +                   Default: ${BUILDJOBS}
> +  --dump-log       On error, dump the logs on the console
> +  --image=IMG      Yocto image or package to build
> +                   Default: xen-image-minimal
> +  --xen-dir=DIR    path to xen hypervisor source tree
> +                   if not provide, normal yocto version of xen is built
> +                   Default: ${XENDIR}
> +  --out-dir=DIR    directory where to create the projectss
> +                   Default: ${BUILDDIR}
> +  --log-dir=DIR    directory to store logs
> +                   Default: ${LOGDIR}
> +  --cache-dir=DIR  directory where to take and store build cache
> +                   Default: ${CACHEDIR}
> +  --layer-dir=DIR  directory containing the checkout of yocto layers
> +                   Default: ${YOCTODIR}
> +EOF
> +}
> +
> +for OPTION in "$@"
> +do
> +    case ${OPTION} in
> +        -h|--help)
> +            help
> +            exit 0
> +            ;;
> +        -v|--verbose)
> +            VERBOSE="y"
> +            ;;
> +        --list-targets)
> +            echo "${TARGET_SUPPORTED}"
> +            exit 0
> +            ;;
> +        --clean)
> +            do_clean="y"
> +            ;;
> +        --no-build)
> +            do_build="n"
> +            ;;
> +        --no-run)
> +            do_run="n"
> +            ;;
> +        --dump-log)
> +            do_dump="y"
> +            ;;
> +        --num-jobs=*)
> +            BUILDJOBS="${OPTION#*=}"
> +            ;;
> +        --image=*)
> +            build_image="${OPTION#*=}"
> +            ;;
> +        --xen-dir=*)
> +            XENDIR="${OPTION#*=}"
> +            if [ ! -e "${XENDIR}/xen/Makefile" ]; then
> +                echo "No Xen source tree in ${XENDIR}"
> +                exit 1
> +            fi
> +            do_localsrc="y"
> +            ;;
> +        --out-dir=*)
> +            BUILDDIR="${OPTION#*=}"
> +            ;;
> +        --log-dir=*)
> +            LOGDIR="${OPTION#*=}"
> +            ;;
> +        --cache-dir=*)
> +            CACHEDIR="${OPTION#*=}"
> +            ;;
> +        --layer-dir=*)
> +            YOCTODIR="${OPTION#*=}"
> +            ;;
> +        --*)
> +            echo "Invalid option ${OPTION}"
> +            help
> +            exit 1
> +            ;;
> +        *)
> +            if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
> +                TARGETLIST="${TARGETLIST} ${OPTION}"
> +            else
> +                echo "Unsupported target ${OPTION}"
> +                exit 1
> +            fi
> +            ;;
> +    esac
> +done
> +
> +# if no target is specified build all targets
> +if [ -z "${TARGETLIST}" ]; then
> +    TARGETLIST="${TARGET_SUPPORTED}"
> +fi
> +
> +mkdir -p "${CACHEDIR}"
> +mkdir -p "${LOGDIR}"
> +mkdir -p "${BUILDDIR}"
> +
> +# Make sure we have an absolute path
> +YOCTODIR=$(realpath -m "${YOCTODIR}")
> +CACHEDIR=$(realpath -m "${CACHEDIR}")
> +BUILDDIR=$(realpath -m "${BUILDDIR}")
> +LOGDIR=$(realpath -m "${LOGDIR}")
> +if [ "${do_localsrc}" = "y" ]; then
> +    XENDIR=$(realpath -m "${XENDIR}")
> +fi
> +
> +# Check that we have all the layers we need
> +for f in ${build_layerlist}; do
> +    if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
> +        echo "Layer ${f} missing in ${YOCTODIR}"
> +        exit 1
> +    fi
> +done
> +
> +for f in ${TARGETLIST}; do
> +    if [ "${do_clean}" = "y" ]; then
> +        run_task project_clean ${f}
> +    fi
> +    if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then
> +        run_task project_create ${f}
> +    fi
> +    if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then
> +        if [ "${do_build}" = "y" ]; then
> +            run_task project_build ${f}
> +        fi
> +        if [ "${do_run}" = "y" ]; then
> +            run_task project_run ${f}
> +        fi
> +
> +    fi
> +done
> +
> +print_progress "Build Complete (${build_result} errors)"
> +echo
> +exit ${build_result}
> +
> diff --git a/automation/build/yocto/yocto.dockerfile.in b/automation/build/yocto/yocto.dockerfile.in
> new file mode 100644
> index 000000000000..5350bb2b87b7
> --- /dev/null
> +++ b/automation/build/yocto/yocto.dockerfile.in
> @@ -0,0 +1,114 @@
> +# Docker file to create an environment to build yocto with virtualization
> +#
> +# Arguments that can be passed during image creation using --build-arg:
> +# "host_uid=$(id -u)": to use current user uid for build user in the image
> +# "host_gid=$(id -g)": to use current user gid for build user in the image
> +# "ubuntu_version=VERS": to select the ubuntu version number
Is is the case, that this dockerfile and the packages installed will work on any
version of ubuntu we will pass here? If not, maybe we should just stick to 22.04 and
not give the user the opportunity to change this.

> +
> +# Use standard ubuntu minimal
> +ARG ubuntu_version=22.04
> +From ##DOCKERPLAT##ubuntu:$ubuntu_version AS base
> +LABEL maintainer.name="The Xen Project " \
> +      maintainer.email="xen-devel@lists.xenproject.org"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +
> +# Install minimal ubuntu requirements for yocto and other tools we need
> +# See https://docs.yoctoproject.org/4.0.1/brief-yoctoprojectqs/index.html#build-host-packages
> +RUN apt-get update && \
> +    apt-get --quiet --yes install \
> +        gawk \
> +        wget \
> +        git \
> +        diffstat \
> +        unzip \
> +        texinfo \
> +        gcc \
> +        build-essential \
> +        chrpath \
> +        socat \
> +        cpio \
> +        python3 \
> +        python3-pip \
> +        python3-pexpect \
> +        xz-utils \
> +        debianutils \
> +        iputils-ping \
> +        python3-git \
> +        python3-jinja2 \
> +        libegl1-mesa \
> +        libsdl1.2-dev \
> +        python3-subunit \
> +        mesa-common-dev \
> +        zstd \
> +        liblz4-tool \
> +        file \
> +        vim \
> +        bison \
> +        expect \
> +        locales \
> +        liblz4-tool \
> +        zstd \
> +        openssl \
> +        libssl3 \
> +        ca-certificates \
> +        && \
> +        apt-get autoremove -y && \
> +        apt-get clean && \
> +        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
> +
> +# Use bash as shell
> +RUN rm /bin/sh && ln -s bash /bin/sh
> +
> +# Fix local for yocto
> +RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
> +    LANG=en_US.UTF-8
> +ENV LANG en_US.UTF-8
> +ENV LC_ALL en_US.UTF-8
> +
> +# Create a user for the build (we don't want to build as root)
> +ENV USER_NAME docker-build
> +ARG host_uid=1000
> +ARG host_gid=1000
> +RUN groupadd -g $host_gid $USER_NAME && \
> +    useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
> +
> +# Switch to our user instead of root and start in its home
> +USER $USER_NAME
> +WORKDIR /home/$USER_NAME
> +
> +# Create needed directories
> +RUN mkdir -p /home/$USER_NAME/yocto-layers \
> +             /home/$USER_NAME/yocto-cache \
> +             /home/$USER_NAME/logs \
> +             /home/$USER_NAME/bin \
> +             /home/$USER_NAME/xen && \
> +    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
> +
> +# clone yocto repositories we need
> +RUN for rep in \
> +                https://github.com/openembedded/meta-openembedded \
> +                https://git.yoctoproject.org/poky \
> +                https://git.yoctoproject.org/meta-virtualization \
> +            ; do \
> +        git -C /home/$USER_NAME/yocto-layers \
> +            clone -b ##YOCTOVERSION## --single-branch $rep; \
> +    done
> +
> +# The builder stage is building an initial cache state that we include in the
> +# final image
> +From base AS builder
> +
> +# This step can take one to several hours depending on your download bandwith
> +# and the speed of your computer
> +COPY ./build-yocto.sh /
> +RUN /build-yocto.sh --dump-log ##YOCTOTARGET##
> +
> +From base
> +
> +# Only copy the cache status
> +COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
> +
> +LABEL maintainer.name="The Xen Project " \
> +      maintainer.email="xen-devel@lists.xenproject.org"
> +
> diff --git a/automation/build/yocto/yocto.inc b/automation/build/yocto/yocto.inc
> new file mode 100644
> index 000000000000..04076bc8d174
> --- /dev/null
> +++ b/automation/build/yocto/yocto.inc
> @@ -0,0 +1,41 @@
> +# This makefile generates the docker files for Yocto builds
> +# The containers for the current architecture are the one built using make all
> +# To build containers for a different architecture, you need to call make for
> +# the image you want explicitely
Could you please add a dot at the end of each sentence above. This will improve readability.

> +# The containers are named this way:
> +# YOCTOVERSION-TARGET for x86_64 hosts
> +# YOCTOVERSION-TARGET-arm64v8 for arm64 hosts
> +
> +# Yocto versions we are currently using
> +YOCTO_VERSION = kirkstone
> +
> +# Yocto BSPs we want to build for
> +YOCTO_TARGETS = qemuarm64 qemuarm qemux86-64
> +
> +# Supported Host platforms (host architecture specific ones)
To avoid mismatch \wrt platform vs architecture I would stick to the latter one.
In the docker world, arm64v8 is an architecture whereas linux/arm64/v8 is a platform.

> +YOCTO_HOSTS = amd64 arm64v8
> +
> +# Architecture we want to use in gitlab CI (depends on runners arch)
> +CI_ARCH = arm64v8
> +
> +define GEN_DOCKER
> +# Make all is generating what we want in the CI
> +ifeq ($(CI_ARCH),$(3))
> +CONTAINERS += yocto/$(1)-$(2)$(4)
> +else
> +CONTAINERS_EXTRA += yocto/$(1)-$(2)$(4)
> +endif
> +
> +yocto/$(1)-$(2)$(4).dockerfile: yocto/yocto.dockerfile.in
> +       echo > $$@
> +       cat $$< | \
> +           sed -e "s,##YOCTOVERSION##,$(1),g" | \
> +           sed -e "s,##YOCTOTARGET##,$(2),g" | \
> +           sed -e "s,##DOCKERPLAT##,$(3)/,g" > $$@
> +
> +endef
> +
> +$(eval $(foreach vers,$(YOCTO_VERSION),\
> +    $(foreach tar,$(YOCTO_TARGETS),\
> +    $(foreach hst,$(YOCTO_HOSTS),\
> +    $(call GEN_DOCKER,$(vers),$(tar),$(hst),$(if $(filter amd64,$(hst)),,-$(hst)))))))
This could be aligned under first foreach. Also I think there is no need to try to shorten the version,target,host.

> --
> 2.25.1
> 

~Michal

Re: [PATCH v4 1/3] automation: Create Yocto docker images

[Bertrand Marquis]

Hi Michal,

> On 18 Nov 2022, at 08:53, Michal Orzel <michal.orzel@amd.com> wrote:
> 
> Hi Bertrand,
> 
> Just, some minor comments.
> 
> On 17/11/2022 10:39, Bertrand Marquis wrote:
>> 
>> 
>> Add containers suitable to run yocto kirkstone build based on ubuntu
>> 22.04. It contains all packages required by Yocto and a checkout of the
>> layers required to build Xen with Yocto.
>> 
>> Add a generic docker image template to be used to automatically generate
>> docker files for different configurations:
>> - specific yocto version
>> - different targets (qemu arm, arm64 and x86)
>> - different host platforms (x86 or arm64)
>> 
>> During a call to 'make all', only the images for the current host
>> platform will be generated.
> This looks like a stale comment. The reason being, in the new version, by default
> we build the images for the architectures expected by the CI and its runners
> to match the current workflow.

Right I will fix the commit message.

> 
>> If needed, images for an other host platform can be generated manually
>> by calling the right make target (see make help).
>> 
>> Add a build script to build and run xen on qemu using Yocto.
>> The script supports arm32, arm64 and x86_64 and checks that dom0 is
>> properly booting. At this stage this does not run any guest on top of
>> dom0. The script is to be executed in one of the docker images to build
>> and run a system using a Xen source tree.
>> 
>> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>> ---
>> Changes in v4:
>> - Rework the system to have one dockerfile template from which make will
>> generate the required dockerfiles for the wanted configuration
> That is great.

Thanks :-)
> 
>> - add support for different host architectures
>> - Merge the generation system into one single dockerfile
>> - Merge patches 1 and 2 in a single patch
>> - Introduce CONTAINERS_EXTRA to have extra containers not built by
>> default (for those not used by CI but useful to users)
>> Changes in v3:
>> - limit number of jobs in yocto by default to 8 and add --num-jobs
>> option to the script to set a custom number of jobs
>> - do not copy anymore the build-yocto.sh script inside the main image so
>> that the current one in the repository is used when running
>> Changes in v2:
>> - add a --dump-log command line option to build-yocto.sh script to dump
>> the logs if an error occurs.
>> Changes in v1:
>> - add --image command line argument to build-yocto.sh to allow building
>> something different than xen-image-minimal.
>> - modify dockerfile to have one layer per line and make it easier to add
>> other. I kept the for loop to keep the number of docker steps lower
>> - update commit message to warn that no guest are tested.
>> - fix build-yocto script to properly return with an error if expect
>> script ends up in timeout or EOF.
>> ---
>> automation/build/Makefile                  |  14 +-
>> automation/build/yocto/build-yocto.sh      | 349 +++++++++++++++++++++
>> automation/build/yocto/yocto.dockerfile.in | 114 +++++++
>> automation/build/yocto/yocto.inc           |  41 +++
>> 4 files changed, 516 insertions(+), 2 deletions(-)
>> create mode 100755 automation/build/yocto/build-yocto.sh
>> create mode 100644 automation/build/yocto/yocto.dockerfile.in
>> create mode 100644 automation/build/yocto/yocto.inc
>> 
>> diff --git a/automation/build/Makefile b/automation/build/Makefile
>> index a4b2b85178cf..72a5335baec1 100644
>> --- a/automation/build/Makefile
>> +++ b/automation/build/Makefile
>> @@ -1,13 +1,18 @@
>> 
>> # the base of where these containers will appear
>> REGISTRY := registry.gitlab.com/xen-project/xen
>> -CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
>> +CONTAINERS = $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard */*.dockerfile)))
>> +CONTAINERS_EXTRA =
>> DOCKER_CMD ?= docker
>> 
>> +include yocto/yocto.inc
>> +
>> help:
>>        @echo "Builds containers for building Xen based on different distros"
>>        @echo "To build one run 'make DISTRO/VERSION'. Available containers:"
>>        @$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
>> +       @echo "Extra containers (not built using make all):"
>> +       @$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo ${file};)
>>        @echo "To push container builds, set the env var PUSH"
>> 
>> %: %.dockerfile ## Builds containers
>> @@ -16,5 +21,10 @@ help:
>>                $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
>>        fi
>> 
>> -.PHONY: all
>> +.PHONY: all clean
>> all: $(CONTAINERS)
>> +
>> +# Remove generated dockerfiles for yocto
>> +clean:
>> +       rm -f yocto/*.dockerfiles
> Are these files needed after make is completed?
> If not, maybe to avoid having some untracked files in the tree, they could be removed after make is done?

I do not agree here, on might want to inspect those files to check.
Cleaning during make clean is more standard.

Also those are intermediate files so make should/could clean them automatically.
I will make a try to flag them intermediate if not handled automatically already but I will still keep this here.

> 
>> +
>> diff --git a/automation/build/yocto/build-yocto.sh b/automation/build/yocto/build-yocto.sh
>> new file mode 100755
>> index 000000000000..d0c93dfaffe0
>> --- /dev/null
>> +++ b/automation/build/yocto/build-yocto.sh
>> @@ -0,0 +1,349 @@
>> +#!/bin/bash
>> +#
>> +# Yocto meta virtualization build and run script
>> +#
>> +# This script is building Yocto xen-image-minimal for qemu targets and run
>> +# them using runqemu inside yocto to check that dom0 is booting properly
> Missing dot at the end of a sentence.

Will fix

> 
>> +# The build is using a local xen source tree so that specific patches can be
>> +# tested.
>> +# In order to optimize the build time, a build cache is used so that only xen
>> +# packages and its dependencies are rebuilt (qemu and final image mainly).
>> +#
>> +
>> +# Directories
>> +YOCTODIR="$HOME/yocto-layers"
>> +CACHEDIR="$HOME/yocto-cache"
>> +LOGDIR="$HOME/logs"
>> +XENDIR="$HOME/xen"
>> +BUILDDIR="$HOME/build"
>> +
>> +# what yocto bsp we support
>> +TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
>> +VERBOSE="n"
>> +TARGETLIST=""
>> +BUILDJOBS="8"
>> +
>> +# actions to do
>> +do_clean="n"
>> +do_build="y"
>> +do_run="y"
>> +do_localsrc="n"
>> +do_dump="n"
>> +build_result=0
>> +
>> +# layers to include in the project
>> +build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
>> +                 meta-openembedded/meta-oe meta-openembedded/meta-python \
>> +                 meta-openembedded/meta-filesystems \
>> +                 meta-openembedded/meta-networking meta-virtualization"
>> +
>> +# yocto image to build
>> +build_image="xen-image-minimal"
>> +
>> +function print_progress() {
>> +    echo -n "$(date +%T) $*"
>> +}
>> +
>> +function run_task() {
>> +    local task_name="$1"
>> +    local task_target="$2"
>> +
>> +    task_log="${task_name//project_}-${task_target}"
>> +
>> +    mkdir -p "${LOGDIR}"
>> +    print_progress
>> +    echo -n "${task_name//project_} ${task_target}: "
>> +    if [ "${VERBOSE}" = "n" ]; then
>> +        $@ > "${LOGDIR}/${task_log}.log" 2>&1
>> +    else
>> +        $@ 2>&1 | tee "${LOGDIR}/${task_log}.log"
>> +    fi
>> +
>> +    if [ ${?} -ne 0 ]; then
>> +        echo "Error"
>> +        build_result=$((build_result+1))
>> +        if [ "${do_dump}" = "y" ]; then
>> +            echo
>> +            echo "############ LOGS-START ############"
>> +            cat "${LOGDIR}/${task_log}.log"
>> +            echo "############  LOGS-END  ############"
>> +            echo
>> +        fi
>> +        return 1
>> +    else
>> +        echo "OK"
>> +        return 0
>> +    fi
>> +}
>> +
>> +function project_create() {
>> +    target="${1:?}"
>> +    destdir="${BUILDDIR}/${target}"
>> +
>> +    (
>> +        # init yocto project
>> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
>> +
>> +        # add needed layers
>> +        for layer in ${build_layerlist}; do
>> +            bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1
>> +        done
>> +    ) || return 1
>> +
>> +    # customize project configuration
>> +    cat <<EOF >> "${destdir}/conf/local.conf"
>> +# Yocto BSP
>> +MACHINE = "${target}"
>> +
>> +# Use local cache to reuse previous builds results
>> +SSTATE_DIR = "${CACHEDIR}/sstate-cache"
>> +DL_DIR = "${CACHEDIR}/downloads"
>> +
>> +# Enable xen and virtualization
>> +DISTRO_FEATURES = " virtualization xen ipv4"
>> +
>> +# Speed up run by not generating ssh host keys
>> +IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
>> +
>> +# Save some disk space
>> +INHERIT += "rm_work"
>> +
>> +# Reduce number of jobs
>> +BB_NUMBER_THREADS="${BUILDJOBS}"
>> +
>> +EOF
>> +
>> +    if [ "${do_localsrc}" = "y" ]; then
>> +        XENVERS=$(grep -e "^XEN_REL" \
>> +            "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
>> +            2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
>> +            | sed -e "s/.*=//" | sort -V | tail -n 1)
>> +
>> +        XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
>> +        XENSUB=$(basename "$(realpath -m "${XENDIR}")")
>> +
>> +        cat <<EOF >> "${destdir}/conf/local.conf"
>> +# Use local sources for xen and xen-tools
>> +FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
>> +FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
>> +
>> +SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
>> +SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
>> +
>> +PREFERRED_VERSION:pn-xen = "${XENVERS}%"
>> +PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
>> +
>> +S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
>> +S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
>> +
>> +SRCREV:pn-xen = "\${AUTOREV}"
>> +SRCREV:pn-xen-tools = "\${AUTOREV}"
>> +
>> +SRCPV:pn-xen = "1"
>> +SRCPV:pn-xen-tools = "1"
>> +
>> +# Disable all QA errors as the recipe is not up to date with changes in Xen
>> +# when we use local sources
>> +ERROR_QA:pn-xen = "arch"
>> +ERROR_QA:pn-xen-tools = "arch"
>> +
>> +EOF
>> +    fi
>> +}
>> +
>> +function project_build() {
>> +    target="${1:?}"
>> +    destdir="${BUILDDIR}/${target}"
>> +
>> +    (
>> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
>> +
>> +        bitbake "${build_image}" || exit 1
>> +    ) || return 1
>> +}
>> +
>> +function project_clean() {
>> +    target="${1:?}"
>> +    destdir="${BUILDDIR}/${target}"
>> +
>> +    rm -rf "${destdir}"
>> +}
>> +
>> +function project_run() {
>> +    target="${1:?}"
>> +    destdir="${BUILDDIR}/${target}"
>> +    (
>> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null 2>&1
>> +
>> +        /usr/bin/expect <<EOF
>> +set timeout 100
>> +spawn bash -c "runqemu serialstdio nographic slirp"
>> +
>> +expect_after {
>> +    -re "(.*)\r" {
>> +        exp_continue
>> +    }
>> +    timeout {send_user "ERROR-Timeout!\n"; exit 1}
>> +    eof {send_user "ERROR-EOF!\n"; exit 1}
>> +}
>> +
>> +# wait initial login
>> +expect -re ".* login: "
>> +send "root\r"
>> +expect -re "root@.*# "
>> +
>> +EOF
>> +    exit $?
>> +    ) || return 1
>> +}
>> +
>> +function help() {
>> +    cat <<EOF
>> +Usage: ${0} [TARGET1] [TARGET2]
>> +
>> +This script is build the yocto xen-image-minimal for different qemu targets
>> +and is running it after.
>> +Without any target specified, all supported targets are done.
>> +
>> +Options:
>> +  -h, --help       Print this help
>> +  -v, --verbose    Verbose build
>> +  --list-target    List supported targets
>> +  --clean          Clean existing project before starting
>> +  --no-build       Do not build (to run an already built project)
>> +  --no-run         Do not run
>> +  --num-jobs=NUM   Define the number of parallel jobs in Yocto.
>> +                   Default: ${BUILDJOBS}
>> +  --dump-log       On error, dump the logs on the console
>> +  --image=IMG      Yocto image or package to build
>> +                   Default: xen-image-minimal
>> +  --xen-dir=DIR    path to xen hypervisor source tree
>> +                   if not provide, normal yocto version of xen is built
>> +                   Default: ${XENDIR}
>> +  --out-dir=DIR    directory where to create the projectss
>> +                   Default: ${BUILDDIR}
>> +  --log-dir=DIR    directory to store logs
>> +                   Default: ${LOGDIR}
>> +  --cache-dir=DIR  directory where to take and store build cache
>> +                   Default: ${CACHEDIR}
>> +  --layer-dir=DIR  directory containing the checkout of yocto layers
>> +                   Default: ${YOCTODIR}
>> +EOF
>> +}
>> +
>> +for OPTION in "$@"
>> +do
>> +    case ${OPTION} in
>> +        -h|--help)
>> +            help
>> +            exit 0
>> +            ;;
>> +        -v|--verbose)
>> +            VERBOSE="y"
>> +            ;;
>> +        --list-targets)
>> +            echo "${TARGET_SUPPORTED}"
>> +            exit 0
>> +            ;;
>> +        --clean)
>> +            do_clean="y"
>> +            ;;
>> +        --no-build)
>> +            do_build="n"
>> +            ;;
>> +        --no-run)
>> +            do_run="n"
>> +            ;;
>> +        --dump-log)
>> +            do_dump="y"
>> +            ;;
>> +        --num-jobs=*)
>> +            BUILDJOBS="${OPTION#*=}"
>> +            ;;
>> +        --image=*)
>> +            build_image="${OPTION#*=}"
>> +            ;;
>> +        --xen-dir=*)
>> +            XENDIR="${OPTION#*=}"
>> +            if [ ! -e "${XENDIR}/xen/Makefile" ]; then
>> +                echo "No Xen source tree in ${XENDIR}"
>> +                exit 1
>> +            fi
>> +            do_localsrc="y"
>> +            ;;
>> +        --out-dir=*)
>> +            BUILDDIR="${OPTION#*=}"
>> +            ;;
>> +        --log-dir=*)
>> +            LOGDIR="${OPTION#*=}"
>> +            ;;
>> +        --cache-dir=*)
>> +            CACHEDIR="${OPTION#*=}"
>> +            ;;
>> +        --layer-dir=*)
>> +            YOCTODIR="${OPTION#*=}"
>> +            ;;
>> +        --*)
>> +            echo "Invalid option ${OPTION}"
>> +            help
>> +            exit 1
>> +            ;;
>> +        *)
>> +            if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
>> +                TARGETLIST="${TARGETLIST} ${OPTION}"
>> +            else
>> +                echo "Unsupported target ${OPTION}"
>> +                exit 1
>> +            fi
>> +            ;;
>> +    esac
>> +done
>> +
>> +# if no target is specified build all targets
>> +if [ -z "${TARGETLIST}" ]; then
>> +    TARGETLIST="${TARGET_SUPPORTED}"
>> +fi
>> +
>> +mkdir -p "${CACHEDIR}"
>> +mkdir -p "${LOGDIR}"
>> +mkdir -p "${BUILDDIR}"
>> +
>> +# Make sure we have an absolute path
>> +YOCTODIR=$(realpath -m "${YOCTODIR}")
>> +CACHEDIR=$(realpath -m "${CACHEDIR}")
>> +BUILDDIR=$(realpath -m "${BUILDDIR}")
>> +LOGDIR=$(realpath -m "${LOGDIR}")
>> +if [ "${do_localsrc}" = "y" ]; then
>> +    XENDIR=$(realpath -m "${XENDIR}")
>> +fi
>> +
>> +# Check that we have all the layers we need
>> +for f in ${build_layerlist}; do
>> +    if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
>> +        echo "Layer ${f} missing in ${YOCTODIR}"
>> +        exit 1
>> +    fi
>> +done
>> +
>> +for f in ${TARGETLIST}; do
>> +    if [ "${do_clean}" = "y" ]; then
>> +        run_task project_clean ${f}
>> +    fi
>> +    if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then
>> +        run_task project_create ${f}
>> +    fi
>> +    if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then
>> +        if [ "${do_build}" = "y" ]; then
>> +            run_task project_build ${f}
>> +        fi
>> +        if [ "${do_run}" = "y" ]; then
>> +            run_task project_run ${f}
>> +        fi
>> +
>> +    fi
>> +done
>> +
>> +print_progress "Build Complete (${build_result} errors)"
>> +echo
>> +exit ${build_result}
>> +
>> diff --git a/automation/build/yocto/yocto.dockerfile.in b/automation/build/yocto/yocto.dockerfile.in
>> new file mode 100644
>> index 000000000000..5350bb2b87b7
>> --- /dev/null
>> +++ b/automation/build/yocto/yocto.dockerfile.in
>> @@ -0,0 +1,114 @@
>> +# Docker file to create an environment to build yocto with virtualization
>> +#
>> +# Arguments that can be passed during image creation using --build-arg:
>> +# "host_uid=$(id -u)": to use current user uid for build user in the image
>> +# "host_gid=$(id -g)": to use current user gid for build user in the image
>> +# "ubuntu_version=VERS": to select the ubuntu version number
> Is is the case, that this dockerfile and the packages installed will work on any
> version of ubuntu we will pass here? If not, maybe we should just stick to 22.04 and
> not give the user the opportunity to change this.

There is no support right now to build for a different version of ubuntu without
modifying the docker file. I kept that as argument here because the generated
file/container does not contain an ubuntu version in its name so changing the
value here in the future will make it clearer.

I am ok to remove the variable and directly use the value if you think this is necessary.

> 
>> +
>> +# Use standard ubuntu minimal
>> +ARG ubuntu_version=22.04
>> +From ##DOCKERPLAT##ubuntu:$ubuntu_version AS base
>> +LABEL maintainer.name="The Xen Project " \
>> +      maintainer.email="xen-devel@lists.xenproject.org"
>> +
>> +ENV DEBIAN_FRONTEND=noninteractive
>> +
>> +# Install minimal ubuntu requirements for yocto and other tools we need
>> +# See https://docs.yoctoproject.org/4.0.1/brief-yoctoprojectqs/index.html#build-host-packages
>> +RUN apt-get update && \
>> +    apt-get --quiet --yes install \
>> +        gawk \
>> +        wget \
>> +        git \
>> +        diffstat \
>> +        unzip \
>> +        texinfo \
>> +        gcc \
>> +        build-essential \
>> +        chrpath \
>> +        socat \
>> +        cpio \
>> +        python3 \
>> +        python3-pip \
>> +        python3-pexpect \
>> +        xz-utils \
>> +        debianutils \
>> +        iputils-ping \
>> +        python3-git \
>> +        python3-jinja2 \
>> +        libegl1-mesa \
>> +        libsdl1.2-dev \
>> +        python3-subunit \
>> +        mesa-common-dev \
>> +        zstd \
>> +        liblz4-tool \
>> +        file \
>> +        vim \
>> +        bison \
>> +        expect \
>> +        locales \
>> +        liblz4-tool \
>> +        zstd \
>> +        openssl \
>> +        libssl3 \
>> +        ca-certificates \
>> +        && \
>> +        apt-get autoremove -y && \
>> +        apt-get clean && \
>> +        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
>> +
>> +# Use bash as shell
>> +RUN rm /bin/sh && ln -s bash /bin/sh
>> +
>> +# Fix local for yocto
>> +RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
>> +    LANG=en_US.UTF-8
>> +ENV LANG en_US.UTF-8
>> +ENV LC_ALL en_US.UTF-8
>> +
>> +# Create a user for the build (we don't want to build as root)
>> +ENV USER_NAME docker-build
>> +ARG host_uid=1000
>> +ARG host_gid=1000
>> +RUN groupadd -g $host_gid $USER_NAME && \
>> +    useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
>> +
>> +# Switch to our user instead of root and start in its home
>> +USER $USER_NAME
>> +WORKDIR /home/$USER_NAME
>> +
>> +# Create needed directories
>> +RUN mkdir -p /home/$USER_NAME/yocto-layers \
>> +             /home/$USER_NAME/yocto-cache \
>> +             /home/$USER_NAME/logs \
>> +             /home/$USER_NAME/bin \
>> +             /home/$USER_NAME/xen && \
>> +    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
>> +
>> +# clone yocto repositories we need
>> +RUN for rep in \
>> +                https://github.com/openembedded/meta-openembedded \
>> +                https://git.yoctoproject.org/poky \
>> +                https://git.yoctoproject.org/meta-virtualization \
>> +            ; do \
>> +        git -C /home/$USER_NAME/yocto-layers \
>> +            clone -b ##YOCTOVERSION## --single-branch $rep; \
>> +    done
>> +
>> +# The builder stage is building an initial cache state that we include in the
>> +# final image
>> +From base AS builder
>> +
>> +# This step can take one to several hours depending on your download bandwith
>> +# and the speed of your computer
>> +COPY ./build-yocto.sh /
>> +RUN /build-yocto.sh --dump-log ##YOCTOTARGET##
>> +
>> +From base
>> +
>> +# Only copy the cache status
>> +COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
>> +
>> +LABEL maintainer.name="The Xen Project " \
>> +      maintainer.email="xen-devel@lists.xenproject.org"
>> +
>> diff --git a/automation/build/yocto/yocto.inc b/automation/build/yocto/yocto.inc
>> new file mode 100644
>> index 000000000000..04076bc8d174
>> --- /dev/null
>> +++ b/automation/build/yocto/yocto.inc
>> @@ -0,0 +1,41 @@
>> +# This makefile generates the docker files for Yocto builds
>> +# The containers for the current architecture are the one built using make all
>> +# To build containers for a different architecture, you need to call make for
>> +# the image you want explicitely
> Could you please add a dot at the end of each sentence above. This will improve readability.

Yes will do.

> 
>> +# The containers are named this way:
>> +# YOCTOVERSION-TARGET for x86_64 hosts
>> +# YOCTOVERSION-TARGET-arm64v8 for arm64 hosts
>> +
>> +# Yocto versions we are currently using
>> +YOCTO_VERSION = kirkstone
>> +
>> +# Yocto BSPs we want to build for
>> +YOCTO_TARGETS = qemuarm64 qemuarm qemux86-64
>> +
>> +# Supported Host platforms (host architecture specific ones)
> To avoid mismatch \wrt platform vs architecture I would stick to the latter one.
> In the docker world, arm64v8 is an architecture whereas linux/arm64/v8 is a platform.

So "yocto target” and “container platform” ?

> 
>> +YOCTO_HOSTS = amd64 arm64v8
>> +
>> +# Architecture we want to use in gitlab CI (depends on runners arch)
>> +CI_ARCH = arm64v8
>> +
>> +define GEN_DOCKER
>> +# Make all is generating what we want in the CI
>> +ifeq ($(CI_ARCH),$(3))
>> +CONTAINERS += yocto/$(1)-$(2)$(4)
>> +else
>> +CONTAINERS_EXTRA += yocto/$(1)-$(2)$(4)
>> +endif
>> +
>> +yocto/$(1)-$(2)$(4).dockerfile: yocto/yocto.dockerfile.in
>> +       echo > $$@
>> +       cat $$< | \
>> +           sed -e "s,##YOCTOVERSION##,$(1),g" | \
>> +           sed -e "s,##YOCTOTARGET##,$(2),g" | \
>> +           sed -e "s,##DOCKERPLAT##,$(3)/,g" > $$@
>> +
>> +endef
>> +
>> +$(eval $(foreach vers,$(YOCTO_VERSION),\
>> +    $(foreach tar,$(YOCTO_TARGETS),\
>> +    $(foreach hst,$(YOCTO_HOSTS),\
>> +    $(call GEN_DOCKER,$(vers),$(tar),$(hst),$(if $(filter amd64,$(hst)),,-$(hst)))))))
> This could be aligned under first foreach. Also I think there is no need to try to shorten the version,target,host.

This is something you requested me in order to have container names for yocto coherent
with other xen containers (ie only have the -arm64v8 suffix for arm).

Do you want me to revert this and have names containing the amd64 suffix ?

Thanks for the review

Cheers
Bertrand

> 
>> --
>> 2.25.1
>> 
> 
> ~Michal

Re: [PATCH v4 1/3] automation: Create Yocto docker images

[Michal Orzel]

Hi Bertrand,

On 18/11/2022 12:44, Bertrand Marquis wrote:
> 
> 
> Hi Michal,
> 
>> On 18 Nov 2022, at 08:53, Michal Orzel <michal.orzel@amd.com> wrote:
>>
>> Hi Bertrand,
>>
>> Just, some minor comments.
>>
>> On 17/11/2022 10:39, Bertrand Marquis wrote:
>>>
>>>
>>> Add containers suitable to run yocto kirkstone build based on ubuntu
>>> 22.04. It contains all packages required by Yocto and a checkout of the
>>> layers required to build Xen with Yocto.
>>>
>>> Add a generic docker image template to be used to automatically generate
>>> docker files for different configurations:
>>> - specific yocto version
>>> - different targets (qemu arm, arm64 and x86)
>>> - different host platforms (x86 or arm64)
>>>
>>> During a call to 'make all', only the images for the current host
>>> platform will be generated.
>> This looks like a stale comment. The reason being, in the new version, by default
>> we build the images for the architectures expected by the CI and its runners
>> to match the current workflow.
> 
> Right I will fix the commit message.
> 
>>
>>> If needed, images for an other host platform can be generated manually
>>> by calling the right make target (see make help).
>>>
>>> Add a build script to build and run xen on qemu using Yocto.
>>> The script supports arm32, arm64 and x86_64 and checks that dom0 is
>>> properly booting. At this stage this does not run any guest on top of
>>> dom0. The script is to be executed in one of the docker images to build
>>> and run a system using a Xen source tree.
>>>
>>> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>>> ---
>>> Changes in v4:
>>> - Rework the system to have one dockerfile template from which make will
>>> generate the required dockerfiles for the wanted configuration
>> That is great.
> 
> Thanks :-)
>>
>>> - add support for different host architectures
>>> - Merge the generation system into one single dockerfile
>>> - Merge patches 1 and 2 in a single patch
>>> - Introduce CONTAINERS_EXTRA to have extra containers not built by
>>> default (for those not used by CI but useful to users)
>>> Changes in v3:
>>> - limit number of jobs in yocto by default to 8 and add --num-jobs
>>> option to the script to set a custom number of jobs
>>> - do not copy anymore the build-yocto.sh script inside the main image so
>>> that the current one in the repository is used when running
>>> Changes in v2:
>>> - add a --dump-log command line option to build-yocto.sh script to dump
>>> the logs if an error occurs.
>>> Changes in v1:
>>> - add --image command line argument to build-yocto.sh to allow building
>>> something different than xen-image-minimal.
>>> - modify dockerfile to have one layer per line and make it easier to add
>>> other. I kept the for loop to keep the number of docker steps lower
>>> - update commit message to warn that no guest are tested.
>>> - fix build-yocto script to properly return with an error if expect
>>> script ends up in timeout or EOF.
>>> ---
>>> automation/build/Makefile                  |  14 +-
>>> automation/build/yocto/build-yocto.sh      | 349 +++++++++++++++++++++
>>> automation/build/yocto/yocto.dockerfile.in | 114 +++++++
>>> automation/build/yocto/yocto.inc           |  41 +++
>>> 4 files changed, 516 insertions(+), 2 deletions(-)
>>> create mode 100755 automation/build/yocto/build-yocto.sh
>>> create mode 100644 automation/build/yocto/yocto.dockerfile.in
>>> create mode 100644 automation/build/yocto/yocto.inc
>>>
>>> diff --git a/automation/build/Makefile b/automation/build/Makefile
>>> index a4b2b85178cf..72a5335baec1 100644
>>> --- a/automation/build/Makefile
>>> +++ b/automation/build/Makefile
>>> @@ -1,13 +1,18 @@
>>>
>>> # the base of where these containers will appear
>>> REGISTRY := registry.gitlab.com/xen-project/xen
>>> -CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
>>> +CONTAINERS = $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard */*.dockerfile)))
>>> +CONTAINERS_EXTRA =
>>> DOCKER_CMD ?= docker
>>>
>>> +include yocto/yocto.inc
>>> +
>>> help:
>>>        @echo "Builds containers for building Xen based on different distros"
>>>        @echo "To build one run 'make DISTRO/VERSION'. Available containers:"
>>>        @$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
>>> +       @echo "Extra containers (not built using make all):"
>>> +       @$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo ${file};)
>>>        @echo "To push container builds, set the env var PUSH"
>>>
>>> %: %.dockerfile ## Builds containers
>>> @@ -16,5 +21,10 @@ help:
>>>                $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
>>>        fi
>>>
>>> -.PHONY: all
>>> +.PHONY: all clean
>>> all: $(CONTAINERS)
>>> +
>>> +# Remove generated dockerfiles for yocto
>>> +clean:
>>> +       rm -f yocto/*.dockerfiles
>> Are these files needed after make is completed?
>> If not, maybe to avoid having some untracked files in the tree, they could be removed after make is done?
> 
> I do not agree here, on might want to inspect those files to check.
> Cleaning during make clean is more standard.
> 
> Also those are intermediate files so make should/could clean them automatically.
> I will make a try to flag them intermediate if not handled automatically already but I will still keep this here.
> 
It is ok for me. I just raised a concern :)

>>
>>> +
>>> diff --git a/automation/build/yocto/build-yocto.sh b/automation/build/yocto/build-yocto.sh
>>> new file mode 100755
>>> index 000000000000..d0c93dfaffe0
>>> --- /dev/null
>>> +++ b/automation/build/yocto/build-yocto.sh
>>> @@ -0,0 +1,349 @@
>>> +#!/bin/bash
>>> +#
>>> +# Yocto meta virtualization build and run script
>>> +#
>>> +# This script is building Yocto xen-image-minimal for qemu targets and run
>>> +# them using runqemu inside yocto to check that dom0 is booting properly
>> Missing dot at the end of a sentence.
> 
> Will fix
> 
>>
>>> +# The build is using a local xen source tree so that specific patches can be
>>> +# tested.
>>> +# In order to optimize the build time, a build cache is used so that only xen
>>> +# packages and its dependencies are rebuilt (qemu and final image mainly).
>>> +#
>>> +
>>> +# Directories
>>> +YOCTODIR="$HOME/yocto-layers"
>>> +CACHEDIR="$HOME/yocto-cache"
>>> +LOGDIR="$HOME/logs"
>>> +XENDIR="$HOME/xen"
>>> +BUILDDIR="$HOME/build"
>>> +
>>> +# what yocto bsp we support
>>> +TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
>>> +VERBOSE="n"
>>> +TARGETLIST=""
>>> +BUILDJOBS="8"
>>> +
>>> +# actions to do
>>> +do_clean="n"
>>> +do_build="y"
>>> +do_run="y"
>>> +do_localsrc="n"
>>> +do_dump="n"
>>> +build_result=0
>>> +
>>> +# layers to include in the project
>>> +build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
>>> +                 meta-openembedded/meta-oe meta-openembedded/meta-python \
>>> +                 meta-openembedded/meta-filesystems \
>>> +                 meta-openembedded/meta-networking meta-virtualization"
>>> +
>>> +# yocto image to build
>>> +build_image="xen-image-minimal"
>>> +
>>> +function print_progress() {
>>> +    echo -n "$(date +%T) $*"
>>> +}
>>> +
>>> +function run_task() {
>>> +    local task_name="$1"
>>> +    local task_target="$2"
>>> +
>>> +    task_log="${task_name//project_}-${task_target}"
>>> +
>>> +    mkdir -p "${LOGDIR}"
>>> +    print_progress
>>> +    echo -n "${task_name//project_} ${task_target}: "
>>> +    if [ "${VERBOSE}" = "n" ]; then
>>> +        $@ > "${LOGDIR}/${task_log}.log" 2>&1
>>> +    else
>>> +        $@ 2>&1 | tee "${LOGDIR}/${task_log}.log"
>>> +    fi
>>> +
>>> +    if [ ${?} -ne 0 ]; then
>>> +        echo "Error"
>>> +        build_result=$((build_result+1))
>>> +        if [ "${do_dump}" = "y" ]; then
>>> +            echo
>>> +            echo "############ LOGS-START ############"
>>> +            cat "${LOGDIR}/${task_log}.log"
>>> +            echo "############  LOGS-END  ############"
>>> +            echo
>>> +        fi
>>> +        return 1
>>> +    else
>>> +        echo "OK"
>>> +        return 0
>>> +    fi
>>> +}
>>> +
>>> +function project_create() {
>>> +    target="${1:?}"
>>> +    destdir="${BUILDDIR}/${target}"
>>> +
>>> +    (
>>> +        # init yocto project
>>> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
>>> +
>>> +        # add needed layers
>>> +        for layer in ${build_layerlist}; do
>>> +            bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1
>>> +        done
>>> +    ) || return 1
>>> +
>>> +    # customize project configuration
>>> +    cat <<EOF >> "${destdir}/conf/local.conf"
>>> +# Yocto BSP
>>> +MACHINE = "${target}"
>>> +
>>> +# Use local cache to reuse previous builds results
>>> +SSTATE_DIR = "${CACHEDIR}/sstate-cache"
>>> +DL_DIR = "${CACHEDIR}/downloads"
>>> +
>>> +# Enable xen and virtualization
>>> +DISTRO_FEATURES = " virtualization xen ipv4"
>>> +
>>> +# Speed up run by not generating ssh host keys
>>> +IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
>>> +
>>> +# Save some disk space
>>> +INHERIT += "rm_work"
>>> +
>>> +# Reduce number of jobs
>>> +BB_NUMBER_THREADS="${BUILDJOBS}"
>>> +
>>> +EOF
>>> +
>>> +    if [ "${do_localsrc}" = "y" ]; then
>>> +        XENVERS=$(grep -e "^XEN_REL" \
>>> +            "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
>>> +            2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
>>> +            | sed -e "s/.*=//" | sort -V | tail -n 1)
>>> +
>>> +        XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
>>> +        XENSUB=$(basename "$(realpath -m "${XENDIR}")")
>>> +
>>> +        cat <<EOF >> "${destdir}/conf/local.conf"
>>> +# Use local sources for xen and xen-tools
>>> +FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
>>> +FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
>>> +
>>> +SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
>>> +SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
>>> +
>>> +PREFERRED_VERSION:pn-xen = "${XENVERS}%"
>>> +PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
>>> +
>>> +S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
>>> +S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
>>> +
>>> +SRCREV:pn-xen = "\${AUTOREV}"
>>> +SRCREV:pn-xen-tools = "\${AUTOREV}"
>>> +
>>> +SRCPV:pn-xen = "1"
>>> +SRCPV:pn-xen-tools = "1"
>>> +
>>> +# Disable all QA errors as the recipe is not up to date with changes in Xen
>>> +# when we use local sources
>>> +ERROR_QA:pn-xen = "arch"
>>> +ERROR_QA:pn-xen-tools = "arch"
>>> +
>>> +EOF
>>> +    fi
>>> +}
>>> +
>>> +function project_build() {
>>> +    target="${1:?}"
>>> +    destdir="${BUILDDIR}/${target}"
>>> +
>>> +    (
>>> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
>>> +
>>> +        bitbake "${build_image}" || exit 1
>>> +    ) || return 1
>>> +}
>>> +
>>> +function project_clean() {
>>> +    target="${1:?}"
>>> +    destdir="${BUILDDIR}/${target}"
>>> +
>>> +    rm -rf "${destdir}"
>>> +}
>>> +
>>> +function project_run() {
>>> +    target="${1:?}"
>>> +    destdir="${BUILDDIR}/${target}"
>>> +    (
>>> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null 2>&1
>>> +
>>> +        /usr/bin/expect <<EOF
>>> +set timeout 100
>>> +spawn bash -c "runqemu serialstdio nographic slirp"
>>> +
>>> +expect_after {
>>> +    -re "(.*)\r" {
>>> +        exp_continue
>>> +    }
>>> +    timeout {send_user "ERROR-Timeout!\n"; exit 1}
>>> +    eof {send_user "ERROR-EOF!\n"; exit 1}
>>> +}
>>> +
>>> +# wait initial login
>>> +expect -re ".* login: "
>>> +send "root\r"
>>> +expect -re "root@.*# "
>>> +
>>> +EOF
>>> +    exit $?
>>> +    ) || return 1
>>> +}
>>> +
>>> +function help() {
>>> +    cat <<EOF
>>> +Usage: ${0} [TARGET1] [TARGET2]
>>> +
>>> +This script is build the yocto xen-image-minimal for different qemu targets
>>> +and is running it after.
>>> +Without any target specified, all supported targets are done.
>>> +
>>> +Options:
>>> +  -h, --help       Print this help
>>> +  -v, --verbose    Verbose build
>>> +  --list-target    List supported targets
>>> +  --clean          Clean existing project before starting
>>> +  --no-build       Do not build (to run an already built project)
>>> +  --no-run         Do not run
>>> +  --num-jobs=NUM   Define the number of parallel jobs in Yocto.
>>> +                   Default: ${BUILDJOBS}
>>> +  --dump-log       On error, dump the logs on the console
>>> +  --image=IMG      Yocto image or package to build
>>> +                   Default: xen-image-minimal
>>> +  --xen-dir=DIR    path to xen hypervisor source tree
>>> +                   if not provide, normal yocto version of xen is built
>>> +                   Default: ${XENDIR}
>>> +  --out-dir=DIR    directory where to create the projectss
>>> +                   Default: ${BUILDDIR}
>>> +  --log-dir=DIR    directory to store logs
>>> +                   Default: ${LOGDIR}
>>> +  --cache-dir=DIR  directory where to take and store build cache
>>> +                   Default: ${CACHEDIR}
>>> +  --layer-dir=DIR  directory containing the checkout of yocto layers
>>> +                   Default: ${YOCTODIR}
>>> +EOF
>>> +}
>>> +
>>> +for OPTION in "$@"
>>> +do
>>> +    case ${OPTION} in
>>> +        -h|--help)
>>> +            help
>>> +            exit 0
>>> +            ;;
>>> +        -v|--verbose)
>>> +            VERBOSE="y"
>>> +            ;;
>>> +        --list-targets)
>>> +            echo "${TARGET_SUPPORTED}"
>>> +            exit 0
>>> +            ;;
>>> +        --clean)
>>> +            do_clean="y"
>>> +            ;;
>>> +        --no-build)
>>> +            do_build="n"
>>> +            ;;
>>> +        --no-run)
>>> +            do_run="n"
>>> +            ;;
>>> +        --dump-log)
>>> +            do_dump="y"
>>> +            ;;
>>> +        --num-jobs=*)
>>> +            BUILDJOBS="${OPTION#*=}"
>>> +            ;;
>>> +        --image=*)
>>> +            build_image="${OPTION#*=}"
>>> +            ;;
>>> +        --xen-dir=*)
>>> +            XENDIR="${OPTION#*=}"
>>> +            if [ ! -e "${XENDIR}/xen/Makefile" ]; then
>>> +                echo "No Xen source tree in ${XENDIR}"
>>> +                exit 1
>>> +            fi
>>> +            do_localsrc="y"
>>> +            ;;
>>> +        --out-dir=*)
>>> +            BUILDDIR="${OPTION#*=}"
>>> +            ;;
>>> +        --log-dir=*)
>>> +            LOGDIR="${OPTION#*=}"
>>> +            ;;
>>> +        --cache-dir=*)
>>> +            CACHEDIR="${OPTION#*=}"
>>> +            ;;
>>> +        --layer-dir=*)
>>> +            YOCTODIR="${OPTION#*=}"
>>> +            ;;
>>> +        --*)
>>> +            echo "Invalid option ${OPTION}"
>>> +            help
>>> +            exit 1
>>> +            ;;
>>> +        *)
>>> +            if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
>>> +                TARGETLIST="${TARGETLIST} ${OPTION}"
>>> +            else
>>> +                echo "Unsupported target ${OPTION}"
>>> +                exit 1
>>> +            fi
>>> +            ;;
>>> +    esac
>>> +done
>>> +
>>> +# if no target is specified build all targets
>>> +if [ -z "${TARGETLIST}" ]; then
>>> +    TARGETLIST="${TARGET_SUPPORTED}"
>>> +fi
>>> +
>>> +mkdir -p "${CACHEDIR}"
>>> +mkdir -p "${LOGDIR}"
>>> +mkdir -p "${BUILDDIR}"
>>> +
>>> +# Make sure we have an absolute path
>>> +YOCTODIR=$(realpath -m "${YOCTODIR}")
>>> +CACHEDIR=$(realpath -m "${CACHEDIR}")
>>> +BUILDDIR=$(realpath -m "${BUILDDIR}")
>>> +LOGDIR=$(realpath -m "${LOGDIR}")
>>> +if [ "${do_localsrc}" = "y" ]; then
>>> +    XENDIR=$(realpath -m "${XENDIR}")
>>> +fi
>>> +
>>> +# Check that we have all the layers we need
>>> +for f in ${build_layerlist}; do
>>> +    if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
>>> +        echo "Layer ${f} missing in ${YOCTODIR}"
>>> +        exit 1
>>> +    fi
>>> +done
>>> +
>>> +for f in ${TARGETLIST}; do
>>> +    if [ "${do_clean}" = "y" ]; then
>>> +        run_task project_clean ${f}
>>> +    fi
>>> +    if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then
>>> +        run_task project_create ${f}
>>> +    fi
>>> +    if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then
>>> +        if [ "${do_build}" = "y" ]; then
>>> +            run_task project_build ${f}
>>> +        fi
>>> +        if [ "${do_run}" = "y" ]; then
>>> +            run_task project_run ${f}
>>> +        fi
>>> +
>>> +    fi
>>> +done
>>> +
>>> +print_progress "Build Complete (${build_result} errors)"
>>> +echo
>>> +exit ${build_result}
>>> +
>>> diff --git a/automation/build/yocto/yocto.dockerfile.in b/automation/build/yocto/yocto.dockerfile.in
>>> new file mode 100644
>>> index 000000000000..5350bb2b87b7
>>> --- /dev/null
>>> +++ b/automation/build/yocto/yocto.dockerfile.in
>>> @@ -0,0 +1,114 @@
>>> +# Docker file to create an environment to build yocto with virtualization
>>> +#
>>> +# Arguments that can be passed during image creation using --build-arg:
>>> +# "host_uid=$(id -u)": to use current user uid for build user in the image
>>> +# "host_gid=$(id -g)": to use current user gid for build user in the image
>>> +# "ubuntu_version=VERS": to select the ubuntu version number
>> Is is the case, that this dockerfile and the packages installed will work on any
>> version of ubuntu we will pass here? If not, maybe we should just stick to 22.04 and
>> not give the user the opportunity to change this.
> 
> There is no support right now to build for a different version of ubuntu without
> modifying the docker file. I kept that as argument here because the generated
> file/container does not contain an ubuntu version in its name so changing the
> value here in the future will make it clearer.
Ok.
> 
> I am ok to remove the variable and directly use the value if you think this is necessary.
No need.

> 
>>
>>> +
>>> +# Use standard ubuntu minimal
>>> +ARG ubuntu_version=22.04
>>> +From ##DOCKERPLAT##ubuntu:$ubuntu_version AS base
>>> +LABEL maintainer.name="The Xen Project " \
>>> +      maintainer.email="xen-devel@lists.xenproject.org"
>>> +
>>> +ENV DEBIAN_FRONTEND=noninteractive
>>> +
>>> +# Install minimal ubuntu requirements for yocto and other tools we need
>>> +# See https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.yoctoproject.org%2F4.0.1%2Fbrief-yoctoprojectqs%2Findex.html%23build-host-packages&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd5eca08f965d4fd1a99508dac95a41e7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638043686755292322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=D%2FAe%2FCUo2%2FWcvJAXjRW4vW9z4pwWcUEOhe1TdlozECw%3D&amp;reserved=0
>>> +RUN apt-get update && \
>>> +    apt-get --quiet --yes install \
>>> +        gawk \
>>> +        wget \
>>> +        git \
>>> +        diffstat \
>>> +        unzip \
>>> +        texinfo \
>>> +        gcc \
>>> +        build-essential \
>>> +        chrpath \
>>> +        socat \
>>> +        cpio \
>>> +        python3 \
>>> +        python3-pip \
>>> +        python3-pexpect \
>>> +        xz-utils \
>>> +        debianutils \
>>> +        iputils-ping \
>>> +        python3-git \
>>> +        python3-jinja2 \
>>> +        libegl1-mesa \
>>> +        libsdl1.2-dev \
>>> +        python3-subunit \
>>> +        mesa-common-dev \
>>> +        zstd \
>>> +        liblz4-tool \
>>> +        file \
>>> +        vim \
>>> +        bison \
>>> +        expect \
>>> +        locales \
>>> +        liblz4-tool \
>>> +        zstd \
>>> +        openssl \
>>> +        libssl3 \
>>> +        ca-certificates \
>>> +        && \
>>> +        apt-get autoremove -y && \
>>> +        apt-get clean && \
>>> +        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
>>> +
>>> +# Use bash as shell
>>> +RUN rm /bin/sh && ln -s bash /bin/sh
>>> +
>>> +# Fix local for yocto
>>> +RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
>>> +    LANG=en_US.UTF-8
>>> +ENV LANG en_US.UTF-8
>>> +ENV LC_ALL en_US.UTF-8
>>> +
>>> +# Create a user for the build (we don't want to build as root)
>>> +ENV USER_NAME docker-build
>>> +ARG host_uid=1000
>>> +ARG host_gid=1000
>>> +RUN groupadd -g $host_gid $USER_NAME && \
>>> +    useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
>>> +
>>> +# Switch to our user instead of root and start in its home
>>> +USER $USER_NAME
>>> +WORKDIR /home/$USER_NAME
>>> +
>>> +# Create needed directories
>>> +RUN mkdir -p /home/$USER_NAME/yocto-layers \
>>> +             /home/$USER_NAME/yocto-cache \
>>> +             /home/$USER_NAME/logs \
>>> +             /home/$USER_NAME/bin \
>>> +             /home/$USER_NAME/xen && \
>>> +    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
>>> +
>>> +# clone yocto repositories we need
>>> +RUN for rep in \
>>> +                https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenembedded%2Fmeta-openembedded&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd5eca08f965d4fd1a99508dac95a41e7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638043686755292322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=5IB%2BO9M0DU3D7hN7q9Ft9OivqfmTVKuOVKgFWuElUu4%3D&amp;reserved=0 \
>>> +                https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.yoctoproject.org%2Fpoky&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd5eca08f965d4fd1a99508dac95a41e7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638043686755448030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=fMRgTFNrjR5athu9kXSNXAL%2FeN2RwO%2Bu6CZxCbhesyU%3D&amp;reserved=0 \
>>> +                https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.yoctoproject.org%2Fmeta-virtualization&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd5eca08f965d4fd1a99508dac95a41e7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638043686755448030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=J6JQP3Ibn2WaD1cTtduC4T%2BbnNWlUHmFPXrLsIe1CSw%3D&amp;reserved=0 \
>>> +            ; do \
>>> +        git -C /home/$USER_NAME/yocto-layers \
>>> +            clone -b ##YOCTOVERSION## --single-branch $rep; \
>>> +    done
>>> +
>>> +# The builder stage is building an initial cache state that we include in the
>>> +# final image
>>> +From base AS builder
>>> +
>>> +# This step can take one to several hours depending on your download bandwith
>>> +# and the speed of your computer
>>> +COPY ./build-yocto.sh /
>>> +RUN /build-yocto.sh --dump-log ##YOCTOTARGET##
>>> +
>>> +From base
>>> +
>>> +# Only copy the cache status
>>> +COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
>>> +
>>> +LABEL maintainer.name="The Xen Project " \
>>> +      maintainer.email="xen-devel@lists.xenproject.org"
>>> +
>>> diff --git a/automation/build/yocto/yocto.inc b/automation/build/yocto/yocto.inc
>>> new file mode 100644
>>> index 000000000000..04076bc8d174
>>> --- /dev/null
>>> +++ b/automation/build/yocto/yocto.inc
>>> @@ -0,0 +1,41 @@
>>> +# This makefile generates the docker files for Yocto builds
>>> +# The containers for the current architecture are the one built using make all
>>> +# To build containers for a different architecture, you need to call make for
>>> +# the image you want explicitely
>> Could you please add a dot at the end of each sentence above. This will improve readability.
> 
> Yes will do.
> 
>>
>>> +# The containers are named this way:
>>> +# YOCTOVERSION-TARGET for x86_64 hosts
>>> +# YOCTOVERSION-TARGET-arm64v8 for arm64 hosts
>>> +
>>> +# Yocto versions we are currently using
>>> +YOCTO_VERSION = kirkstone
>>> +
>>> +# Yocto BSPs we want to build for
>>> +YOCTO_TARGETS = qemuarm64 qemuarm qemux86-64
>>> +
>>> +# Supported Host platforms (host architecture specific ones)
>> To avoid mismatch \wrt platform vs architecture I would stick to the latter one.
>> In the docker world, arm64v8 is an architecture whereas linux/arm64/v8 is a platform.
> 
> So "yocto target” and “container platform” ?
container architecture

> 
>>
>>> +YOCTO_HOSTS = amd64 arm64v8
>>> +
>>> +# Architecture we want to use in gitlab CI (depends on runners arch)
>>> +CI_ARCH = arm64v8
>>> +
>>> +define GEN_DOCKER
>>> +# Make all is generating what we want in the CI
>>> +ifeq ($(CI_ARCH),$(3))
>>> +CONTAINERS += yocto/$(1)-$(2)$(4)
>>> +else
>>> +CONTAINERS_EXTRA += yocto/$(1)-$(2)$(4)
>>> +endif
>>> +
>>> +yocto/$(1)-$(2)$(4).dockerfile: yocto/yocto.dockerfile.in
>>> +       echo > $$@
>>> +       cat $$< | \
>>> +           sed -e "s,##YOCTOVERSION##,$(1),g" | \
>>> +           sed -e "s,##YOCTOTARGET##,$(2),g" | \
>>> +           sed -e "s,##DOCKERPLAT##,$(3)/,g" > $$@
>>> +
>>> +endef
>>> +
>>> +$(eval $(foreach vers,$(YOCTO_VERSION),\
>>> +    $(foreach tar,$(YOCTO_TARGETS),\
>>> +    $(foreach hst,$(YOCTO_HOSTS),\
>>> +    $(call GEN_DOCKER,$(vers),$(tar),$(hst),$(if $(filter amd64,$(hst)),,-$(hst)))))))
>> This could be aligned under first foreach. Also I think there is no need to try to shorten the version,target,host.
> 
> This is something you requested me in order to have container names for yocto coherent
> with other xen containers (ie only have the -arm64v8 suffix for arm).
> 
> Do you want me to revert this and have names containing the amd64 suffix ?
This is not what I meant.
I just wanted to use the full words like target,host,version instead of tar,hst,vers
and for the "foreach" to be aligned.

> 
> Thanks for the review
> 
> Cheers
> Bertrand
> 
>>
>>> --
>>> 2.25.1
>>>
>>
>> ~Michal
> 
~Michal

Re: [PATCH v4 1/3] automation: Create Yocto docker images

[Bertrand Marquis]

Hi,

> On 18 Nov 2022, at 11:54, Michal Orzel <michal.orzel@amd.com> wrote:
> 
> Hi Bertrand,
> 
> On 18/11/2022 12:44, Bertrand Marquis wrote:
>> 
>> 
>> Hi Michal,
>> 
>>> On 18 Nov 2022, at 08:53, Michal Orzel <michal.orzel@amd.com> wrote:
>>> 
>>> Hi Bertrand,
>>> 
>>> Just, some minor comments.
>>> 
>>> On 17/11/2022 10:39, Bertrand Marquis wrote:
>>>> 
>>>> 
>>>> Add containers suitable to run yocto kirkstone build based on ubuntu
>>>> 22.04. It contains all packages required by Yocto and a checkout of the
>>>> layers required to build Xen with Yocto.
>>>> 
>>>> Add a generic docker image template to be used to automatically generate
>>>> docker files for different configurations:
>>>> - specific yocto version
>>>> - different targets (qemu arm, arm64 and x86)
>>>> - different host platforms (x86 or arm64)
>>>> 
>>>> During a call to 'make all', only the images for the current host
>>>> platform will be generated.
>>> This looks like a stale comment. The reason being, in the new version, by default
>>> we build the images for the architectures expected by the CI and its runners
>>> to match the current workflow.
>> 
>> Right I will fix the commit message.
>> 
>>> 
>>>> If needed, images for an other host platform can be generated manually
>>>> by calling the right make target (see make help).
>>>> 
>>>> Add a build script to build and run xen on qemu using Yocto.
>>>> The script supports arm32, arm64 and x86_64 and checks that dom0 is
>>>> properly booting. At this stage this does not run any guest on top of
>>>> dom0. The script is to be executed in one of the docker images to build
>>>> and run a system using a Xen source tree.
>>>> 
>>>> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>>>> ---
>>>> Changes in v4:
>>>> - Rework the system to have one dockerfile template from which make will
>>>> generate the required dockerfiles for the wanted configuration
>>> That is great.
>> 
>> Thanks :-)
>>> 
>>>> - add support for different host architectures
>>>> - Merge the generation system into one single dockerfile
>>>> - Merge patches 1 and 2 in a single patch
>>>> - Introduce CONTAINERS_EXTRA to have extra containers not built by
>>>> default (for those not used by CI but useful to users)
>>>> Changes in v3:
>>>> - limit number of jobs in yocto by default to 8 and add --num-jobs
>>>> option to the script to set a custom number of jobs
>>>> - do not copy anymore the build-yocto.sh script inside the main image so
>>>> that the current one in the repository is used when running
>>>> Changes in v2:
>>>> - add a --dump-log command line option to build-yocto.sh script to dump
>>>> the logs if an error occurs.
>>>> Changes in v1:
>>>> - add --image command line argument to build-yocto.sh to allow building
>>>> something different than xen-image-minimal.
>>>> - modify dockerfile to have one layer per line and make it easier to add
>>>> other. I kept the for loop to keep the number of docker steps lower
>>>> - update commit message to warn that no guest are tested.
>>>> - fix build-yocto script to properly return with an error if expect
>>>> script ends up in timeout or EOF.
>>>> ---
>>>> automation/build/Makefile                  |  14 +-
>>>> automation/build/yocto/build-yocto.sh      | 349 +++++++++++++++++++++
>>>> automation/build/yocto/yocto.dockerfile.in | 114 +++++++
>>>> automation/build/yocto/yocto.inc           |  41 +++
>>>> 4 files changed, 516 insertions(+), 2 deletions(-)
>>>> create mode 100755 automation/build/yocto/build-yocto.sh
>>>> create mode 100644 automation/build/yocto/yocto.dockerfile.in
>>>> create mode 100644 automation/build/yocto/yocto.inc
>>>> 
>>>> diff --git a/automation/build/Makefile b/automation/build/Makefile
>>>> index a4b2b85178cf..72a5335baec1 100644
>>>> --- a/automation/build/Makefile
>>>> +++ b/automation/build/Makefile
>>>> @@ -1,13 +1,18 @@
>>>> 
>>>> # the base of where these containers will appear
>>>> REGISTRY := registry.gitlab.com/xen-project/xen
>>>> -CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
>>>> +CONTAINERS = $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard */*.dockerfile)))
>>>> +CONTAINERS_EXTRA =
>>>> DOCKER_CMD ?= docker
>>>> 
>>>> +include yocto/yocto.inc
>>>> +
>>>> help:
>>>>       @echo "Builds containers for building Xen based on different distros"
>>>>       @echo "To build one run 'make DISTRO/VERSION'. Available containers:"
>>>>       @$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
>>>> +       @echo "Extra containers (not built using make all):"
>>>> +       @$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo ${file};)
>>>>       @echo "To push container builds, set the env var PUSH"
>>>> 
>>>> %: %.dockerfile ## Builds containers
>>>> @@ -16,5 +21,10 @@ help:
>>>>               $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
>>>>       fi
>>>> 
>>>> -.PHONY: all
>>>> +.PHONY: all clean
>>>> all: $(CONTAINERS)
>>>> +
>>>> +# Remove generated dockerfiles for yocto
>>>> +clean:
>>>> +       rm -f yocto/*.dockerfiles
>>> Are these files needed after make is completed?
>>> If not, maybe to avoid having some untracked files in the tree, they could be removed after make is done?
>> 
>> I do not agree here, on might want to inspect those files to check.
>> Cleaning during make clean is more standard.
>> 
>> Also those are intermediate files so make should/could clean them automatically.
>> I will make a try to flag them intermediate if not handled automatically already but I will still keep this here.
>> 
> It is ok for me. I just raised a concern :)
> 
>>> 
>>>> +
>>>> diff --git a/automation/build/yocto/build-yocto.sh b/automation/build/yocto/build-yocto.sh
>>>> new file mode 100755
>>>> index 000000000000..d0c93dfaffe0
>>>> --- /dev/null
>>>> +++ b/automation/build/yocto/build-yocto.sh
>>>> @@ -0,0 +1,349 @@
>>>> +#!/bin/bash
>>>> +#
>>>> +# Yocto meta virtualization build and run script
>>>> +#
>>>> +# This script is building Yocto xen-image-minimal for qemu targets and run
>>>> +# them using runqemu inside yocto to check that dom0 is booting properly
>>> Missing dot at the end of a sentence.
>> 
>> Will fix
>> 
>>> 
>>>> +# The build is using a local xen source tree so that specific patches can be
>>>> +# tested.
>>>> +# In order to optimize the build time, a build cache is used so that only xen
>>>> +# packages and its dependencies are rebuilt (qemu and final image mainly).
>>>> +#
>>>> +
>>>> +# Directories
>>>> +YOCTODIR="$HOME/yocto-layers"
>>>> +CACHEDIR="$HOME/yocto-cache"
>>>> +LOGDIR="$HOME/logs"
>>>> +XENDIR="$HOME/xen"
>>>> +BUILDDIR="$HOME/build"
>>>> +
>>>> +# what yocto bsp we support
>>>> +TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
>>>> +VERBOSE="n"
>>>> +TARGETLIST=""
>>>> +BUILDJOBS="8"
>>>> +
>>>> +# actions to do
>>>> +do_clean="n"
>>>> +do_build="y"
>>>> +do_run="y"
>>>> +do_localsrc="n"
>>>> +do_dump="n"
>>>> +build_result=0
>>>> +
>>>> +# layers to include in the project
>>>> +build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
>>>> +                 meta-openembedded/meta-oe meta-openembedded/meta-python \
>>>> +                 meta-openembedded/meta-filesystems \
>>>> +                 meta-openembedded/meta-networking meta-virtualization"
>>>> +
>>>> +# yocto image to build
>>>> +build_image="xen-image-minimal"
>>>> +
>>>> +function print_progress() {
>>>> +    echo -n "$(date +%T) $*"
>>>> +}
>>>> +
>>>> +function run_task() {
>>>> +    local task_name="$1"
>>>> +    local task_target="$2"
>>>> +
>>>> +    task_log="${task_name//project_}-${task_target}"
>>>> +
>>>> +    mkdir -p "${LOGDIR}"
>>>> +    print_progress
>>>> +    echo -n "${task_name//project_} ${task_target}: "
>>>> +    if [ "${VERBOSE}" = "n" ]; then
>>>> +        $@ > "${LOGDIR}/${task_log}.log" 2>&1
>>>> +    else
>>>> +        $@ 2>&1 | tee "${LOGDIR}/${task_log}.log"
>>>> +    fi
>>>> +
>>>> +    if [ ${?} -ne 0 ]; then
>>>> +        echo "Error"
>>>> +        build_result=$((build_result+1))
>>>> +        if [ "${do_dump}" = "y" ]; then
>>>> +            echo
>>>> +            echo "############ LOGS-START ############"
>>>> +            cat "${LOGDIR}/${task_log}.log"
>>>> +            echo "############  LOGS-END  ############"
>>>> +            echo
>>>> +        fi
>>>> +        return 1
>>>> +    else
>>>> +        echo "OK"
>>>> +        return 0
>>>> +    fi
>>>> +}
>>>> +
>>>> +function project_create() {
>>>> +    target="${1:?}"
>>>> +    destdir="${BUILDDIR}/${target}"
>>>> +
>>>> +    (
>>>> +        # init yocto project
>>>> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
>>>> +
>>>> +        # add needed layers
>>>> +        for layer in ${build_layerlist}; do
>>>> +            bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1
>>>> +        done
>>>> +    ) || return 1
>>>> +
>>>> +    # customize project configuration
>>>> +    cat <<EOF >> "${destdir}/conf/local.conf"
>>>> +# Yocto BSP
>>>> +MACHINE = "${target}"
>>>> +
>>>> +# Use local cache to reuse previous builds results
>>>> +SSTATE_DIR = "${CACHEDIR}/sstate-cache"
>>>> +DL_DIR = "${CACHEDIR}/downloads"
>>>> +
>>>> +# Enable xen and virtualization
>>>> +DISTRO_FEATURES = " virtualization xen ipv4"
>>>> +
>>>> +# Speed up run by not generating ssh host keys
>>>> +IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
>>>> +
>>>> +# Save some disk space
>>>> +INHERIT += "rm_work"
>>>> +
>>>> +# Reduce number of jobs
>>>> +BB_NUMBER_THREADS="${BUILDJOBS}"
>>>> +
>>>> +EOF
>>>> +
>>>> +    if [ "${do_localsrc}" = "y" ]; then
>>>> +        XENVERS=$(grep -e "^XEN_REL" \
>>>> +            "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
>>>> +            2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
>>>> +            | sed -e "s/.*=//" | sort -V | tail -n 1)
>>>> +
>>>> +        XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
>>>> +        XENSUB=$(basename "$(realpath -m "${XENDIR}")")
>>>> +
>>>> +        cat <<EOF >> "${destdir}/conf/local.conf"
>>>> +# Use local sources for xen and xen-tools
>>>> +FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
>>>> +FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
>>>> +
>>>> +SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
>>>> +SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
>>>> +
>>>> +PREFERRED_VERSION:pn-xen = "${XENVERS}%"
>>>> +PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
>>>> +
>>>> +S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
>>>> +S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
>>>> +
>>>> +SRCREV:pn-xen = "\${AUTOREV}"
>>>> +SRCREV:pn-xen-tools = "\${AUTOREV}"
>>>> +
>>>> +SRCPV:pn-xen = "1"
>>>> +SRCPV:pn-xen-tools = "1"
>>>> +
>>>> +# Disable all QA errors as the recipe is not up to date with changes in Xen
>>>> +# when we use local sources
>>>> +ERROR_QA:pn-xen = "arch"
>>>> +ERROR_QA:pn-xen-tools = "arch"
>>>> +
>>>> +EOF
>>>> +    fi
>>>> +}
>>>> +
>>>> +function project_build() {
>>>> +    target="${1:?}"
>>>> +    destdir="${BUILDDIR}/${target}"
>>>> +
>>>> +    (
>>>> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
>>>> +
>>>> +        bitbake "${build_image}" || exit 1
>>>> +    ) || return 1
>>>> +}
>>>> +
>>>> +function project_clean() {
>>>> +    target="${1:?}"
>>>> +    destdir="${BUILDDIR}/${target}"
>>>> +
>>>> +    rm -rf "${destdir}"
>>>> +}
>>>> +
>>>> +function project_run() {
>>>> +    target="${1:?}"
>>>> +    destdir="${BUILDDIR}/${target}"
>>>> +    (
>>>> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null 2>&1
>>>> +
>>>> +        /usr/bin/expect <<EOF
>>>> +set timeout 100
>>>> +spawn bash -c "runqemu serialstdio nographic slirp"
>>>> +
>>>> +expect_after {
>>>> +    -re "(.*)\r" {
>>>> +        exp_continue
>>>> +    }
>>>> +    timeout {send_user "ERROR-Timeout!\n"; exit 1}
>>>> +    eof {send_user "ERROR-EOF!\n"; exit 1}
>>>> +}
>>>> +
>>>> +# wait initial login
>>>> +expect -re ".* login: "
>>>> +send "root\r"
>>>> +expect -re "root@.*# "
>>>> +
>>>> +EOF
>>>> +    exit $?
>>>> +    ) || return 1
>>>> +}
>>>> +
>>>> +function help() {
>>>> +    cat <<EOF
>>>> +Usage: ${0} [TARGET1] [TARGET2]
>>>> +
>>>> +This script is build the yocto xen-image-minimal for different qemu targets
>>>> +and is running it after.
>>>> +Without any target specified, all supported targets are done.
>>>> +
>>>> +Options:
>>>> +  -h, --help       Print this help
>>>> +  -v, --verbose    Verbose build
>>>> +  --list-target    List supported targets
>>>> +  --clean          Clean existing project before starting
>>>> +  --no-build       Do not build (to run an already built project)
>>>> +  --no-run         Do not run
>>>> +  --num-jobs=NUM   Define the number of parallel jobs in Yocto.
>>>> +                   Default: ${BUILDJOBS}
>>>> +  --dump-log       On error, dump the logs on the console
>>>> +  --image=IMG      Yocto image or package to build
>>>> +                   Default: xen-image-minimal
>>>> +  --xen-dir=DIR    path to xen hypervisor source tree
>>>> +                   if not provide, normal yocto version of xen is built
>>>> +                   Default: ${XENDIR}
>>>> +  --out-dir=DIR    directory where to create the projectss
>>>> +                   Default: ${BUILDDIR}
>>>> +  --log-dir=DIR    directory to store logs
>>>> +                   Default: ${LOGDIR}
>>>> +  --cache-dir=DIR  directory where to take and store build cache
>>>> +                   Default: ${CACHEDIR}
>>>> +  --layer-dir=DIR  directory containing the checkout of yocto layers
>>>> +                   Default: ${YOCTODIR}
>>>> +EOF
>>>> +}
>>>> +
>>>> +for OPTION in "$@"
>>>> +do
>>>> +    case ${OPTION} in
>>>> +        -h|--help)
>>>> +            help
>>>> +            exit 0
>>>> +            ;;
>>>> +        -v|--verbose)
>>>> +            VERBOSE="y"
>>>> +            ;;
>>>> +        --list-targets)
>>>> +            echo "${TARGET_SUPPORTED}"
>>>> +            exit 0
>>>> +            ;;
>>>> +        --clean)
>>>> +            do_clean="y"
>>>> +            ;;
>>>> +        --no-build)
>>>> +            do_build="n"
>>>> +            ;;
>>>> +        --no-run)
>>>> +            do_run="n"
>>>> +            ;;
>>>> +        --dump-log)
>>>> +            do_dump="y"
>>>> +            ;;
>>>> +        --num-jobs=*)
>>>> +            BUILDJOBS="${OPTION#*=}"
>>>> +            ;;
>>>> +        --image=*)
>>>> +            build_image="${OPTION#*=}"
>>>> +            ;;
>>>> +        --xen-dir=*)
>>>> +            XENDIR="${OPTION#*=}"
>>>> +            if [ ! -e "${XENDIR}/xen/Makefile" ]; then
>>>> +                echo "No Xen source tree in ${XENDIR}"
>>>> +                exit 1
>>>> +            fi
>>>> +            do_localsrc="y"
>>>> +            ;;
>>>> +        --out-dir=*)
>>>> +            BUILDDIR="${OPTION#*=}"
>>>> +            ;;
>>>> +        --log-dir=*)
>>>> +            LOGDIR="${OPTION#*=}"
>>>> +            ;;
>>>> +        --cache-dir=*)
>>>> +            CACHEDIR="${OPTION#*=}"
>>>> +            ;;
>>>> +        --layer-dir=*)
>>>> +            YOCTODIR="${OPTION#*=}"
>>>> +            ;;
>>>> +        --*)
>>>> +            echo "Invalid option ${OPTION}"
>>>> +            help
>>>> +            exit 1
>>>> +            ;;
>>>> +        *)
>>>> +            if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
>>>> +                TARGETLIST="${TARGETLIST} ${OPTION}"
>>>> +            else
>>>> +                echo "Unsupported target ${OPTION}"
>>>> +                exit 1
>>>> +            fi
>>>> +            ;;
>>>> +    esac
>>>> +done
>>>> +
>>>> +# if no target is specified build all targets
>>>> +if [ -z "${TARGETLIST}" ]; then
>>>> +    TARGETLIST="${TARGET_SUPPORTED}"
>>>> +fi
>>>> +
>>>> +mkdir -p "${CACHEDIR}"
>>>> +mkdir -p "${LOGDIR}"
>>>> +mkdir -p "${BUILDDIR}"
>>>> +
>>>> +# Make sure we have an absolute path
>>>> +YOCTODIR=$(realpath -m "${YOCTODIR}")
>>>> +CACHEDIR=$(realpath -m "${CACHEDIR}")
>>>> +BUILDDIR=$(realpath -m "${BUILDDIR}")
>>>> +LOGDIR=$(realpath -m "${LOGDIR}")
>>>> +if [ "${do_localsrc}" = "y" ]; then
>>>> +    XENDIR=$(realpath -m "${XENDIR}")
>>>> +fi
>>>> +
>>>> +# Check that we have all the layers we need
>>>> +for f in ${build_layerlist}; do
>>>> +    if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
>>>> +        echo "Layer ${f} missing in ${YOCTODIR}"
>>>> +        exit 1
>>>> +    fi
>>>> +done
>>>> +
>>>> +for f in ${TARGETLIST}; do
>>>> +    if [ "${do_clean}" = "y" ]; then
>>>> +        run_task project_clean ${f}
>>>> +    fi
>>>> +    if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then
>>>> +        run_task project_create ${f}
>>>> +    fi
>>>> +    if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then
>>>> +        if [ "${do_build}" = "y" ]; then
>>>> +            run_task project_build ${f}
>>>> +        fi
>>>> +        if [ "${do_run}" = "y" ]; then
>>>> +            run_task project_run ${f}
>>>> +        fi
>>>> +
>>>> +    fi
>>>> +done
>>>> +
>>>> +print_progress "Build Complete (${build_result} errors)"
>>>> +echo
>>>> +exit ${build_result}
>>>> +
>>>> diff --git a/automation/build/yocto/yocto.dockerfile.in b/automation/build/yocto/yocto.dockerfile.in
>>>> new file mode 100644
>>>> index 000000000000..5350bb2b87b7
>>>> --- /dev/null
>>>> +++ b/automation/build/yocto/yocto.dockerfile.in
>>>> @@ -0,0 +1,114 @@
>>>> +# Docker file to create an environment to build yocto with virtualization
>>>> +#
>>>> +# Arguments that can be passed during image creation using --build-arg:
>>>> +# "host_uid=$(id -u)": to use current user uid for build user in the image
>>>> +# "host_gid=$(id -g)": to use current user gid for build user in the image
>>>> +# "ubuntu_version=VERS": to select the ubuntu version number
>>> Is is the case, that this dockerfile and the packages installed will work on any
>>> version of ubuntu we will pass here? If not, maybe we should just stick to 22.04 and
>>> not give the user the opportunity to change this.
>> 
>> There is no support right now to build for a different version of ubuntu without
>> modifying the docker file. I kept that as argument here because the generated
>> file/container does not contain an ubuntu version in its name so changing the
>> value here in the future will make it clearer.
> Ok.
>> 
>> I am ok to remove the variable and directly use the value if you think this is necessary.
> No need.
> 
>> 
>>> 
>>>> +
>>>> +# Use standard ubuntu minimal
>>>> +ARG ubuntu_version=22.04
>>>> +From ##DOCKERPLAT##ubuntu:$ubuntu_version AS base
>>>> +LABEL maintainer.name="The Xen Project " \
>>>> +      maintainer.email="xen-devel@lists.xenproject.org"
>>>> +
>>>> +ENV DEBIAN_FRONTEND=noninteractive
>>>> +
>>>> +# Install minimal ubuntu requirements for yocto and other tools we need
>>>> +# See https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.yoctoproject.org%2F4.0.1%2Fbrief-yoctoprojectqs%2Findex.html%23build-host-packages&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd5eca08f965d4fd1a99508dac95a41e7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638043686755292322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=D%2FAe%2FCUo2%2FWcvJAXjRW4vW9z4pwWcUEOhe1TdlozECw%3D&amp;reserved=0
>>>> +RUN apt-get update && \
>>>> +    apt-get --quiet --yes install \
>>>> +        gawk \
>>>> +        wget \
>>>> +        git \
>>>> +        diffstat \
>>>> +        unzip \
>>>> +        texinfo \
>>>> +        gcc \
>>>> +        build-essential \
>>>> +        chrpath \
>>>> +        socat \
>>>> +        cpio \
>>>> +        python3 \
>>>> +        python3-pip \
>>>> +        python3-pexpect \
>>>> +        xz-utils \
>>>> +        debianutils \
>>>> +        iputils-ping \
>>>> +        python3-git \
>>>> +        python3-jinja2 \
>>>> +        libegl1-mesa \
>>>> +        libsdl1.2-dev \
>>>> +        python3-subunit \
>>>> +        mesa-common-dev \
>>>> +        zstd \
>>>> +        liblz4-tool \
>>>> +        file \
>>>> +        vim \
>>>> +        bison \
>>>> +        expect \
>>>> +        locales \
>>>> +        liblz4-tool \
>>>> +        zstd \
>>>> +        openssl \
>>>> +        libssl3 \
>>>> +        ca-certificates \
>>>> +        && \
>>>> +        apt-get autoremove -y && \
>>>> +        apt-get clean && \
>>>> +        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
>>>> +
>>>> +# Use bash as shell
>>>> +RUN rm /bin/sh && ln -s bash /bin/sh
>>>> +
>>>> +# Fix local for yocto
>>>> +RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
>>>> +    LANG=en_US.UTF-8
>>>> +ENV LANG en_US.UTF-8
>>>> +ENV LC_ALL en_US.UTF-8
>>>> +
>>>> +# Create a user for the build (we don't want to build as root)
>>>> +ENV USER_NAME docker-build
>>>> +ARG host_uid=1000
>>>> +ARG host_gid=1000
>>>> +RUN groupadd -g $host_gid $USER_NAME && \
>>>> +    useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
>>>> +
>>>> +# Switch to our user instead of root and start in its home
>>>> +USER $USER_NAME
>>>> +WORKDIR /home/$USER_NAME
>>>> +
>>>> +# Create needed directories
>>>> +RUN mkdir -p /home/$USER_NAME/yocto-layers \
>>>> +             /home/$USER_NAME/yocto-cache \
>>>> +             /home/$USER_NAME/logs \
>>>> +             /home/$USER_NAME/bin \
>>>> +             /home/$USER_NAME/xen && \
>>>> +    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
>>>> +
>>>> +# clone yocto repositories we need
>>>> +RUN for rep in \
>>>> +                https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenembedded%2Fmeta-openembedded&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd5eca08f965d4fd1a99508dac95a41e7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638043686755292322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=5IB%2BO9M0DU3D7hN7q9Ft9OivqfmTVKuOVKgFWuElUu4%3D&amp;reserved=0 \
>>>> +                https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.yoctoproject.org%2Fpoky&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd5eca08f965d4fd1a99508dac95a41e7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638043686755448030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=fMRgTFNrjR5athu9kXSNXAL%2FeN2RwO%2Bu6CZxCbhesyU%3D&amp;reserved=0 \
>>>> +                https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.yoctoproject.org%2Fmeta-virtualization&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd5eca08f965d4fd1a99508dac95a41e7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638043686755448030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=J6JQP3Ibn2WaD1cTtduC4T%2BbnNWlUHmFPXrLsIe1CSw%3D&amp;reserved=0 \
>>>> +            ; do \
>>>> +        git -C /home/$USER_NAME/yocto-layers \
>>>> +            clone -b ##YOCTOVERSION## --single-branch $rep; \
>>>> +    done
>>>> +
>>>> +# The builder stage is building an initial cache state that we include in the
>>>> +# final image
>>>> +From base AS builder
>>>> +
>>>> +# This step can take one to several hours depending on your download bandwith
>>>> +# and the speed of your computer
>>>> +COPY ./build-yocto.sh /
>>>> +RUN /build-yocto.sh --dump-log ##YOCTOTARGET##
>>>> +
>>>> +From base
>>>> +
>>>> +# Only copy the cache status
>>>> +COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
>>>> +
>>>> +LABEL maintainer.name="The Xen Project " \
>>>> +      maintainer.email="xen-devel@lists.xenproject.org"
>>>> +
>>>> diff --git a/automation/build/yocto/yocto.inc b/automation/build/yocto/yocto.inc
>>>> new file mode 100644
>>>> index 000000000000..04076bc8d174
>>>> --- /dev/null
>>>> +++ b/automation/build/yocto/yocto.inc
>>>> @@ -0,0 +1,41 @@
>>>> +# This makefile generates the docker files for Yocto builds
>>>> +# The containers for the current architecture are the one built using make all
>>>> +# To build containers for a different architecture, you need to call make for
>>>> +# the image you want explicitely
>>> Could you please add a dot at the end of each sentence above. This will improve readability.
>> 
>> Yes will do.
>> 
>>> 
>>>> +# The containers are named this way:
>>>> +# YOCTOVERSION-TARGET for x86_64 hosts
>>>> +# YOCTOVERSION-TARGET-arm64v8 for arm64 hosts
>>>> +
>>>> +# Yocto versions we are currently using
>>>> +YOCTO_VERSION = kirkstone
>>>> +
>>>> +# Yocto BSPs we want to build for
>>>> +YOCTO_TARGETS = qemuarm64 qemuarm qemux86-64
>>>> +
>>>> +# Supported Host platforms (host architecture specific ones)
>>> To avoid mismatch \wrt platform vs architecture I would stick to the latter one.
>>> In the docker world, arm64v8 is an architecture whereas linux/arm64/v8 is a platform.
>> 
>> So "yocto target” and “container platform” ?
> container architecture
> 

Ok

>> 
>>> 
>>>> +YOCTO_HOSTS = amd64 arm64v8
>>>> +
>>>> +# Architecture we want to use in gitlab CI (depends on runners arch)
>>>> +CI_ARCH = arm64v8
>>>> +
>>>> +define GEN_DOCKER
>>>> +# Make all is generating what we want in the CI
>>>> +ifeq ($(CI_ARCH),$(3))
>>>> +CONTAINERS += yocto/$(1)-$(2)$(4)
>>>> +else
>>>> +CONTAINERS_EXTRA += yocto/$(1)-$(2)$(4)
>>>> +endif
>>>> +
>>>> +yocto/$(1)-$(2)$(4).dockerfile: yocto/yocto.dockerfile.in
>>>> +       echo > $$@
>>>> +       cat $$< | \
>>>> +           sed -e "s,##YOCTOVERSION##,$(1),g" | \
>>>> +           sed -e "s,##YOCTOTARGET##,$(2),g" | \
>>>> +           sed -e "s,##DOCKERPLAT##,$(3)/,g" > $$@
>>>> +
>>>> +endef
>>>> +
>>>> +$(eval $(foreach vers,$(YOCTO_VERSION),\
>>>> +    $(foreach tar,$(YOCTO_TARGETS),\
>>>> +    $(foreach hst,$(YOCTO_HOSTS),\
>>>> +    $(call GEN_DOCKER,$(vers),$(tar),$(hst),$(if $(filter amd64,$(hst)),,-$(hst)))))))
>>> This could be aligned under first foreach. Also I think there is no need to try to shorten the version,target,host.
>> 
>> This is something you requested me in order to have container names for yocto coherent
>> with other xen containers (ie only have the -arm64v8 suffix for arm).
>> 
>> Do you want me to revert this and have names containing the amd64 suffix ?
> This is not what I meant.
> I just wanted to use the full words like target,host,version instead of tar,hst,vers
> and for the "foreach" to be aligned.

Ah ok this is more clear, I will change that.

Cheers
Bertrand

> 
>> 
>> Thanks for the review
>> 
>> Cheers
>> Bertrand
>> 
>>> 
>>>> --
>>>> 2.25.1
>>>> 
>>> 
>>> ~Michal
>> 
> ~Michal

Re: [PATCH v4 1/3] automation: Create Yocto docker images

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 20835 bytes --]

On Thu, 17 Nov 2022, Bertrand Marquis wrote:
> Add containers suitable to run yocto kirkstone build based on ubuntu
> 22.04. It contains all packages required by Yocto and a checkout of the
> layers required to build Xen with Yocto.
> 
> Add a generic docker image template to be used to automatically generate
> docker files for different configurations:
> - specific yocto version
> - different targets (qemu arm, arm64 and x86)
> - different host platforms (x86 or arm64)
> 
> During a call to 'make all', only the images for the current host
> platform will be generated.
> If needed, images for an other host platform can be generated manually
> by calling the right make target (see make help).
> 
> Add a build script to build and run xen on qemu using Yocto.
> The script supports arm32, arm64 and x86_64 and checks that dom0 is
> properly booting. At this stage this does not run any guest on top of
> dom0. The script is to be executed in one of the docker images to build
> and run a system using a Xen source tree.
> 
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>

This series looks great and I think it is almost ready to be committed.
Only one comment from me below.

Also, I tested it successfully both building the containers, and also
the full gitlab-ci pipeline, hurray!

The only issue is that it works for qemuarm64 and qemuarm, but not
qemux86-64: the container build doesn't work. I am attaching the error
logs from 2 different builds on 2 different arm64 machines.

To be honest, if qemux86-64 on arm64 turns out to be an issue, I'd be
happy to take the series without it for now.


> ---
> Changes in v4:
> - Rework the system to have one dockerfile template from which make will
> generate the required dockerfiles for the wanted configuration
> - add support for different host architectures
> - Merge the generation system into one single dockerfile
> - Merge patches 1 and 2 in a single patch
> - Introduce CONTAINERS_EXTRA to have extra containers not built by
> default (for those not used by CI but useful to users)
> Changes in v3:
> - limit number of jobs in yocto by default to 8 and add --num-jobs
> option to the script to set a custom number of jobs
> - do not copy anymore the build-yocto.sh script inside the main image so
> that the current one in the repository is used when running
> Changes in v2:
> - add a --dump-log command line option to build-yocto.sh script to dump
> the logs if an error occurs.
> Changes in v1:
> - add --image command line argument to build-yocto.sh to allow building
> something different than xen-image-minimal.
> - modify dockerfile to have one layer per line and make it easier to add
> other. I kept the for loop to keep the number of docker steps lower
> - update commit message to warn that no guest are tested.
> - fix build-yocto script to properly return with an error if expect
> script ends up in timeout or EOF.
> ---
>  automation/build/Makefile                  |  14 +-
>  automation/build/yocto/build-yocto.sh      | 349 +++++++++++++++++++++
>  automation/build/yocto/yocto.dockerfile.in | 114 +++++++
>  automation/build/yocto/yocto.inc           |  41 +++
>  4 files changed, 516 insertions(+), 2 deletions(-)
>  create mode 100755 automation/build/yocto/build-yocto.sh
>  create mode 100644 automation/build/yocto/yocto.dockerfile.in
>  create mode 100644 automation/build/yocto/yocto.inc
> 
> diff --git a/automation/build/Makefile b/automation/build/Makefile
> index a4b2b85178cf..72a5335baec1 100644
> --- a/automation/build/Makefile
> +++ b/automation/build/Makefile
> @@ -1,13 +1,18 @@
>  
>  # the base of where these containers will appear
>  REGISTRY := registry.gitlab.com/xen-project/xen
> -CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
> +CONTAINERS = $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard */*.dockerfile)))
> +CONTAINERS_EXTRA =
>  DOCKER_CMD ?= docker
>  
> +include yocto/yocto.inc

I think it would be good to add a comment here to say how make could be
invoked for the yocto containers, for instance "make
yocto/kirkstone-qemuarm64" or "make yocto/kirkstone-qemuarm64-arm64v8"
and explain the difference. People tend to build locally individual
containers, rather than the whole set.


>  help:
>  	@echo "Builds containers for building Xen based on different distros"
>  	@echo "To build one run 'make DISTRO/VERSION'. Available containers:"
>  	@$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
> +	@echo "Extra containers (not built using make all):"
> +	@$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo ${file};)
>  	@echo "To push container builds, set the env var PUSH"
>  
>  %: %.dockerfile ## Builds containers
> @@ -16,5 +21,10 @@ help:
>  		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
>  	fi
>  
> -.PHONY: all
> +.PHONY: all clean
>  all: $(CONTAINERS)
> +
> +# Remove generated dockerfiles for yocto
> +clean:
> +	rm -f yocto/*.dockerfiles
> +
> diff --git a/automation/build/yocto/build-yocto.sh b/automation/build/yocto/build-yocto.sh
> new file mode 100755
> index 000000000000..d0c93dfaffe0
> --- /dev/null
> +++ b/automation/build/yocto/build-yocto.sh
> @@ -0,0 +1,349 @@
> +#!/bin/bash
> +#
> +# Yocto meta virtualization build and run script
> +#
> +# This script is building Yocto xen-image-minimal for qemu targets and run
> +# them using runqemu inside yocto to check that dom0 is booting properly
> +# The build is using a local xen source tree so that specific patches can be
> +# tested.
> +# In order to optimize the build time, a build cache is used so that only xen
> +# packages and its dependencies are rebuilt (qemu and final image mainly).
> +#
> +
> +# Directories
> +YOCTODIR="$HOME/yocto-layers"
> +CACHEDIR="$HOME/yocto-cache"
> +LOGDIR="$HOME/logs"
> +XENDIR="$HOME/xen"
> +BUILDDIR="$HOME/build"
> +
> +# what yocto bsp we support
> +TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
> +VERBOSE="n"
> +TARGETLIST=""
> +BUILDJOBS="8"
> +
> +# actions to do
> +do_clean="n"
> +do_build="y"
> +do_run="y"
> +do_localsrc="n"
> +do_dump="n"
> +build_result=0
> +
> +# layers to include in the project
> +build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
> +                 meta-openembedded/meta-oe meta-openembedded/meta-python \
> +                 meta-openembedded/meta-filesystems \
> +                 meta-openembedded/meta-networking meta-virtualization"
> +
> +# yocto image to build
> +build_image="xen-image-minimal"
> +
> +function print_progress() {
> +    echo -n "$(date +%T) $*"
> +}
> +
> +function run_task() {
> +    local task_name="$1"
> +    local task_target="$2"
> +
> +    task_log="${task_name//project_}-${task_target}"
> +
> +    mkdir -p "${LOGDIR}"
> +    print_progress
> +    echo -n "${task_name//project_} ${task_target}: "
> +    if [ "${VERBOSE}" = "n" ]; then
> +        $@ > "${LOGDIR}/${task_log}.log" 2>&1
> +    else
> +        $@ 2>&1 | tee "${LOGDIR}/${task_log}.log"
> +    fi
> +
> +    if [ ${?} -ne 0 ]; then
> +        echo "Error"
> +        build_result=$((build_result+1))
> +        if [ "${do_dump}" = "y" ]; then
> +            echo
> +            echo "############ LOGS-START ############"
> +            cat "${LOGDIR}/${task_log}.log"
> +            echo "############  LOGS-END  ############"
> +            echo
> +        fi
> +        return 1
> +    else
> +        echo "OK"
> +        return 0
> +    fi
> +}
> +
> +function project_create() {
> +    target="${1:?}"
> +    destdir="${BUILDDIR}/${target}"
> +
> +    (
> +        # init yocto project
> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
> +
> +        # add needed layers
> +        for layer in ${build_layerlist}; do
> +            bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1
> +        done
> +    ) || return 1
> +
> +    # customize project configuration
> +    cat <<EOF >> "${destdir}/conf/local.conf"
> +# Yocto BSP
> +MACHINE = "${target}"
> +
> +# Use local cache to reuse previous builds results
> +SSTATE_DIR = "${CACHEDIR}/sstate-cache"
> +DL_DIR = "${CACHEDIR}/downloads"
> +
> +# Enable xen and virtualization
> +DISTRO_FEATURES = " virtualization xen ipv4"
> +
> +# Speed up run by not generating ssh host keys
> +IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
> +
> +# Save some disk space
> +INHERIT += "rm_work"
> +
> +# Reduce number of jobs
> +BB_NUMBER_THREADS="${BUILDJOBS}"
> +
> +EOF
> +
> +    if [ "${do_localsrc}" = "y" ]; then
> +        XENVERS=$(grep -e "^XEN_REL" \
> +            "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
> +            2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
> +            | sed -e "s/.*=//" | sort -V | tail -n 1)
> +
> +        XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
> +        XENSUB=$(basename "$(realpath -m "${XENDIR}")")
> +
> +        cat <<EOF >> "${destdir}/conf/local.conf"
> +# Use local sources for xen and xen-tools
> +FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
> +FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
> +
> +SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
> +SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
> +
> +PREFERRED_VERSION:pn-xen = "${XENVERS}%"
> +PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
> +
> +S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
> +S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
> +
> +SRCREV:pn-xen = "\${AUTOREV}"
> +SRCREV:pn-xen-tools = "\${AUTOREV}"
> +
> +SRCPV:pn-xen = "1"
> +SRCPV:pn-xen-tools = "1"
> +
> +# Disable all QA errors as the recipe is not up to date with changes in Xen
> +# when we use local sources
> +ERROR_QA:pn-xen = "arch"
> +ERROR_QA:pn-xen-tools = "arch"
> +
> +EOF
> +    fi
> +}
> +
> +function project_build() {
> +    target="${1:?}"
> +    destdir="${BUILDDIR}/${target}"
> +
> +    (
> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
> +
> +        bitbake "${build_image}" || exit 1
> +    ) || return 1
> +}
> +
> +function project_clean() {
> +    target="${1:?}"
> +    destdir="${BUILDDIR}/${target}"
> +
> +    rm -rf "${destdir}"
> +}
> +
> +function project_run() {
> +    target="${1:?}"
> +    destdir="${BUILDDIR}/${target}"
> +    (
> +        source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null 2>&1
> +
> +        /usr/bin/expect <<EOF
> +set timeout 100
> +spawn bash -c "runqemu serialstdio nographic slirp"
> +
> +expect_after {
> +    -re "(.*)\r" {
> +        exp_continue
> +    }
> +    timeout {send_user "ERROR-Timeout!\n"; exit 1}
> +    eof {send_user "ERROR-EOF!\n"; exit 1}
> +}
> +
> +# wait initial login
> +expect -re ".* login: "
> +send "root\r"
> +expect -re "root@.*# "
> +
> +EOF
> +    exit $?
> +    ) || return 1
> +}
> +
> +function help() {
> +    cat <<EOF
> +Usage: ${0} [TARGET1] [TARGET2]
> +
> +This script is build the yocto xen-image-minimal for different qemu targets
> +and is running it after.
> +Without any target specified, all supported targets are done.
> +
> +Options:
> +  -h, --help       Print this help
> +  -v, --verbose    Verbose build
> +  --list-target    List supported targets
> +  --clean          Clean existing project before starting
> +  --no-build       Do not build (to run an already built project)
> +  --no-run         Do not run
> +  --num-jobs=NUM   Define the number of parallel jobs in Yocto.
> +                   Default: ${BUILDJOBS}
> +  --dump-log       On error, dump the logs on the console
> +  --image=IMG      Yocto image or package to build
> +                   Default: xen-image-minimal
> +  --xen-dir=DIR    path to xen hypervisor source tree
> +                   if not provide, normal yocto version of xen is built
> +                   Default: ${XENDIR}
> +  --out-dir=DIR    directory where to create the projectss
> +                   Default: ${BUILDDIR}
> +  --log-dir=DIR    directory to store logs
> +                   Default: ${LOGDIR}
> +  --cache-dir=DIR  directory where to take and store build cache
> +                   Default: ${CACHEDIR}
> +  --layer-dir=DIR  directory containing the checkout of yocto layers
> +                   Default: ${YOCTODIR}
> +EOF
> +}
> +
> +for OPTION in "$@"
> +do
> +    case ${OPTION} in
> +        -h|--help)
> +            help
> +            exit 0
> +            ;;
> +        -v|--verbose)
> +            VERBOSE="y"
> +            ;;
> +        --list-targets)
> +            echo "${TARGET_SUPPORTED}"
> +            exit 0
> +            ;;
> +        --clean)
> +            do_clean="y"
> +            ;;
> +        --no-build)
> +            do_build="n"
> +            ;;
> +        --no-run)
> +            do_run="n"
> +            ;;
> +        --dump-log)
> +            do_dump="y"
> +            ;;
> +        --num-jobs=*)
> +            BUILDJOBS="${OPTION#*=}"
> +            ;;
> +        --image=*)
> +            build_image="${OPTION#*=}"
> +            ;;
> +        --xen-dir=*)
> +            XENDIR="${OPTION#*=}"
> +            if [ ! -e "${XENDIR}/xen/Makefile" ]; then
> +                echo "No Xen source tree in ${XENDIR}"
> +                exit 1
> +            fi
> +            do_localsrc="y"
> +            ;;
> +        --out-dir=*)
> +            BUILDDIR="${OPTION#*=}"
> +            ;;
> +        --log-dir=*)
> +            LOGDIR="${OPTION#*=}"
> +            ;;
> +        --cache-dir=*)
> +            CACHEDIR="${OPTION#*=}"
> +            ;;
> +        --layer-dir=*)
> +            YOCTODIR="${OPTION#*=}"
> +            ;;
> +        --*)
> +            echo "Invalid option ${OPTION}"
> +            help
> +            exit 1
> +            ;;
> +        *)
> +            if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
> +                TARGETLIST="${TARGETLIST} ${OPTION}"
> +            else
> +                echo "Unsupported target ${OPTION}"
> +                exit 1
> +            fi
> +            ;;
> +    esac
> +done
> +
> +# if no target is specified build all targets
> +if [ -z "${TARGETLIST}" ]; then
> +    TARGETLIST="${TARGET_SUPPORTED}"
> +fi
> +
> +mkdir -p "${CACHEDIR}"
> +mkdir -p "${LOGDIR}"
> +mkdir -p "${BUILDDIR}"
> +
> +# Make sure we have an absolute path
> +YOCTODIR=$(realpath -m "${YOCTODIR}")
> +CACHEDIR=$(realpath -m "${CACHEDIR}")
> +BUILDDIR=$(realpath -m "${BUILDDIR}")
> +LOGDIR=$(realpath -m "${LOGDIR}")
> +if [ "${do_localsrc}" = "y" ]; then
> +    XENDIR=$(realpath -m "${XENDIR}")
> +fi
> +
> +# Check that we have all the layers we need
> +for f in ${build_layerlist}; do
> +    if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
> +        echo "Layer ${f} missing in ${YOCTODIR}"
> +        exit 1
> +    fi
> +done
> +
> +for f in ${TARGETLIST}; do
> +    if [ "${do_clean}" = "y" ]; then
> +        run_task project_clean ${f}
> +    fi
> +    if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then
> +        run_task project_create ${f}
> +    fi
> +    if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then
> +        if [ "${do_build}" = "y" ]; then
> +            run_task project_build ${f}
> +        fi
> +        if [ "${do_run}" = "y" ]; then
> +            run_task project_run ${f}
> +        fi
> +
> +    fi
> +done
> +
> +print_progress "Build Complete (${build_result} errors)"
> +echo
> +exit ${build_result}
> +
> diff --git a/automation/build/yocto/yocto.dockerfile.in b/automation/build/yocto/yocto.dockerfile.in
> new file mode 100644
> index 000000000000..5350bb2b87b7
> --- /dev/null
> +++ b/automation/build/yocto/yocto.dockerfile.in
> @@ -0,0 +1,114 @@
> +# Docker file to create an environment to build yocto with virtualization
> +#
> +# Arguments that can be passed during image creation using --build-arg:
> +# "host_uid=$(id -u)": to use current user uid for build user in the image
> +# "host_gid=$(id -g)": to use current user gid for build user in the image
> +# "ubuntu_version=VERS": to select the ubuntu version number
> +
> +# Use standard ubuntu minimal
> +ARG ubuntu_version=22.04
> +From ##DOCKERPLAT##ubuntu:$ubuntu_version AS base
> +LABEL maintainer.name="The Xen Project " \
> +      maintainer.email="xen-devel@lists.xenproject.org"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +
> +# Install minimal ubuntu requirements for yocto and other tools we need
> +# See https://docs.yoctoproject.org/4.0.1/brief-yoctoprojectqs/index.html#build-host-packages
> +RUN apt-get update && \
> +    apt-get --quiet --yes install \
> +        gawk \
> +        wget \
> +        git \
> +        diffstat \
> +        unzip \
> +        texinfo \
> +        gcc \
> +        build-essential \
> +        chrpath \
> +        socat \
> +        cpio \
> +        python3 \
> +        python3-pip \
> +        python3-pexpect \
> +        xz-utils \
> +        debianutils \
> +        iputils-ping \
> +        python3-git \
> +        python3-jinja2 \
> +        libegl1-mesa \
> +        libsdl1.2-dev \
> +        python3-subunit \
> +        mesa-common-dev \
> +        zstd \
> +        liblz4-tool \
> +        file \
> +        vim \
> +        bison \
> +        expect \
> +        locales \
> +        liblz4-tool \
> +        zstd \
> +        openssl \
> +        libssl3 \
> +        ca-certificates \
> +        && \
> +        apt-get autoremove -y && \
> +        apt-get clean && \
> +        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
> +
> +# Use bash as shell
> +RUN rm /bin/sh && ln -s bash /bin/sh
> +
> +# Fix local for yocto
> +RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
> +    LANG=en_US.UTF-8
> +ENV LANG en_US.UTF-8
> +ENV LC_ALL en_US.UTF-8
> +
> +# Create a user for the build (we don't want to build as root)
> +ENV USER_NAME docker-build
> +ARG host_uid=1000
> +ARG host_gid=1000
> +RUN groupadd -g $host_gid $USER_NAME && \
> +    useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
> +
> +# Switch to our user instead of root and start in its home
> +USER $USER_NAME
> +WORKDIR /home/$USER_NAME
> +
> +# Create needed directories
> +RUN mkdir -p /home/$USER_NAME/yocto-layers \
> +             /home/$USER_NAME/yocto-cache \
> +             /home/$USER_NAME/logs \
> +             /home/$USER_NAME/bin \
> +             /home/$USER_NAME/xen && \
> +    chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
> +
> +# clone yocto repositories we need
> +RUN for rep in \
> +                https://github.com/openembedded/meta-openembedded \
> +                https://git.yoctoproject.org/poky \
> +                https://git.yoctoproject.org/meta-virtualization \
> +            ; do \
> +        git -C /home/$USER_NAME/yocto-layers \
> +            clone -b ##YOCTOVERSION## --single-branch $rep; \
> +    done
> +
> +# The builder stage is building an initial cache state that we include in the
> +# final image
> +From base AS builder
> +
> +# This step can take one to several hours depending on your download bandwith
> +# and the speed of your computer
> +COPY ./build-yocto.sh /
> +RUN /build-yocto.sh --dump-log ##YOCTOTARGET##
> +
> +From base
> +
> +# Only copy the cache status
> +COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
> +
> +LABEL maintainer.name="The Xen Project " \
> +      maintainer.email="xen-devel@lists.xenproject.org"
> +
> diff --git a/automation/build/yocto/yocto.inc b/automation/build/yocto/yocto.inc
> new file mode 100644
> index 000000000000..04076bc8d174
> --- /dev/null
> +++ b/automation/build/yocto/yocto.inc
> @@ -0,0 +1,41 @@
> +# This makefile generates the docker files for Yocto builds
> +# The containers for the current architecture are the one built using make all
> +# To build containers for a different architecture, you need to call make for
> +# the image you want explicitely
> +# The containers are named this way:
> +# YOCTOVERSION-TARGET for x86_64 hosts
> +# YOCTOVERSION-TARGET-arm64v8 for arm64 hosts
> +
> +# Yocto versions we are currently using
> +YOCTO_VERSION = kirkstone
> +
> +# Yocto BSPs we want to build for
> +YOCTO_TARGETS = qemuarm64 qemuarm qemux86-64
> +
> +# Supported Host platforms (host architecture specific ones)
> +YOCTO_HOSTS = amd64 arm64v8
> +
> +# Architecture we want to use in gitlab CI (depends on runners arch)
> +CI_ARCH = arm64v8
> +
> +define GEN_DOCKER
> +# Make all is generating what we want in the CI
> +ifeq ($(CI_ARCH),$(3))
> +CONTAINERS += yocto/$(1)-$(2)$(4)
> +else
> +CONTAINERS_EXTRA += yocto/$(1)-$(2)$(4)
> +endif
> +
> +yocto/$(1)-$(2)$(4).dockerfile: yocto/yocto.dockerfile.in
> +	echo > $$@
> +	cat $$< | \
> +	    sed -e "s,##YOCTOVERSION##,$(1),g" | \
> +	    sed -e "s,##YOCTOTARGET##,$(2),g" | \
> +	    sed -e "s,##DOCKERPLAT##,$(3)/,g" > $$@
> +
> +endef
> +
> +$(eval $(foreach vers,$(YOCTO_VERSION),\
> +    $(foreach tar,$(YOCTO_TARGETS),\
> +    $(foreach hst,$(YOCTO_HOSTS),\
> +    $(call GEN_DOCKER,$(vers),$(tar),$(hst),$(if $(filter amd64,$(hst)),,-$(hst)))))))
> -- 
> 2.25.1
> 

[-- Attachment #2: Type: application/octet-stream, Size: 35906 bytes --]

| sed -e 's/AmlCode/ssdt_s4/g' -e 's/_aml_code//g' /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_s4.hex >/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_s4.h
| ASL Input:     ssdt_tpm.asl -     912 bytes      3 keywords     30 source lines
| AML Output:    /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_tpm.aml -      76 bytes      0 opcodes       3 named objects
| Hex Dump:      /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_tpm.hex -    1153 bytes
| 
| Compilation successful. 0 Errors, 0 Warnings, 0 Remarks, 0 Optimizations
| sed -e 's/AmlCode/ssdt_tpm/g' -e 's/_aml_code//g' /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_tpm.hex >/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_tpm.h
| ASL Input:     ssdt_laptop_slate.asl -    1086 bytes      4 keywords     40 source lines
| AML Output:    /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_laptop_slate.aml -      73 bytes      1 opcodes       3 named objects
| Hex Dump:      /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_laptop_slate.hex -    1189 bytes
| 
| Compilation successful. 0 Errors, 0 Warnings, 0 Remarks, 0 Optimizations
| sed -e 's/AmlCode/ssdt_laptop_slate/g' -e 's/_aml_code//g' /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_laptop_slate.hex >/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_laptop_slate.h
| ssdt_pm.asl    397:                 Store (\_SB.PB2, Local0)
| Warning  3144 -       Method Local is set but never used ^  (Local0)
| 
| ASL Input:     ssdt_pm.asl -   12575 bytes    192 keywords    420 source lines
| AML Output:    /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_pm.aml -    1447 bytes    131 opcodes      61 named objects
| Hex Dump:      /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_pm.hex -   13975 bytes
| 
| Compilation successful. 0 Errors, 1 Warnings, 0 Remarks, 31 Optimizations
| sed -e 's/AmlCode/ssdt_pm/g' -e 's/_aml_code//g' /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_pm.hex >/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_pm.h
| rm -f /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_s3.aml /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_s3.hex
| rm -f /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_tpm.aml /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_tpm.hex
| x86_64-poky-linux-gcc  --sysroot=/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/recipe-sysroot   -m64 -DBUILD_ID -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable -Wno-unused-local-typedefs   -O2 -fomit-frame-pointer -D__XEN_INTERFACE_VERSION__=__XEN_LATEST_INTERFACE_VERSION__ -MMD -MP -MF ._libxl.api-for-check.d -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DCONFIG_PCI_SUPP_LEGACY_IRQ -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security  -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0=/usr/src/debug/xen-tools/4.16+stableAUTOINC+f265444922-r0                      -fdebug-prefix-map=/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0=/usr/src/debug/xen-tools/4.16+stableAUTOINC+f265444922-r0                      -fdebug-prefix-map=/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/recipe-sysroot-native=  -Wno-format-zero-length -Wmissing-declarations -Wno-declaration-after-statement -Wformat-nonliteral -I. -pthread -Werror -Wmissing-prototypes -I./include -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include -D__XEN_TOOLS__  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/recipe-sysroot/usr/include/libnl3 -Wshadow -include /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/config.h -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -c -E /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include/libxl.h  \
|       -DLIBXL_EXTERNAL_CALLERS_ONLY=LIBXL_EXTERNAL_CALLERS_ONLY \
|       >_libxl.api-for-check.new
| rm -f /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_s4.aml /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_s4.hex
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/recipe-sysroot-native/usr/bin/python3-native/python3 gentest.py libxl_types.idl testidl.c.new
| rm -f /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_laptop_slate.aml /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_laptop_slate.hex
| rm -f /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_pm.aml /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_pm.hex
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:34: error: "___DEFINE_XEN_GUEST_HANDLE" redefined [-Werror]
|    34 | #define ___DEFINE_XEN_GUEST_HANDLE(name, type) \
|       |
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:176: note: this is the location of the previous definition
|   176 | #define ___DEFINE_XEN_GUEST_HANDLE(name, type)                  \
|       |
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:53: error: "__XEN_GUEST_HANDLE" redefined [-Werror]
|    53 | #define __XEN_GUEST_HANDLE(name)        __guest_handle_ ## name
|       |
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:193: note: this is the location of the previous definition
|   193 | #define __XEN_GUEST_HANDLE(name)        __guest_handle_64_ ## name
|       |
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:55: error: "XEN_GUEST_HANDLE_PARAM" redefined [-Werror]
|    55 | #define XEN_GUEST_HANDLE_PARAM(name)    XEN_GUEST_HANDLE(name)
|       |
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:195: note: this is the location of the previous definition
|   195 | #define XEN_GUEST_HANDLE_PARAM(name)    __guest_handle_ ## name
|       |
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:56: error: "set_xen_guest_handle_raw" redefined [-Werror]
|    56 | #define set_xen_guest_handle_raw(hnd, val)  do { (hnd).p = val; } while (0)
|       |
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:196: note: this is the location of the previous definition
|   196 | #define set_xen_guest_handle_raw(hnd, val)                  \
|       |
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:77: error: "PRI_xen_pfn" redefined [-Werror]
|    77 | #define PRI_xen_pfn "lx"
|       |
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:205: note: this is the location of the previous definition
|   205 | #define PRI_xen_pfn PRIx64
|       |
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:78: error: "PRIu_xen_pfn" redefined [-Werror]
|    78 | #define PRIu_xen_pfn "lu"
|       |
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:206: note: this is the location of the previous definition
|   206 | #define PRIu_xen_pfn PRIu64
|       |
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:117: error: "XEN_LEGACY_MAX_VCPUS" redefined [-Werror]
|   117 | #define XEN_LEGACY_MAX_VCPUS 32
|       |
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:212: note: this is the location of the previous definition
|   212 | #define XEN_LEGACY_MAX_VCPUS 1
|       |
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:122: error: "PRI_xen_ulong" redefined [-Werror]
|   122 | #define PRI_xen_ulong "lx"
|       |
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:215: note: this is the location of the previous definition
|   215 | #define PRI_xen_ulong PRIx64
|       |
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:179:8: error: redefinition of ‘struct vcpu_guest_context’
|   179 | struct vcpu_guest_context {
|       |        ^~~~~~~~~~~~~~~~~~
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:293:8: note: originally defined here
|   293 | struct vcpu_guest_context {
|       |        ^~~~~~~~~~~~~~~~~~
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:192: error: "_VGCF_online" redefined [-Werror]
|   192 | #define _VGCF_online                   5
|       |
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:294: note: this is the location of the previous definition
|   294 | #define _VGCF_online                   0
|       |
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:195:26: error: field ‘user_regs’ has incomplete type
|   195 |     struct cpu_user_regs user_regs;         /* User-level CPU registers     */
|       |                          ^~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:231:35: error: conflicting types for ‘vcpu_guest_context_t’; have ‘struct vcpu_guest_context’
|   231 | typedef struct vcpu_guest_context vcpu_guest_context_t;
|       |                                   ^~~~~~~~~~~~~~~~~~~~
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:303:35: note: previous declaration of ‘vcpu_guest_context_t’ with type ‘vcpu_guest_context_t’ {aka ‘struct vcpu_guest_context’}
|   303 | typedef struct vcpu_guest_context vcpu_guest_context_t;
|       |                                   ^~~~~~~~~~~~~~~~~~~~
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:35:33: error: conflicting types for ‘__guest_handle_vcpu_guest_context_t’; have ‘struct <anonymous>’
|    35 |     typedef struct { type *p; } __guest_handle_ ## name
|       |                                 ^~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:50:5: note: in expansion of macro ‘___DEFINE_XEN_GUEST_HANDLE’
|    50 |     ___DEFINE_XEN_GUEST_HANDLE(name, type);   \
|       |     ^~~~~~~~~~~~~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:52:41: note: in expansion of macro ‘__DEFINE_XEN_GUEST_HANDLE’
|    52 | #define DEFINE_XEN_GUEST_HANDLE(name)   __DEFINE_XEN_GUEST_HANDLE(name, name)
|       |                                         ^~~~~~~~~~~~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:232:1: note: in expansion of macro ‘DEFINE_XEN_GUEST_HANDLE’
|   232 | DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
|       | ^~~~~~~~~~~~~~~~~~~~~~~
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:178:9: note: previous declaration of ‘__guest_handle_vcpu_guest_context_t’ with type ‘__guest_handle_vcpu_guest_context_t’
|   178 |         __guest_handle_ ## name;                                \
|       |         ^~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:190:5: note: in expansion of macro ‘___DEFINE_XEN_GUEST_HANDLE’
|   190 |     ___DEFINE_XEN_GUEST_HANDLE(name, type);   \
|       |     ^~~~~~~~~~~~~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:192:41: note: in expansion of macro ‘__DEFINE_XEN_GUEST_HANDLE’
|   192 | #define DEFINE_XEN_GUEST_HANDLE(name)   __DEFINE_XEN_GUEST_HANDLE(name, name)
|       |                                         ^~~~~~~~~~~~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:304:1: note: in expansion of macro ‘DEFINE_XEN_GUEST_HANDLE’
|   304 | DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
|       | ^~~~~~~~~~~~~~~~~~~~~~~
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:35:33: error: conflicting types for ‘__guest_handle_const_vcpu_guest_context_t’; have ‘struct <anonymous>’
|    35 |     typedef struct { type *p; } __guest_handle_ ## name
|       |                                 ^~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:51:5: note: in expansion of macro ‘___DEFINE_XEN_GUEST_HANDLE’
|    51 |     ___DEFINE_XEN_GUEST_HANDLE(const_##name, const type)
|       |     ^~~~~~~~~~~~~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:52:41: note: in expansion of macro ‘__DEFINE_XEN_GUEST_HANDLE’
|    52 | #define DEFINE_XEN_GUEST_HANDLE(name)   __DEFINE_XEN_GUEST_HANDLE(name, name)
|       |                                         ^~~~~~~~~~~~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:232:1: note: in expansion of macro ‘DEFINE_XEN_GUEST_HANDLE’
|   232 | DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
|       | ^~~~~~~~~~~~~~~~~~~~~~~
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:178:9: note: previous declaration of ‘__guest_handle_const_vcpu_guest_context_t’ with type ‘__guest_handle_const_vcpu_guest_context_t’
|   178 |         __guest_handle_ ## name;                                \
|       |         ^~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:191:5: note: in expansion of macro ‘___DEFINE_XEN_GUEST_HANDLE’
|   191 |     ___DEFINE_XEN_GUEST_HANDLE(const_##name, const type)
|       |     ^~~~~~~~~~~~~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:192:41: note: in expansion of macro ‘__DEFINE_XEN_GUEST_HANDLE’
|   192 | #define DEFINE_XEN_GUEST_HANDLE(name)   __DEFINE_XEN_GUEST_HANDLE(name, name)
|       |                                         ^~~~~~~~~~~~~~~~~~~~~~~~~
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:304:1: note: in expansion of macro ‘DEFINE_XEN_GUEST_HANDLE’
|   304 | DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
|       | ^~~~~~~~~~~~~~~~~~~~~~~
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:234:8: error: redefinition of ‘struct arch_shared_info’
|   234 | struct arch_shared_info {
|       |        ^~~~~~~~~~~~~~~~
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:345:8: note: originally defined here
|   345 | struct arch_shared_info {
|       |        ^~~~~~~~~~~~~~~~
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:275:33: error: conflicting types for ‘arch_shared_info_t’; have ‘struct arch_shared_info’
|   275 | typedef struct arch_shared_info arch_shared_info_t;
|       |                                 ^~~~~~~~~~~~~~~~~~
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:347:33: note: previous declaration of ‘arch_shared_info_t’ with type ‘arch_shared_info_t’ {aka ‘struct arch_shared_info’}
|   347 | typedef struct arch_shared_info arch_shared_info_t;
|       |                                 ^~~~~~~~~~~~~~~~~~
| In file included from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:282:8: error: redefinition of ‘struct xen_arch_domainconfig’
|   282 | struct xen_arch_domainconfig {
|       |        ^~~~~~~~~~~~~~~~~~~~~
| In file included from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../xen.h:35,
|                  from /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/xen.h:27,
|                  from mk_dsdt.c:21:
| /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi/../../tools/include/xen/arch-x86/../arch-arm.h:317:8: note: originally defined here
|   317 | struct xen_arch_domainconfig {
|       |        ^~~~~~~~~~~~~~~~~~~~~
| cc1: all warnings being treated as errors
| make[7]: *** [Makefile:50: /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/mk_dsdt] Error 1
| make[7]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi'
| make[6]: *** [Makefile:41: acpi] Error 2
| make[6]: *** Waiting for unfinished jobs....
| Parsing libxl_types.idl
| mv -f _libxl.api-for-check.new _libxl.api-for-check
| mv testidl.c.new testidl.c
| make[6]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light'
| make[5]: *** [/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/../../tools/Rules.mk:166: subdir-install-light] Error 2
| make[5]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs'
| make[4]: *** [/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/../../tools/Rules.mk:161: subdirs-install] Error 2
| make[4]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs'
| make[3]: *** [/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/../tools/Rules.mk:166: subdir-install-libs] Error 2
| make[3]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools'
| make[2]: *** [/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/../tools/Rules.mk:161: subdirs-install] Error 2
| make[2]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools'
| make[1]: *** [Makefile:66: install] Error 2
| make[1]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools'
| make: *** [Makefile:140: install-tools] Error 2
| ERROR: oe_runmake failed
| WARNING: /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2581894:202 exit 1 from 'exit 1'
| WARNING: Backtrace (BB generated script):
|       #1: bbfatal_log, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2581894, line 202
|       #2: die, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2581894, line 186
|       #3: oe_runmake, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2581894, line 181
|       #4: do_compile, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2581894, line 175
|       #5: main, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2581894, line 215
NOTE: recipe xen-tools-4.16+stableAUTOINC+f265444922-r0: task do_compile: Failed
ERROR: Task (/home/docker-build/yocto-layers/meta-virtualization/recipes-extended/xen/xen-tools_4.16.bb:do_compile) failed with exit code '1'
NOTE: recipe gettext-0.21-r0: task do_package_write_rpm: Succeeded
NOTE: recipe xen-4.16+stableAUTOINC+f265444922-r0: task do_compile: Succeeded
NOTE: Tasks Summary: Attempted 3396 tasks of which 0 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/docker-build/yocto-layers/meta-virtualization/recipes-extended/xen/xen-tools_4.16.bb:do_compile
Summary: There was 1 WARNING message.
Summary: There were 2 ERROR messages, returning a non-zero exit code.
############  LOGS-END  ############

06:32:18 run qemux86-64: Error

############ LOGS-START ############
spawn bash -c runqemu serialstdio nographic slirp
runqemu - INFO - Running bitbake -e ...
ls: cannot access '/home/docker-build/build/qemux86-64/tmp/deploy/images/qemux86-64/*.qemuboot.conf': No such file or directory
runqemu - ERROR - Command 'ls -t /home/docker-build/build/qemux86-64/tmp/deploy/images/qemux86-64/*.qemuboot.conf' returned non-zero exit status 2.
runqemu - INFO - Cleaning up
runqemu - INFO - Host uptime: 122011.81

tput: No value for $TERM and no -T specified
ERROR-EOF!
############  LOGS-END  ############

06:32:20 Build Complete (2 errors)
The command '/bin/sh -c /build-yocto.sh --dump-log qemux86-64' returned a non-zero code: 2
make: *** [Makefile:19: yocto/kirkstone-qemux86-64-arm64v8] Error 2

[-- Attachment #3: Type: application/octet-stream, Size: 6741 bytes --]

-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/recipe-sysroot-native=  -Wno-format-zero-length -Wmissing-declarations -Wno-declaration-after-statement -Wformat-nonliteral -I. -pthread -Werror -Wmissing-prototypes -I./include -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include -D__XEN_TOOLS__  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include  -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/include -I/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/recipe-sysroot/usr/include/libnl3 -Wshadow -include /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/../../../tools/config.h  -c -o libxl_bootloader.o libxl_bootloader.c
| rm -f /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_pm.aml /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light/ssdt_pm.hex
| make[7]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libacpi'
| make[6]: *** [Makefile:41: acpi] Error 2
| make[6]: *** Waiting for unfinished jobs....
| make[6]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/light'
| make[5]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs'
| make[5]: *** [/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/../../tools/Rules.mk:166: subdir-install-light] Error 2
| make[4]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs'
| make[3]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools'
| make[4]: *** [/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/libs/../../tools/Rules.mk:161: subdirs-install] Error 2
| make[3]: *** [/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/../tools/Rules.mk:166: subdir-install-libs] Error 2
| make[2]: *** [/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools/../tools/Rules.mk:161: subdirs-install] Error 2
| make[2]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools'
| make[1]: *** [Makefile:66: install] Error 2
| make[1]: Leaving directory '/home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/git/tools'
| make: *** [Makefile:140: install-tools] Error 2
| ERROR: oe_runmake failed
| WARNING: /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2240384:202 exit 1 from 'exit 1'
| WARNING: Backtrace (BB generated script):
|       #1: bbfatal_log, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2240384, line 202
|       #2: die, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2240384, line 186
|       #3: oe_runmake, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2240384, line 181
|       #4: do_compile, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2240384, line 175
|       #5: main, /home/docker-build/build/qemux86-64/tmp/work/core2-64-poky-linux/xen-tools/4.16+stableAUTOINC+f265444922-r0/temp/run.do_compile.2240384, line 215
NOTE: recipe xen-tools-4.16+stableAUTOINC+f265444922-r0: task do_compile: Failed
ERROR: Task (/home/docker-build/yocto-layers/meta-virtualization/recipes-extended/xen/xen-tools_4.16.bb:do_compile) failed with exit code '1'
NOTE: recipe xen-4.16+stableAUTOINC+f265444922-r0: task do_compile: Succeeded
NOTE: recipe qemu-system-native-6.2.0-r0: task do_compile: Succeeded
NOTE: Tasks Summary: Attempted 3389 tasks of which 0 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/docker-build/yocto-layers/meta-virtualization/recipes-extended/xen/xen-tools_4.16.bb:do_compile
Summary: There were 2 ERROR messages, returning a non-zero exit code.
############  LOGS-END  ############

23:17:37 run qemux86-64: Error

############ LOGS-START ############
spawn bash -c runqemu serialstdio nographic slirp
runqemu - INFO - Running bitbake -e ...
ls: cannot access '/home/docker-build/build/qemux86-64/tmp/deploy/images/qemux86-64/*.qemuboot.conf': No such file or directory
runqemu - ERROR - Command 'ls -t /home/docker-build/build/qemux86-64/tmp/deploy/images/qemux86-64/*.qemuboot.conf' returned non-zero exit status 2.
runqemu - INFO - Cleaning up
runqemu - INFO - Host uptime: 81239.22

tput: No value for $TERM and no -T specified
ERROR-EOF!
############  LOGS-END  ############

23:17:38 Build Complete (2 errors)
The command '/bin/sh -c /build-yocto.sh --dump-log qemux86-64' returned a non-zero code: 2
make: *** [Makefile:19: yocto/kirkstone-qemux86-64-arm64v8] Error 2

[PATCH v4 2/3] automation: Add a clean rule for containers

[Bertrand Marquis]

Add make clean support to remove the containers from the local docker
registry:
- make clean: remove all images
- clean-yocto/kirkstone-qemuarm: remove yocto kirkstone for qemuarm
image

Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Changes in v4:
- also generate clean rule for CONTAINERS_EXTRA
Changes in v3:
- none
Changes in v2:
- none
Changes in v1:
- patch added
---
 automation/build/Makefile | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/automation/build/Makefile b/automation/build/Makefile
index 72a5335baec1..5b5f10c63ea4 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -28,3 +28,15 @@ all: $(CONTAINERS)
 clean:
 	rm -f yocto/*.dockerfiles
 
+define CLEAN_RULE
+.PHONY: clean-$(1)
+clean-$(1):
+ifneq ($$(shell docker image ls -q $(REGISTRY)/$(subst /,:,$(1))),)
+	docker image rm $(REGISTRY)/$(subst /,:,$(1))
+endif
+
+clean: clean-$(1)
+
+endef
+
+$(eval $(foreach img,$(CONTAINERS) $(CONTAINERS_EXTRA),$(call CLEAN_RULE,$(img))))
-- 
2.25.1

Re: [PATCH v4 2/3] automation: Add a clean rule for containers

[Michal Orzel]

Hi Bertrand,

On 17/11/2022 10:39, Bertrand Marquis wrote:
> 
> 
> Add make clean support to remove the containers from the local docker
> registry:
I have to say that I am a bit scared of adding a clean rule.
make clean is something that can easily sneak into this directory and can
remove the yocto images that we had to spent several hours to build.
The accidental clean can have severe consequences :)

In any case, if we want to add such possibility I would stick to calling always:
make clean-<image>
and remove the option to call "make clean" to remove all the images.

~Michal

Re: [PATCH v4 2/3] automation: Add a clean rule for containers

[Bertrand Marquis]

Hi Michal,

> On 18 Nov 2022, at 08:58, Michal Orzel <michal.orzel@amd.com> wrote:
> 
> Hi Bertrand,
> 
> On 17/11/2022 10:39, Bertrand Marquis wrote:
>> 
>> 
>> Add make clean support to remove the containers from the local docker
>> registry:
> I have to say that I am a bit scared of adding a clean rule.
> make clean is something that can easily sneak into this directory and can
> remove the yocto images that we had to spent several hours to build.
> The accidental clean can have severe consequences :)
> 
> In any case, if we want to add such possibility I would stick to calling always:
> make clean-<image>
> and remove the option to call "make clean" to remove all the images.

Make sense, I will remove those to be automatically called from make clean

Cheers
Bertrand

> 
> ~Michal

[PATCH v4 3/3] automation: Add CI test jobs for Yocto

[Bertrand Marquis]

From: Michal Orzel <michal.orzel@amd.com>

Populate test jobs for Yocto based tests using the provided containers.
Due to the size restrictions, it is currently not possible to split the
build and run tasks, therefore everything is done in a single step.

Test jobs for the supported Yocto targets are generic to avoid the
necessity to add new ones after each Yocto release. The only thing
required to be changed after updating the containers is the variable
YOCTO_VERSION stored in a .yocto-test template.
The .yocto-test-arm64 template is to be used for jobs running on an
arm64 hardware and .yocto-test-x86-64 for x86. Current jobs are using
the arm64 version (x86 templates are here as a provision).

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Changes in v4:
- add .yocto-test for arm64 and x86
- make yocto jobs use arm64 version
Changes in v3:
- patch added
---
 automation/gitlab-ci/test.yaml | 43 ++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index c7e0078e04f1..6ce2fd63eee6 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -54,6 +54,33 @@
   tags:
     - x86_64
 
+.yocto-test:
+  extends: .test-jobs-common
+  script:
+    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` ${YOCTO_BOARD}
+  variables:
+    YOCTO_VERSION: kirkstone
+    CONTAINER: yocto:${YOCTO_VERSION}-${YOCTO_BOARD}-${YOCTO_HOST}
+  artifacts:
+    paths:
+      - 'logs/*'
+    when: always
+  needs: []
+
+.yocto-test-arm64:
+  extends: .yocto-test
+  variables:
+    YOCTO_HOST: arm64v8
+  tags:
+    - arm64
+
+.yocto-test-x86-64:
+  extends: .yocto-test
+  variables:
+    YOCTO_HOST: amd64
+  tags:
+    - x86_64
+
 # Test jobs
 build-each-commit-gcc:
   extends: .test-jobs-common
@@ -188,3 +215,19 @@ qemu-smoke-x86-64-clang-pvh:
     - ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE}
   needs:
     - debian-unstable-clang-debug
+
+# Yocto test jobs
+yocto-qemuarm64:
+  extends: .yocto-test-arm64
+  variables:
+    YOCTO_BOARD: qemuarm64
+
+yocto-qemuarm:
+  extends: .yocto-test-arm64
+  variables:
+    YOCTO_BOARD: qemuarm
+
+yocto-qemux86-64:
+  extends: .yocto-test-arm64
+  variables:
+    YOCTO_BOARD: qemux86-64
-- 
2.25.1