* [PATCH v6 0/3] Yocto Gitlab CI
@ 2022-12-01 14:25 Bertrand Marquis
2022-12-01 14:25 ` [PATCH v6 1/3] automation: Create Yocto docker images Bertrand Marquis
` (3 more replies)
0 siblings, 4 replies; 10+ messages in thread
From: Bertrand Marquis @ 2022-12-01 14:25 UTC (permalink / raw)
To: xen-devel
Cc: michal.orzel, Andrew Cooper, George Dunlap, Jan Beulich,
Julien Grall, Stefano Stabellini, Wei Liu, Doug Goldstein
This patch series is a first attempt to check if we could use Yocto in
gitlab ci to build and run xen on qemu for arm, arm64 and x86.
The first patch is introducing a container template from which container
files can be generated for all combinations we want to support (qemu
arm, arm64 and x86 targets and hosts x86_64 and arm64).
It is also introducing a generic build script (build-yocto.sh) that is
used to create, build and run a Yocto project.
The second patch is adding a way to easily clean locally created
containers.
The third patch is introducing some gitlab CI templates and jobs so that
we can test a Xen source tree in gitlab-CI using Yocto.
This has been tested on a x86 host machine and on an arm host machine
(with mk_dsdt.c fix).
Changes in v6:
- fix qemux86 errors on arm hosts
- added some comments
- fix Makefile to not call docker a lot during each parse
- several fixes and improvments described in each patch
Changes in v5:
- automatic cleanup of generated docker files using INTERMEDIATE
- typo fixes
- rebase on Stefano's next branch
- remove clean rule cleaning all containers
Changes in v4:
- rework the container generation to support multiple hosts
- rework the container generation to use a single template for all
docker files (make process is generating the docker files).
Changes in v3:
- limit number of jobs in yocto
- do not copy build script inside container
- add patch from Michal to create gitlab jobs
Changes in v2:
- remove gitignore patch which was merged
- add a --dump-log support in build-yocto.sh script and use it during
container creation to see the error logs.
*** BLURB HERE ***
Bertrand Marquis (2):
automation: Create Yocto docker images
automation: Add a clean rule for containers
Michal Orzel (1):
automation: Add CI test jobs for Yocto
.gitignore | 1 +
automation/build/Makefile | 26 +-
automation/build/yocto/build-yocto.sh | 356 +++++++++++++++++++++
automation/build/yocto/yocto.dockerfile.in | 114 +++++++
automation/build/yocto/yocto.inc | 44 +++
automation/gitlab-ci/test.yaml | 45 +++
6 files changed, 583 insertions(+), 3 deletions(-)
create mode 100755 automation/build/yocto/build-yocto.sh
create mode 100644 automation/build/yocto/yocto.dockerfile.in
create mode 100644 automation/build/yocto/yocto.inc
--
2.25.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v6 1/3] automation: Create Yocto docker images
2022-12-01 14:25 [PATCH v6 0/3] Yocto Gitlab CI Bertrand Marquis
@ 2022-12-01 14:25 ` Bertrand Marquis
2022-12-01 21:09 ` Stefano Stabellini
2022-12-02 11:03 ` Michal Orzel
2022-12-01 14:25 ` [PATCH v6 2/3] automation: Add a clean rule for containers Bertrand Marquis
` (2 subsequent siblings)
3 siblings, 2 replies; 10+ messages in thread
From: Bertrand Marquis @ 2022-12-01 14:25 UTC (permalink / raw)
To: xen-devel
Cc: michal.orzel, Andrew Cooper, George Dunlap, Jan Beulich,
Julien Grall, Stefano Stabellini, Wei Liu, Doug Goldstein
Add containers suitable to run yocto kirkstone build based on ubuntu
22.04. It contains all packages required by Yocto and a checkout of the
layers required to build Xen with Yocto.
Add a generic docker image template to be used to automatically generate
docker files for different configurations:
- specific yocto version
- different targets (qemu arm, arm64 and x86)
- different host platforms (x86 or arm64)
During a call to 'make all', only the images for the host platform we
run it on in the CI is generated.
If needed, images for an other host platform can be generated manually
by calling the right make target (see make help).
Add a build script to build and run xen on qemu using Yocto.
The script supports arm32, arm64 and x86_64 and checks that dom0 is
properly booting. At this stage this does not run any guest on top of
dom0. The script is to be executed in one of the docker images to build
and run a system using a Xen source tree.
Add automation/build/yocto/*.dockerfile to gitignore as those files are
generated.
Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Changes in v6:
- move include of yocto.inc to keep help the default make target
- add yocto/*.dockerfile to gitignore
- fix typo {} instead of ()
- fix clean rule to remove s to dockerfile
- tell make to evaluate CONTAINER right away
- add example command in comment in yocto.inc
- fix compilation of qemux86 on arm by switching to AUTOREV to use the
latest version of Xen which includes fixes required.
Changes in v5:
- typo fixes
- add INTERMEDIATE to remove intermediate dockerfile
- use container architecture
- use full words in foreach loops
- fix shellcheck findings in build-yocto.sh
- rebase on sstabellini next branch
Changes in v4:
- Rework the system to have one dockerfile template from which make will
generate the required dockerfiles for the wanted configuration
- add support for different host architectures
- Merge the generation system into one single dockerfile
- Merge patches 1 and 2 in a single patch
- Introduce CONTAINERS_EXTRA to have extra containers not built by
default (for those not used by CI but useful to users)
Changes in v3:
- limit number of jobs in yocto by default to 8 and add --num-jobs
option to the script to set a custom number of jobs
- do not copy anymore the build-yocto.sh script inside the main image so
that the current one in the repository is used when running
Changes in v2:
- add a --dump-log command line option to build-yocto.sh script to dump
the logs if an error occurs.
Changes in v1:
- add --image command line argument to build-yocto.sh to allow building
something different than xen-image-minimal.
- modify dockerfile to have one layer per line and make it easier to add
other. I kept the for loop to keep the number of docker steps lower
- update commit message to warn that no guest are tested.
- fix build-yocto script to properly return with an error if expect
script ends up in timeout or EOF.
---
.gitignore | 1 +
automation/build/Makefile | 16 +-
automation/build/yocto/build-yocto.sh | 356 +++++++++++++++++++++
automation/build/yocto/yocto.dockerfile.in | 114 +++++++
automation/build/yocto/yocto.inc | 44 +++
5 files changed, 528 insertions(+), 3 deletions(-)
create mode 100755 automation/build/yocto/build-yocto.sh
create mode 100644 automation/build/yocto/yocto.dockerfile.in
create mode 100644 automation/build/yocto/yocto.inc
diff --git a/.gitignore b/.gitignore
index ea3243af9dde..ce7caa56af6d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -38,6 +38,7 @@ dist
stubdom/*.tar.gz
autom4te.cache/
+automation/build/yocto/*.dockerfile
build-*/
config.log
config.status
diff --git a/automation/build/Makefile b/automation/build/Makefile
index a4b2b85178cf..c4aaa747ffdf 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -1,20 +1,30 @@
# the base of where these containers will appear
REGISTRY := registry.gitlab.com/xen-project/xen
-CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
+CONTAINERS := $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard */*.dockerfile)))
+CONTAINERS_EXTRA =
DOCKER_CMD ?= docker
help:
@echo "Builds containers for building Xen based on different distros"
@echo "To build one run 'make DISTRO/VERSION'. Available containers:"
- @$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
+ @$(foreach file,$(sort $(CONTAINERS)),echo $(file);)
+ @echo "Extra containers (not built using make all):"
+ @$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo $(file);)
@echo "To push container builds, set the env var PUSH"
+include yocto/yocto.inc
+
%: %.dockerfile ## Builds containers
$(DOCKER_CMD) build -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
@if [ ! -z $${PUSH+x} ]; then \
$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
fi
-.PHONY: all
+.PHONY: all clean
all: $(CONTAINERS)
+
+# Remove generated dockerfiles for yocto
+clean:
+ rm -f yocto/*.dockerfile
+
diff --git a/automation/build/yocto/build-yocto.sh b/automation/build/yocto/build-yocto.sh
new file mode 100755
index 000000000000..3601cebc3cff
--- /dev/null
+++ b/automation/build/yocto/build-yocto.sh
@@ -0,0 +1,356 @@
+#!/bin/bash
+#
+# Yocto meta virtualization build and run script
+#
+# This script is building Yocto xen-image-minimal for qemu targets and run
+# them using runqemu inside yocto to check that dom0 is booting properly.
+# The build is using a local xen source tree so that specific patches can be
+# tested.
+# In order to optimize the build time, a build cache is used so that only xen
+# packages and its dependencies are rebuilt (qemu and final image mainly).
+#
+# get command error even when piped.
+set -o pipefail
+
+# Directories
+YOCTODIR="$HOME/yocto-layers"
+CACHEDIR="$HOME/yocto-cache"
+LOGDIR="$HOME/logs"
+XENDIR="$HOME/xen"
+BUILDDIR="$HOME/build"
+
+# what yocto bsp we support
+TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
+VERBOSE="n"
+TARGETLIST=""
+BUILDJOBS="8"
+
+# actions to do
+do_clean="n"
+do_build="y"
+do_run="y"
+do_localsrc="n"
+do_dump="n"
+build_result=0
+
+# layers to include in the project
+build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
+ meta-openembedded/meta-oe meta-openembedded/meta-python \
+ meta-openembedded/meta-filesystems \
+ meta-openembedded/meta-networking meta-virtualization"
+
+# yocto image to build
+build_image="xen-image-minimal"
+
+function print_progress() {
+ echo -n "$(date +%T) $*"
+}
+
+function run_task() {
+ local task_name="$1"
+ local task_target="$2"
+
+ task_log="${task_name//project_}-${task_target}"
+
+ mkdir -p "${LOGDIR}"
+ print_progress
+ echo -n "${task_name//project_} ${task_target}: "
+ if [ "${VERBOSE}" = "n" ]; then
+ "$@" > "${LOGDIR}/${task_log}.log" 2>&1
+ else
+ "$@" 2>&1 | tee "${LOGDIR}/${task_log}.log"
+ fi
+
+ if [ ${?} -ne 0 ]; then
+ echo "Error"
+ build_result=$((build_result+1))
+ if [ "${do_dump}" = "y" ]; then
+ echo
+ echo "############ LOGS-START ############"
+ cat "${LOGDIR}/${task_log}.log"
+ echo "############ LOGS-END ############"
+ echo
+ fi
+ return 1
+ else
+ echo "OK"
+ return 0
+ fi
+}
+
+function project_create() {
+ target="${1:?}"
+ destdir="${BUILDDIR}/${target}"
+
+ (
+ # init yocto project
+ source "${YOCTODIR}/poky/oe-init-build-env" "${destdir}"
+
+ # add needed layers
+ for layer in ${build_layerlist}; do
+ bitbake-layers add-layer "${YOCTODIR}/${layer}" || exit 1
+ done
+ ) || return 1
+
+ # Detect latest version available in Yocto and use it instead of default
+ # one.
+ XENVERS=$(grep -e "^XEN_REL" \
+ "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
+ 2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
+ | sed -e "s/.*=//" | sort -V | tail -n 1)
+
+ # customize project configuration
+ cat <<EOF >> "${destdir}/conf/local.conf"
+# Yocto BSP
+MACHINE = "${target}"
+
+# Use local cache to reuse previous builds results
+SSTATE_DIR = "${CACHEDIR}/sstate-cache"
+DL_DIR = "${CACHEDIR}/downloads"
+
+# Enable xen and virtualization
+DISTRO_FEATURES = " virtualization xen ipv4"
+
+# Speed up run by not generating ssh host keys
+IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
+
+# Save some disk space
+INHERIT += "rm_work"
+
+# Reduce number of jobs
+BB_NUMBER_THREADS="${BUILDJOBS}"
+
+# Use latest Xen version
+PREFERRED_VERSION:pn-xen = "${XENVERS}%"
+PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
+
+# Use autorev for now as Xen SHA used by latest yocto recipe for Xen does not
+# include fixes required to build x86 on arm
+SRCREV:pn-xen = "\${AUTOREV}"
+SRCREV:pn-xen-tools = "\${AUTOREV}"
+
+# Disable all QA errors as the recipe is not up to date with changes in Xen
+# when we use local sources
+ERROR_QA:pn-xen = "arch"
+ERROR_QA:pn-xen-tools = "arch"
+
+EOF
+
+ if [ "${do_localsrc}" = "y" ]; then
+ XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
+ XENSUB=$(basename "$(realpath -m "${XENDIR}")")
+
+ cat <<EOF >> "${destdir}/conf/local.conf"
+# Use local sources for xen and xen-tools
+FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
+FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
+
+SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
+SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
+
+S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
+S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
+
+SRCPV:pn-xen = "1"
+SRCPV:pn-xen-tools = "1"
+
+EOF
+ fi
+}
+
+function project_build() {
+ target="${1:?}"
+ destdir="${BUILDDIR}/${target}"
+
+ (
+ source "${YOCTODIR}/poky/oe-init-build-env" "${destdir}"
+
+ bitbake "${build_image}" || exit 1
+ ) || return 1
+}
+
+function project_clean() {
+ target="${1:?}"
+ destdir="${BUILDDIR}/${target}"
+
+ rm -rf "${destdir}"
+}
+
+function project_run() {
+ target="${1:?}"
+ destdir="${BUILDDIR}/${target}"
+ (
+ source "${YOCTODIR}/poky/oe-init-build-env" "${destdir}" > /dev/null 2>&1
+
+ /usr/bin/expect <<EOF
+set timeout 1000
+spawn bash -c "runqemu serialstdio nographic slirp"
+
+expect_after {
+ -re "(.*)\r" {
+ exp_continue
+ }
+ timeout {send_user "ERROR-Timeout!\n"; exit 1}
+ eof {send_user "ERROR-EOF!\n"; exit 1}
+}
+
+# wait initial login
+expect -re ".* login: "
+send "root\r"
+expect -re "root@.*# "
+
+EOF
+ exit $?
+ ) || return 1
+}
+
+function help() {
+ cat <<EOF
+Usage: ${0} [TARGET1] [TARGET2]
+
+This script is build the yocto xen-image-minimal for different qemu targets
+and is running it after.
+Without any target specified, all supported targets are done.
+
+Options:
+ -h, --help Print this help
+ -v, --verbose Verbose build
+ --list-target List supported targets
+ --clean Clean existing project before starting
+ --no-build Do not build (to run an already built project)
+ --no-run Do not run
+ --num-jobs=NUM Define the number of parallel jobs in Yocto.
+ Default: ${BUILDJOBS}
+ --dump-log On error, dump the logs on the console
+ --image=IMG Yocto image or package to build
+ Default: xen-image-minimal
+ --xen-dir=DIR path to xen hypervisor source tree
+ if not provide, normal yocto version of xen is built
+ Default: ${XENDIR}
+ --out-dir=DIR directory where to create the projectss
+ Default: ${BUILDDIR}
+ --log-dir=DIR directory to store logs
+ Default: ${LOGDIR}
+ --cache-dir=DIR directory where to take and store build cache
+ Default: ${CACHEDIR}
+ --layer-dir=DIR directory containing the checkout of yocto layers
+ Default: ${YOCTODIR}
+EOF
+}
+
+for OPTION in "$@"
+do
+ case ${OPTION} in
+ -h|--help)
+ help
+ exit 0
+ ;;
+ -v|--verbose)
+ VERBOSE="y"
+ ;;
+ --list-targets)
+ echo "${TARGET_SUPPORTED}"
+ exit 0
+ ;;
+ --clean)
+ do_clean="y"
+ ;;
+ --no-build)
+ do_build="n"
+ ;;
+ --no-run)
+ do_run="n"
+ ;;
+ --dump-log)
+ do_dump="y"
+ ;;
+ --num-jobs=*)
+ BUILDJOBS="${OPTION#*=}"
+ ;;
+ --image=*)
+ build_image="${OPTION#*=}"
+ ;;
+ --xen-dir=*)
+ XENDIR="${OPTION#*=}"
+ if [ ! -e "${XENDIR}/xen/Makefile" ]; then
+ echo "No Xen source tree in ${XENDIR}"
+ exit 1
+ fi
+ do_localsrc="y"
+ ;;
+ --out-dir=*)
+ BUILDDIR="${OPTION#*=}"
+ ;;
+ --log-dir=*)
+ LOGDIR="${OPTION#*=}"
+ ;;
+ --cache-dir=*)
+ CACHEDIR="${OPTION#*=}"
+ ;;
+ --layer-dir=*)
+ YOCTODIR="${OPTION#*=}"
+ ;;
+ --*)
+ echo "Invalid option ${OPTION}"
+ help
+ exit 1
+ ;;
+ *)
+ if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
+ TARGETLIST="${TARGETLIST} ${OPTION}"
+ else
+ echo "Unsupported target ${OPTION}"
+ exit 1
+ fi
+ ;;
+ esac
+done
+
+# if no target is specified build all targets
+if [ -z "${TARGETLIST}" ]; then
+ TARGETLIST="${TARGET_SUPPORTED}"
+fi
+
+mkdir -p "${CACHEDIR}"
+mkdir -p "${LOGDIR}"
+mkdir -p "${BUILDDIR}"
+
+# Make sure we have an absolute path
+YOCTODIR=$(realpath -m "${YOCTODIR}")
+CACHEDIR=$(realpath -m "${CACHEDIR}")
+BUILDDIR=$(realpath -m "${BUILDDIR}")
+LOGDIR=$(realpath -m "${LOGDIR}")
+if [ "${do_localsrc}" = "y" ]; then
+ XENDIR=$(realpath -m "${XENDIR}")
+fi
+
+# Check that we have all the layers we need
+for f in ${build_layerlist}; do
+ if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
+ echo "Layer ${f} missing in ${YOCTODIR}"
+ exit 1
+ fi
+done
+
+for f in ${TARGETLIST}; do
+ if [ "${do_clean}" = "y" ]; then
+ run_task project_clean "${f}"
+ fi
+ if [ ! -f "${BUILDDIR}/${f}/conf/local.conf" ]; then
+ run_task project_create "${f}"
+ fi
+ if [ -f "${BUILDDIR}/${f}/conf/local.conf" ]; then
+ if [ "${do_build}" = "y" ]; then
+ run_task project_build "${f}"
+ fi
+ if [ "${do_run}" = "y" ]; then
+ run_task project_run "${f}"
+ fi
+
+ fi
+done
+
+print_progress "Build Complete (${build_result} errors)"
+echo
+exit ${build_result}
+
diff --git a/automation/build/yocto/yocto.dockerfile.in b/automation/build/yocto/yocto.dockerfile.in
new file mode 100644
index 000000000000..b0892d420377
--- /dev/null
+++ b/automation/build/yocto/yocto.dockerfile.in
@@ -0,0 +1,114 @@
+# Docker file to create an environment to build yocto with virtualization
+#
+# Arguments that can be passed during image creation using --build-arg:
+# "host_uid=$(id -u)": to use current user uid for build user in the image
+# "host_gid=$(id -g)": to use current user gid for build user in the image
+# "ubuntu_version=VERS": to select the ubuntu version number
+
+# Use standard ubuntu minimal.
+ARG ubuntu_version=22.04
+From ##DOCKERPLAT##ubuntu:$ubuntu_version AS base
+LABEL maintainer.name="The Xen Project " \
+ maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Install minimal ubuntu requirements for yocto and other tools we need.
+# See https://docs.yoctoproject.org/4.0.1/brief-yoctoprojectqs/index.html#build-host-packages
+RUN apt-get update && \
+ apt-get --quiet --yes install \
+ gawk \
+ wget \
+ git \
+ diffstat \
+ unzip \
+ texinfo \
+ gcc \
+ build-essential \
+ chrpath \
+ socat \
+ cpio \
+ python3 \
+ python3-pip \
+ python3-pexpect \
+ xz-utils \
+ debianutils \
+ iputils-ping \
+ python3-git \
+ python3-jinja2 \
+ libegl1-mesa \
+ libsdl1.2-dev \
+ python3-subunit \
+ mesa-common-dev \
+ zstd \
+ liblz4-tool \
+ file \
+ vim \
+ bison \
+ expect \
+ locales \
+ liblz4-tool \
+ zstd \
+ openssl \
+ libssl3 \
+ ca-certificates \
+ && \
+ apt-get autoremove -y && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
+
+# Use bash as shell.
+RUN rm /bin/sh && ln -s bash /bin/sh
+
+# Fix local for yocto.
+RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
+ LANG=en_US.UTF-8
+ENV LANG en_US.UTF-8
+ENV LC_ALL en_US.UTF-8
+
+# Create a user for the build (we don't want to build as root).
+ENV USER_NAME docker-build
+ARG host_uid=1000
+ARG host_gid=1000
+RUN groupadd -g $host_gid $USER_NAME && \
+ useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
+
+# Switch to our user instead of root and start in its home.
+USER $USER_NAME
+WORKDIR /home/$USER_NAME
+
+# Create needed directories
+RUN mkdir -p /home/$USER_NAME/yocto-layers \
+ /home/$USER_NAME/yocto-cache \
+ /home/$USER_NAME/logs \
+ /home/$USER_NAME/bin \
+ /home/$USER_NAME/xen && \
+ chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
+
+# clone yocto repositories we need.
+RUN for rep in \
+ https://github.com/openembedded/meta-openembedded \
+ https://git.yoctoproject.org/poky \
+ https://git.yoctoproject.org/meta-virtualization \
+ ; do \
+ git -C /home/$USER_NAME/yocto-layers \
+ clone -b ##YOCTOVERSION## --single-branch $rep; \
+ done
+
+# The builder stage is building an initial cache state that we include in the
+# final image.
+From base AS builder
+
+# This step can take one to several hours depending on your download bandwith
+# and the speed of your computer.
+COPY ./build-yocto.sh /
+RUN /build-yocto.sh --dump-log ##YOCTOTARGET##
+
+From base
+
+# Only copy the cache status.
+COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
+
+LABEL maintainer.name="The Xen Project " \
+ maintainer.email="xen-devel@lists.xenproject.org"
+
diff --git a/automation/build/yocto/yocto.inc b/automation/build/yocto/yocto.inc
new file mode 100644
index 000000000000..e4328ec7f378
--- /dev/null
+++ b/automation/build/yocto/yocto.inc
@@ -0,0 +1,44 @@
+# This makefile generates the docker files for Yocto builds.
+# The containers for the current architecture are the one built using make all.
+# To build containers for a different architecture, you need to call make for
+# the image you want explicitely.
+# The containers are named this way:
+# YOCTOVERSION-TARGET for x86_64 hosts
+# YOCTOVERSION-TARGET-arm64v8 for arm64 hosts
+# For example you can build an arm64 container with the following command:
+# make yocto/kirkstone-qemuarm64-arm64v8
+
+# Yocto versions we are currently using.
+YOCTO_VERSION = kirkstone
+
+# Yocto BSPs we want to build for.
+YOCTO_TARGETS = qemuarm64 qemuarm qemux86-64
+
+# Supported container architectures.
+YOCTO_ARCHS = amd64 arm64v8
+
+# Architecture we want to use in gitlab CI (depends on runners arch).
+CI_ARCH = arm64v8
+
+define GEN_DOCKER
+# Make all is generating architecture we use in the CI.
+ifeq ($(CI_ARCH),$(3))
+CONTAINERS += yocto/$(1)-$(2)$(4)
+else
+CONTAINERS_EXTRA += yocto/$(1)-$(2)$(4)
+endif
+
+.INTERMEDIATE: yocto/$(1)-$(2)$(4).dockerfile
+
+yocto/$(1)-$(2)$(4).dockerfile: yocto/yocto.dockerfile.in
+ @cat $$< | \
+ sed -e "s,##YOCTOVERSION##,$(1),g" | \
+ sed -e "s,##YOCTOTARGET##,$(2),g" | \
+ sed -e "s,##DOCKERPLAT##,$(3)/,g" > $$@
+
+endef
+
+$(eval $(foreach version,$(YOCTO_VERSION),\
+ $(foreach target,$(YOCTO_TARGETS),\
+ $(foreach arch,$(YOCTO_ARCHS),\
+ $(call GEN_DOCKER,$(version),$(target),$(arch),$(if $(filter amd64,$(arch)),,-$(arch)))))))
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v6 2/3] automation: Add a clean rule for containers
2022-12-01 14:25 [PATCH v6 0/3] Yocto Gitlab CI Bertrand Marquis
2022-12-01 14:25 ` [PATCH v6 1/3] automation: Create Yocto docker images Bertrand Marquis
@ 2022-12-01 14:25 ` Bertrand Marquis
2022-12-01 21:10 ` Stefano Stabellini
2022-12-02 11:01 ` Michal Orzel
2022-12-01 14:25 ` [PATCH v6 3/3] automation: Add CI test jobs for Yocto Bertrand Marquis
2022-12-01 21:11 ` [PATCH v6 0/3] Yocto Gitlab CI Stefano Stabellini
3 siblings, 2 replies; 10+ messages in thread
From: Bertrand Marquis @ 2022-12-01 14:25 UTC (permalink / raw)
To: xen-devel; +Cc: michal.orzel, Doug Goldstein, Stefano Stabellini
Add make clean support to remove the containers from the local docker
registry.
make clean-<image_name> must be called to remove an image:
make clean-yocto/kirkstone-qemuarm: remove yocto kirkstone for qemuarm
image
Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Changes in v6:
- Rework clean rule to prevent calling docker during make parsing
Changes in v5:
- remove cleaning of all images using make clean
Changes in v4:
- also generate clean rule for CONTAINERS_EXTRA
Changes in v3:
- none
Changes in v2:
- none
Changes in v1:
- patch added
---
automation/build/Makefile | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/automation/build/Makefile b/automation/build/Makefile
index c4aaa747ffdf..f1d56ce532a1 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -28,3 +28,13 @@ all: $(CONTAINERS)
clean:
rm -f yocto/*.dockerfile
+define CLEAN_RULE
+.PHONY: clean-$(1)
+clean-$(1):
+ if [ -n "$$$$(docker image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
+ docker image rm $(REGISTRY)/$(subst /,:,$(1)); \
+ fi
+
+endef
+
+$(eval $(foreach img,$(CONTAINERS) $(CONTAINERS_EXTRA),$(call CLEAN_RULE,$(img))))
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v6 3/3] automation: Add CI test jobs for Yocto
2022-12-01 14:25 [PATCH v6 0/3] Yocto Gitlab CI Bertrand Marquis
2022-12-01 14:25 ` [PATCH v6 1/3] automation: Create Yocto docker images Bertrand Marquis
2022-12-01 14:25 ` [PATCH v6 2/3] automation: Add a clean rule for containers Bertrand Marquis
@ 2022-12-01 14:25 ` Bertrand Marquis
2022-12-01 21:11 ` Stefano Stabellini
2022-12-01 21:11 ` [PATCH v6 0/3] Yocto Gitlab CI Stefano Stabellini
3 siblings, 1 reply; 10+ messages in thread
From: Bertrand Marquis @ 2022-12-01 14:25 UTC (permalink / raw)
To: xen-devel; +Cc: michal.orzel, Doug Goldstein, Stefano Stabellini
From: Michal Orzel <michal.orzel@amd.com>
Populate test jobs for Yocto based tests using the provided containers.
Due to the size restrictions, it is currently not possible to split the
build and run tasks, therefore everything is done in a single step.
Test jobs for the supported Yocto targets are generic to avoid the
necessity to add new ones after each Yocto release. The only thing
required to be changed after updating the containers is the variable
YOCTO_VERSION stored in a .yocto-test template.
Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
Changes in v6:
- add comment explaining why the unused x86 yocto entry is there
Changes in v5:
- none
Changes in v4:
- add .yocto-test for arm64 and x86
- make yocto jobs use arm64 version
Changes in v3:
- patch added
---
automation/gitlab-ci/test.yaml | 45 ++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index c7e0078e04f1..2d57f4feb74a 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -54,6 +54,35 @@
tags:
- x86_64
+.yocto-test:
+ extends: .test-jobs-common
+ script:
+ - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` ${YOCTO_BOARD}
+ variables:
+ YOCTO_VERSION: kirkstone
+ CONTAINER: yocto:${YOCTO_VERSION}-${YOCTO_BOARD}-${YOCTO_HOST}
+ artifacts:
+ paths:
+ - 'logs/*'
+ when: always
+ needs: []
+
+.yocto-test-arm64:
+ extends: .yocto-test
+ variables:
+ YOCTO_HOST: arm64v8
+ tags:
+ - arm64
+
+# This is not used by any test job as we only run Yocto on arm based machines.
+# Keep it here so that someone having x86 hardware can easily add jobs.
+.yocto-test-x86-64:
+ extends: .yocto-test
+ variables:
+ YOCTO_HOST: amd64
+ tags:
+ - x86_64
+
# Test jobs
build-each-commit-gcc:
extends: .test-jobs-common
@@ -188,3 +217,19 @@ qemu-smoke-x86-64-clang-pvh:
- ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE}
needs:
- debian-unstable-clang-debug
+
+# Yocto test jobs
+yocto-qemuarm64:
+ extends: .yocto-test-arm64
+ variables:
+ YOCTO_BOARD: qemuarm64
+
+yocto-qemuarm:
+ extends: .yocto-test-arm64
+ variables:
+ YOCTO_BOARD: qemuarm
+
+yocto-qemux86-64:
+ extends: .yocto-test-arm64
+ variables:
+ YOCTO_BOARD: qemux86-64
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v6 1/3] automation: Create Yocto docker images
2022-12-01 14:25 ` [PATCH v6 1/3] automation: Create Yocto docker images Bertrand Marquis
@ 2022-12-01 21:09 ` Stefano Stabellini
2022-12-02 11:03 ` Michal Orzel
1 sibling, 0 replies; 10+ messages in thread
From: Stefano Stabellini @ 2022-12-01 21:09 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel, michal.orzel, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu,
Doug Goldstein
On Thu, 1 Dec 2022, Bertrand Marquis wrote:
> Add containers suitable to run yocto kirkstone build based on ubuntu
> 22.04. It contains all packages required by Yocto and a checkout of the
> layers required to build Xen with Yocto.
>
> Add a generic docker image template to be used to automatically generate
> docker files for different configurations:
> - specific yocto version
> - different targets (qemu arm, arm64 and x86)
> - different host platforms (x86 or arm64)
>
> During a call to 'make all', only the images for the host platform we
> run it on in the CI is generated.
> If needed, images for an other host platform can be generated manually
> by calling the right make target (see make help).
>
> Add a build script to build and run xen on qemu using Yocto.
> The script supports arm32, arm64 and x86_64 and checks that dom0 is
> properly booting. At this stage this does not run any guest on top of
> dom0. The script is to be executed in one of the docker images to build
> and run a system using a Xen source tree.
>
> Add automation/build/yocto/*.dockerfile to gitignore as those files are
> generated.
>
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> Changes in v6:
> - move include of yocto.inc to keep help the default make target
> - add yocto/*.dockerfile to gitignore
> - fix typo {} instead of ()
> - fix clean rule to remove s to dockerfile
> - tell make to evaluate CONTAINER right away
> - add example command in comment in yocto.inc
> - fix compilation of qemux86 on arm by switching to AUTOREV to use the
> latest version of Xen which includes fixes required.
> Changes in v5:
> - typo fixes
> - add INTERMEDIATE to remove intermediate dockerfile
> - use container architecture
> - use full words in foreach loops
> - fix shellcheck findings in build-yocto.sh
> - rebase on sstabellini next branch
> Changes in v4:
> - Rework the system to have one dockerfile template from which make will
> generate the required dockerfiles for the wanted configuration
> - add support for different host architectures
> - Merge the generation system into one single dockerfile
> - Merge patches 1 and 2 in a single patch
> - Introduce CONTAINERS_EXTRA to have extra containers not built by
> default (for those not used by CI but useful to users)
> Changes in v3:
> - limit number of jobs in yocto by default to 8 and add --num-jobs
> option to the script to set a custom number of jobs
> - do not copy anymore the build-yocto.sh script inside the main image so
> that the current one in the repository is used when running
> Changes in v2:
> - add a --dump-log command line option to build-yocto.sh script to dump
> the logs if an error occurs.
> Changes in v1:
> - add --image command line argument to build-yocto.sh to allow building
> something different than xen-image-minimal.
> - modify dockerfile to have one layer per line and make it easier to add
> other. I kept the for loop to keep the number of docker steps lower
> - update commit message to warn that no guest are tested.
> - fix build-yocto script to properly return with an error if expect
> script ends up in timeout or EOF.
> ---
> .gitignore | 1 +
> automation/build/Makefile | 16 +-
> automation/build/yocto/build-yocto.sh | 356 +++++++++++++++++++++
> automation/build/yocto/yocto.dockerfile.in | 114 +++++++
> automation/build/yocto/yocto.inc | 44 +++
> 5 files changed, 528 insertions(+), 3 deletions(-)
> create mode 100755 automation/build/yocto/build-yocto.sh
> create mode 100644 automation/build/yocto/yocto.dockerfile.in
> create mode 100644 automation/build/yocto/yocto.inc
>
> diff --git a/.gitignore b/.gitignore
> index ea3243af9dde..ce7caa56af6d 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -38,6 +38,7 @@ dist
> stubdom/*.tar.gz
>
> autom4te.cache/
> +automation/build/yocto/*.dockerfile
> build-*/
> config.log
> config.status
> diff --git a/automation/build/Makefile b/automation/build/Makefile
> index a4b2b85178cf..c4aaa747ffdf 100644
> --- a/automation/build/Makefile
> +++ b/automation/build/Makefile
> @@ -1,20 +1,30 @@
>
> # the base of where these containers will appear
> REGISTRY := registry.gitlab.com/xen-project/xen
> -CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
> +CONTAINERS := $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard */*.dockerfile)))
> +CONTAINERS_EXTRA =
> DOCKER_CMD ?= docker
>
> help:
> @echo "Builds containers for building Xen based on different distros"
> @echo "To build one run 'make DISTRO/VERSION'. Available containers:"
> - @$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
> + @$(foreach file,$(sort $(CONTAINERS)),echo $(file);)
> + @echo "Extra containers (not built using make all):"
> + @$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo $(file);)
> @echo "To push container builds, set the env var PUSH"
>
> +include yocto/yocto.inc
> +
> %: %.dockerfile ## Builds containers
> $(DOCKER_CMD) build -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
> @if [ ! -z $${PUSH+x} ]; then \
> $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
> fi
>
> -.PHONY: all
> +.PHONY: all clean
> all: $(CONTAINERS)
> +
> +# Remove generated dockerfiles for yocto
> +clean:
> + rm -f yocto/*.dockerfile
> +
> diff --git a/automation/build/yocto/build-yocto.sh b/automation/build/yocto/build-yocto.sh
> new file mode 100755
> index 000000000000..3601cebc3cff
> --- /dev/null
> +++ b/automation/build/yocto/build-yocto.sh
> @@ -0,0 +1,356 @@
> +#!/bin/bash
> +#
> +# Yocto meta virtualization build and run script
> +#
> +# This script is building Yocto xen-image-minimal for qemu targets and run
> +# them using runqemu inside yocto to check that dom0 is booting properly.
> +# The build is using a local xen source tree so that specific patches can be
> +# tested.
> +# In order to optimize the build time, a build cache is used so that only xen
> +# packages and its dependencies are rebuilt (qemu and final image mainly).
> +#
> +# get command error even when piped.
> +set -o pipefail
> +
> +# Directories
> +YOCTODIR="$HOME/yocto-layers"
> +CACHEDIR="$HOME/yocto-cache"
> +LOGDIR="$HOME/logs"
> +XENDIR="$HOME/xen"
> +BUILDDIR="$HOME/build"
> +
> +# what yocto bsp we support
> +TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
> +VERBOSE="n"
> +TARGETLIST=""
> +BUILDJOBS="8"
> +
> +# actions to do
> +do_clean="n"
> +do_build="y"
> +do_run="y"
> +do_localsrc="n"
> +do_dump="n"
> +build_result=0
> +
> +# layers to include in the project
> +build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
> + meta-openembedded/meta-oe meta-openembedded/meta-python \
> + meta-openembedded/meta-filesystems \
> + meta-openembedded/meta-networking meta-virtualization"
> +
> +# yocto image to build
> +build_image="xen-image-minimal"
> +
> +function print_progress() {
> + echo -n "$(date +%T) $*"
> +}
> +
> +function run_task() {
> + local task_name="$1"
> + local task_target="$2"
> +
> + task_log="${task_name//project_}-${task_target}"
> +
> + mkdir -p "${LOGDIR}"
> + print_progress
> + echo -n "${task_name//project_} ${task_target}: "
> + if [ "${VERBOSE}" = "n" ]; then
> + "$@" > "${LOGDIR}/${task_log}.log" 2>&1
> + else
> + "$@" 2>&1 | tee "${LOGDIR}/${task_log}.log"
> + fi
> +
> + if [ ${?} -ne 0 ]; then
> + echo "Error"
> + build_result=$((build_result+1))
> + if [ "${do_dump}" = "y" ]; then
> + echo
> + echo "############ LOGS-START ############"
> + cat "${LOGDIR}/${task_log}.log"
> + echo "############ LOGS-END ############"
> + echo
> + fi
> + return 1
> + else
> + echo "OK"
> + return 0
> + fi
> +}
> +
> +function project_create() {
> + target="${1:?}"
> + destdir="${BUILDDIR}/${target}"
> +
> + (
> + # init yocto project
> + source "${YOCTODIR}/poky/oe-init-build-env" "${destdir}"
> +
> + # add needed layers
> + for layer in ${build_layerlist}; do
> + bitbake-layers add-layer "${YOCTODIR}/${layer}" || exit 1
> + done
> + ) || return 1
> +
> + # Detect latest version available in Yocto and use it instead of default
> + # one.
> + XENVERS=$(grep -e "^XEN_REL" \
> + "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
> + 2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
> + | sed -e "s/.*=//" | sort -V | tail -n 1)
> +
> + # customize project configuration
> + cat <<EOF >> "${destdir}/conf/local.conf"
> +# Yocto BSP
> +MACHINE = "${target}"
> +
> +# Use local cache to reuse previous builds results
> +SSTATE_DIR = "${CACHEDIR}/sstate-cache"
> +DL_DIR = "${CACHEDIR}/downloads"
> +
> +# Enable xen and virtualization
> +DISTRO_FEATURES = " virtualization xen ipv4"
> +
> +# Speed up run by not generating ssh host keys
> +IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
> +
> +# Save some disk space
> +INHERIT += "rm_work"
> +
> +# Reduce number of jobs
> +BB_NUMBER_THREADS="${BUILDJOBS}"
> +
> +# Use latest Xen version
> +PREFERRED_VERSION:pn-xen = "${XENVERS}%"
> +PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
> +
> +# Use autorev for now as Xen SHA used by latest yocto recipe for Xen does not
> +# include fixes required to build x86 on arm
> +SRCREV:pn-xen = "\${AUTOREV}"
> +SRCREV:pn-xen-tools = "\${AUTOREV}"
> +
> +# Disable all QA errors as the recipe is not up to date with changes in Xen
> +# when we use local sources
> +ERROR_QA:pn-xen = "arch"
> +ERROR_QA:pn-xen-tools = "arch"
> +
> +EOF
> +
> + if [ "${do_localsrc}" = "y" ]; then
> + XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
> + XENSUB=$(basename "$(realpath -m "${XENDIR}")")
> +
> + cat <<EOF >> "${destdir}/conf/local.conf"
> +# Use local sources for xen and xen-tools
> +FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
> +FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
> +
> +SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
> +SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
> +
> +S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
> +S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
> +
> +SRCPV:pn-xen = "1"
> +SRCPV:pn-xen-tools = "1"
> +
> +EOF
> + fi
> +}
> +
> +function project_build() {
> + target="${1:?}"
> + destdir="${BUILDDIR}/${target}"
> +
> + (
> + source "${YOCTODIR}/poky/oe-init-build-env" "${destdir}"
> +
> + bitbake "${build_image}" || exit 1
> + ) || return 1
> +}
> +
> +function project_clean() {
> + target="${1:?}"
> + destdir="${BUILDDIR}/${target}"
> +
> + rm -rf "${destdir}"
> +}
> +
> +function project_run() {
> + target="${1:?}"
> + destdir="${BUILDDIR}/${target}"
> + (
> + source "${YOCTODIR}/poky/oe-init-build-env" "${destdir}" > /dev/null 2>&1
> +
> + /usr/bin/expect <<EOF
> +set timeout 1000
> +spawn bash -c "runqemu serialstdio nographic slirp"
> +
> +expect_after {
> + -re "(.*)\r" {
> + exp_continue
> + }
> + timeout {send_user "ERROR-Timeout!\n"; exit 1}
> + eof {send_user "ERROR-EOF!\n"; exit 1}
> +}
> +
> +# wait initial login
> +expect -re ".* login: "
> +send "root\r"
> +expect -re "root@.*# "
> +
> +EOF
> + exit $?
> + ) || return 1
> +}
> +
> +function help() {
> + cat <<EOF
> +Usage: ${0} [TARGET1] [TARGET2]
> +
> +This script is build the yocto xen-image-minimal for different qemu targets
> +and is running it after.
> +Without any target specified, all supported targets are done.
> +
> +Options:
> + -h, --help Print this help
> + -v, --verbose Verbose build
> + --list-target List supported targets
> + --clean Clean existing project before starting
> + --no-build Do not build (to run an already built project)
> + --no-run Do not run
> + --num-jobs=NUM Define the number of parallel jobs in Yocto.
> + Default: ${BUILDJOBS}
> + --dump-log On error, dump the logs on the console
> + --image=IMG Yocto image or package to build
> + Default: xen-image-minimal
> + --xen-dir=DIR path to xen hypervisor source tree
> + if not provide, normal yocto version of xen is built
> + Default: ${XENDIR}
> + --out-dir=DIR directory where to create the projectss
> + Default: ${BUILDDIR}
> + --log-dir=DIR directory to store logs
> + Default: ${LOGDIR}
> + --cache-dir=DIR directory where to take and store build cache
> + Default: ${CACHEDIR}
> + --layer-dir=DIR directory containing the checkout of yocto layers
> + Default: ${YOCTODIR}
> +EOF
> +}
> +
> +for OPTION in "$@"
> +do
> + case ${OPTION} in
> + -h|--help)
> + help
> + exit 0
> + ;;
> + -v|--verbose)
> + VERBOSE="y"
> + ;;
> + --list-targets)
> + echo "${TARGET_SUPPORTED}"
> + exit 0
> + ;;
> + --clean)
> + do_clean="y"
> + ;;
> + --no-build)
> + do_build="n"
> + ;;
> + --no-run)
> + do_run="n"
> + ;;
> + --dump-log)
> + do_dump="y"
> + ;;
> + --num-jobs=*)
> + BUILDJOBS="${OPTION#*=}"
> + ;;
> + --image=*)
> + build_image="${OPTION#*=}"
> + ;;
> + --xen-dir=*)
> + XENDIR="${OPTION#*=}"
> + if [ ! -e "${XENDIR}/xen/Makefile" ]; then
> + echo "No Xen source tree in ${XENDIR}"
> + exit 1
> + fi
> + do_localsrc="y"
> + ;;
> + --out-dir=*)
> + BUILDDIR="${OPTION#*=}"
> + ;;
> + --log-dir=*)
> + LOGDIR="${OPTION#*=}"
> + ;;
> + --cache-dir=*)
> + CACHEDIR="${OPTION#*=}"
> + ;;
> + --layer-dir=*)
> + YOCTODIR="${OPTION#*=}"
> + ;;
> + --*)
> + echo "Invalid option ${OPTION}"
> + help
> + exit 1
> + ;;
> + *)
> + if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
> + TARGETLIST="${TARGETLIST} ${OPTION}"
> + else
> + echo "Unsupported target ${OPTION}"
> + exit 1
> + fi
> + ;;
> + esac
> +done
> +
> +# if no target is specified build all targets
> +if [ -z "${TARGETLIST}" ]; then
> + TARGETLIST="${TARGET_SUPPORTED}"
> +fi
> +
> +mkdir -p "${CACHEDIR}"
> +mkdir -p "${LOGDIR}"
> +mkdir -p "${BUILDDIR}"
> +
> +# Make sure we have an absolute path
> +YOCTODIR=$(realpath -m "${YOCTODIR}")
> +CACHEDIR=$(realpath -m "${CACHEDIR}")
> +BUILDDIR=$(realpath -m "${BUILDDIR}")
> +LOGDIR=$(realpath -m "${LOGDIR}")
> +if [ "${do_localsrc}" = "y" ]; then
> + XENDIR=$(realpath -m "${XENDIR}")
> +fi
> +
> +# Check that we have all the layers we need
> +for f in ${build_layerlist}; do
> + if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
> + echo "Layer ${f} missing in ${YOCTODIR}"
> + exit 1
> + fi
> +done
> +
> +for f in ${TARGETLIST}; do
> + if [ "${do_clean}" = "y" ]; then
> + run_task project_clean "${f}"
> + fi
> + if [ ! -f "${BUILDDIR}/${f}/conf/local.conf" ]; then
> + run_task project_create "${f}"
> + fi
> + if [ -f "${BUILDDIR}/${f}/conf/local.conf" ]; then
> + if [ "${do_build}" = "y" ]; then
> + run_task project_build "${f}"
> + fi
> + if [ "${do_run}" = "y" ]; then
> + run_task project_run "${f}"
> + fi
> +
> + fi
> +done
> +
> +print_progress "Build Complete (${build_result} errors)"
> +echo
> +exit ${build_result}
> +
> diff --git a/automation/build/yocto/yocto.dockerfile.in b/automation/build/yocto/yocto.dockerfile.in
> new file mode 100644
> index 000000000000..b0892d420377
> --- /dev/null
> +++ b/automation/build/yocto/yocto.dockerfile.in
> @@ -0,0 +1,114 @@
> +# Docker file to create an environment to build yocto with virtualization
> +#
> +# Arguments that can be passed during image creation using --build-arg:
> +# "host_uid=$(id -u)": to use current user uid for build user in the image
> +# "host_gid=$(id -g)": to use current user gid for build user in the image
> +# "ubuntu_version=VERS": to select the ubuntu version number
> +
> +# Use standard ubuntu minimal.
> +ARG ubuntu_version=22.04
> +From ##DOCKERPLAT##ubuntu:$ubuntu_version AS base
> +LABEL maintainer.name="The Xen Project " \
> + maintainer.email="xen-devel@lists.xenproject.org"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +
> +# Install minimal ubuntu requirements for yocto and other tools we need.
> +# See https://docs.yoctoproject.org/4.0.1/brief-yoctoprojectqs/index.html#build-host-packages
> +RUN apt-get update && \
> + apt-get --quiet --yes install \
> + gawk \
> + wget \
> + git \
> + diffstat \
> + unzip \
> + texinfo \
> + gcc \
> + build-essential \
> + chrpath \
> + socat \
> + cpio \
> + python3 \
> + python3-pip \
> + python3-pexpect \
> + xz-utils \
> + debianutils \
> + iputils-ping \
> + python3-git \
> + python3-jinja2 \
> + libegl1-mesa \
> + libsdl1.2-dev \
> + python3-subunit \
> + mesa-common-dev \
> + zstd \
> + liblz4-tool \
> + file \
> + vim \
> + bison \
> + expect \
> + locales \
> + liblz4-tool \
> + zstd \
> + openssl \
> + libssl3 \
> + ca-certificates \
> + && \
> + apt-get autoremove -y && \
> + apt-get clean && \
> + rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
> +
> +# Use bash as shell.
> +RUN rm /bin/sh && ln -s bash /bin/sh
> +
> +# Fix local for yocto.
> +RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
> + LANG=en_US.UTF-8
> +ENV LANG en_US.UTF-8
> +ENV LC_ALL en_US.UTF-8
> +
> +# Create a user for the build (we don't want to build as root).
> +ENV USER_NAME docker-build
> +ARG host_uid=1000
> +ARG host_gid=1000
> +RUN groupadd -g $host_gid $USER_NAME && \
> + useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
> +
> +# Switch to our user instead of root and start in its home.
> +USER $USER_NAME
> +WORKDIR /home/$USER_NAME
> +
> +# Create needed directories
> +RUN mkdir -p /home/$USER_NAME/yocto-layers \
> + /home/$USER_NAME/yocto-cache \
> + /home/$USER_NAME/logs \
> + /home/$USER_NAME/bin \
> + /home/$USER_NAME/xen && \
> + chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
> +
> +# clone yocto repositories we need.
> +RUN for rep in \
> + https://github.com/openembedded/meta-openembedded \
> + https://git.yoctoproject.org/poky \
> + https://git.yoctoproject.org/meta-virtualization \
> + ; do \
> + git -C /home/$USER_NAME/yocto-layers \
> + clone -b ##YOCTOVERSION## --single-branch $rep; \
> + done
> +
> +# The builder stage is building an initial cache state that we include in the
> +# final image.
> +From base AS builder
> +
> +# This step can take one to several hours depending on your download bandwith
> +# and the speed of your computer.
> +COPY ./build-yocto.sh /
> +RUN /build-yocto.sh --dump-log ##YOCTOTARGET##
> +
> +From base
> +
> +# Only copy the cache status.
> +COPY --from=builder /home/$USER_NAME/yocto-cache /home/$USER_NAME/yocto-cache/.
> +
> +LABEL maintainer.name="The Xen Project " \
> + maintainer.email="xen-devel@lists.xenproject.org"
> +
> diff --git a/automation/build/yocto/yocto.inc b/automation/build/yocto/yocto.inc
> new file mode 100644
> index 000000000000..e4328ec7f378
> --- /dev/null
> +++ b/automation/build/yocto/yocto.inc
> @@ -0,0 +1,44 @@
> +# This makefile generates the docker files for Yocto builds.
> +# The containers for the current architecture are the one built using make all.
> +# To build containers for a different architecture, you need to call make for
> +# the image you want explicitely.
> +# The containers are named this way:
> +# YOCTOVERSION-TARGET for x86_64 hosts
> +# YOCTOVERSION-TARGET-arm64v8 for arm64 hosts
> +# For example you can build an arm64 container with the following command:
> +# make yocto/kirkstone-qemuarm64-arm64v8
> +
> +# Yocto versions we are currently using.
> +YOCTO_VERSION = kirkstone
> +
> +# Yocto BSPs we want to build for.
> +YOCTO_TARGETS = qemuarm64 qemuarm qemux86-64
> +
> +# Supported container architectures.
> +YOCTO_ARCHS = amd64 arm64v8
> +
> +# Architecture we want to use in gitlab CI (depends on runners arch).
> +CI_ARCH = arm64v8
> +
> +define GEN_DOCKER
> +# Make all is generating architecture we use in the CI.
> +ifeq ($(CI_ARCH),$(3))
> +CONTAINERS += yocto/$(1)-$(2)$(4)
> +else
> +CONTAINERS_EXTRA += yocto/$(1)-$(2)$(4)
> +endif
> +
> +.INTERMEDIATE: yocto/$(1)-$(2)$(4).dockerfile
> +
> +yocto/$(1)-$(2)$(4).dockerfile: yocto/yocto.dockerfile.in
> + @cat $$< | \
> + sed -e "s,##YOCTOVERSION##,$(1),g" | \
> + sed -e "s,##YOCTOTARGET##,$(2),g" | \
> + sed -e "s,##DOCKERPLAT##,$(3)/,g" > $$@
> +
> +endef
> +
> +$(eval $(foreach version,$(YOCTO_VERSION),\
> + $(foreach target,$(YOCTO_TARGETS),\
> + $(foreach arch,$(YOCTO_ARCHS),\
> + $(call GEN_DOCKER,$(version),$(target),$(arch),$(if $(filter amd64,$(arch)),,-$(arch)))))))
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v6 2/3] automation: Add a clean rule for containers
2022-12-01 14:25 ` [PATCH v6 2/3] automation: Add a clean rule for containers Bertrand Marquis
@ 2022-12-01 21:10 ` Stefano Stabellini
2022-12-02 11:01 ` Michal Orzel
1 sibling, 0 replies; 10+ messages in thread
From: Stefano Stabellini @ 2022-12-01 21:10 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel, michal.orzel, Doug Goldstein, Stefano Stabellini
On Thu, 1 Dec 2022, Bertrand Marquis wrote:
> Add make clean support to remove the containers from the local docker
> registry.
> make clean-<image_name> must be called to remove an image:
> make clean-yocto/kirkstone-qemuarm: remove yocto kirkstone for qemuarm
> image
>
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> Changes in v6:
> - Rework clean rule to prevent calling docker during make parsing
> Changes in v5:
> - remove cleaning of all images using make clean
> Changes in v4:
> - also generate clean rule for CONTAINERS_EXTRA
> Changes in v3:
> - none
> Changes in v2:
> - none
> Changes in v1:
> - patch added
> ---
> automation/build/Makefile | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/automation/build/Makefile b/automation/build/Makefile
> index c4aaa747ffdf..f1d56ce532a1 100644
> --- a/automation/build/Makefile
> +++ b/automation/build/Makefile
> @@ -28,3 +28,13 @@ all: $(CONTAINERS)
> clean:
> rm -f yocto/*.dockerfile
>
> +define CLEAN_RULE
> +.PHONY: clean-$(1)
> +clean-$(1):
> + if [ -n "$$$$(docker image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
> + docker image rm $(REGISTRY)/$(subst /,:,$(1)); \
> + fi
> +
> +endef
> +
> +$(eval $(foreach img,$(CONTAINERS) $(CONTAINERS_EXTRA),$(call CLEAN_RULE,$(img))))
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v6 3/3] automation: Add CI test jobs for Yocto
2022-12-01 14:25 ` [PATCH v6 3/3] automation: Add CI test jobs for Yocto Bertrand Marquis
@ 2022-12-01 21:11 ` Stefano Stabellini
0 siblings, 0 replies; 10+ messages in thread
From: Stefano Stabellini @ 2022-12-01 21:11 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel, michal.orzel, Doug Goldstein, Stefano Stabellini
On Thu, 1 Dec 2022, Bertrand Marquis wrote:
> From: Michal Orzel <michal.orzel@amd.com>
>
> Populate test jobs for Yocto based tests using the provided containers.
> Due to the size restrictions, it is currently not possible to split the
> build and run tasks, therefore everything is done in a single step.
>
> Test jobs for the supported Yocto targets are generic to avoid the
> necessity to add new ones after each Yocto release. The only thing
> required to be changed after updating the containers is the variable
> YOCTO_VERSION stored in a .yocto-test template.
>
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> Changes in v6:
> - add comment explaining why the unused x86 yocto entry is there
> Changes in v5:
> - none
> Changes in v4:
> - add .yocto-test for arm64 and x86
> - make yocto jobs use arm64 version
> Changes in v3:
> - patch added
> ---
> automation/gitlab-ci/test.yaml | 45 ++++++++++++++++++++++++++++++++++
> 1 file changed, 45 insertions(+)
>
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index c7e0078e04f1..2d57f4feb74a 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -54,6 +54,35 @@
> tags:
> - x86_64
>
> +.yocto-test:
> + extends: .test-jobs-common
> + script:
> + - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs --xen-dir=`pwd` ${YOCTO_BOARD}
> + variables:
> + YOCTO_VERSION: kirkstone
> + CONTAINER: yocto:${YOCTO_VERSION}-${YOCTO_BOARD}-${YOCTO_HOST}
> + artifacts:
> + paths:
> + - 'logs/*'
> + when: always
> + needs: []
> +
> +.yocto-test-arm64:
> + extends: .yocto-test
> + variables:
> + YOCTO_HOST: arm64v8
> + tags:
> + - arm64
> +
> +# This is not used by any test job as we only run Yocto on arm based machines.
> +# Keep it here so that someone having x86 hardware can easily add jobs.
> +.yocto-test-x86-64:
> + extends: .yocto-test
> + variables:
> + YOCTO_HOST: amd64
> + tags:
> + - x86_64
> +
> # Test jobs
> build-each-commit-gcc:
> extends: .test-jobs-common
> @@ -188,3 +217,19 @@ qemu-smoke-x86-64-clang-pvh:
> - ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE}
> needs:
> - debian-unstable-clang-debug
> +
> +# Yocto test jobs
> +yocto-qemuarm64:
> + extends: .yocto-test-arm64
> + variables:
> + YOCTO_BOARD: qemuarm64
> +
> +yocto-qemuarm:
> + extends: .yocto-test-arm64
> + variables:
> + YOCTO_BOARD: qemuarm
> +
> +yocto-qemux86-64:
> + extends: .yocto-test-arm64
> + variables:
> + YOCTO_BOARD: qemux86-64
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v6 0/3] Yocto Gitlab CI
2022-12-01 14:25 [PATCH v6 0/3] Yocto Gitlab CI Bertrand Marquis
` (2 preceding siblings ...)
2022-12-01 14:25 ` [PATCH v6 3/3] automation: Add CI test jobs for Yocto Bertrand Marquis
@ 2022-12-01 21:11 ` Stefano Stabellini
3 siblings, 0 replies; 10+ messages in thread
From: Stefano Stabellini @ 2022-12-01 21:11 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel, michal.orzel, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Wei Liu,
Doug Goldstein
Tested-by: Stefano Stabellini <sstabellini@kernel.org>
On Thu, 1 Dec 2022, Bertrand Marquis wrote:
> This patch series is a first attempt to check if we could use Yocto in
> gitlab ci to build and run xen on qemu for arm, arm64 and x86.
>
> The first patch is introducing a container template from which container
> files can be generated for all combinations we want to support (qemu
> arm, arm64 and x86 targets and hosts x86_64 and arm64).
> It is also introducing a generic build script (build-yocto.sh) that is
> used to create, build and run a Yocto project.
>
> The second patch is adding a way to easily clean locally created
> containers.
>
> The third patch is introducing some gitlab CI templates and jobs so that
> we can test a Xen source tree in gitlab-CI using Yocto.
>
> This has been tested on a x86 host machine and on an arm host machine
> (with mk_dsdt.c fix).
>
> Changes in v6:
> - fix qemux86 errors on arm hosts
> - added some comments
> - fix Makefile to not call docker a lot during each parse
> - several fixes and improvments described in each patch
>
> Changes in v5:
> - automatic cleanup of generated docker files using INTERMEDIATE
> - typo fixes
> - rebase on Stefano's next branch
> - remove clean rule cleaning all containers
>
> Changes in v4:
> - rework the container generation to support multiple hosts
> - rework the container generation to use a single template for all
> docker files (make process is generating the docker files).
>
> Changes in v3:
> - limit number of jobs in yocto
> - do not copy build script inside container
> - add patch from Michal to create gitlab jobs
>
> Changes in v2:
> - remove gitignore patch which was merged
> - add a --dump-log support in build-yocto.sh script and use it during
> container creation to see the error logs.
>
>
> *** BLURB HERE ***
>
> Bertrand Marquis (2):
> automation: Create Yocto docker images
> automation: Add a clean rule for containers
>
> Michal Orzel (1):
> automation: Add CI test jobs for Yocto
>
> .gitignore | 1 +
> automation/build/Makefile | 26 +-
> automation/build/yocto/build-yocto.sh | 356 +++++++++++++++++++++
> automation/build/yocto/yocto.dockerfile.in | 114 +++++++
> automation/build/yocto/yocto.inc | 44 +++
> automation/gitlab-ci/test.yaml | 45 +++
> 6 files changed, 583 insertions(+), 3 deletions(-)
> create mode 100755 automation/build/yocto/build-yocto.sh
> create mode 100644 automation/build/yocto/yocto.dockerfile.in
> create mode 100644 automation/build/yocto/yocto.inc
>
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v6 2/3] automation: Add a clean rule for containers
2022-12-01 14:25 ` [PATCH v6 2/3] automation: Add a clean rule for containers Bertrand Marquis
2022-12-01 21:10 ` Stefano Stabellini
@ 2022-12-02 11:01 ` Michal Orzel
1 sibling, 0 replies; 10+ messages in thread
From: Michal Orzel @ 2022-12-02 11:01 UTC (permalink / raw)
To: Bertrand Marquis, xen-devel; +Cc: Doug Goldstein, Stefano Stabellini
Hi Bertrand,
On 01/12/2022 15:25, Bertrand Marquis wrote:
>
>
> Add make clean support to remove the containers from the local docker
> registry.
> make clean-<image_name> must be called to remove an image:
> make clean-yocto/kirkstone-qemuarm: remove yocto kirkstone for qemuarm
> image
>
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
~Michal
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v6 1/3] automation: Create Yocto docker images
2022-12-01 14:25 ` [PATCH v6 1/3] automation: Create Yocto docker images Bertrand Marquis
2022-12-01 21:09 ` Stefano Stabellini
@ 2022-12-02 11:03 ` Michal Orzel
1 sibling, 0 replies; 10+ messages in thread
From: Michal Orzel @ 2022-12-02 11:03 UTC (permalink / raw)
To: Bertrand Marquis, xen-devel
Cc: Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Wei Liu, Doug Goldstein
Hi Bertrand,
On 01/12/2022 15:25, Bertrand Marquis wrote:
>
>
> Add containers suitable to run yocto kirkstone build based on ubuntu
> 22.04. It contains all packages required by Yocto and a checkout of the
> layers required to build Xen with Yocto.
>
> Add a generic docker image template to be used to automatically generate
> docker files for different configurations:
> - specific yocto version
> - different targets (qemu arm, arm64 and x86)
> - different host platforms (x86 or arm64)
>
> During a call to 'make all', only the images for the host platform we
> run it on in the CI is generated.
> If needed, images for an other host platform can be generated manually
> by calling the right make target (see make help).
>
> Add a build script to build and run xen on qemu using Yocto.
> The script supports arm32, arm64 and x86_64 and checks that dom0 is
> properly booting. At this stage this does not run any guest on top of
> dom0. The script is to be executed in one of the docker images to build
> and run a system using a Xen source tree.
>
> Add automation/build/yocto/*.dockerfile to gitignore as those files are
> generated.
>
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
Reviewed-by: Michal Orzel <michal.orzel@amd.com>
I know that Stefano tested the build/run on arm64. I did the same locally
on x86 successfully.
~Michal
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2022-12-02 11:04 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-01 14:25 [PATCH v6 0/3] Yocto Gitlab CI Bertrand Marquis
2022-12-01 14:25 ` [PATCH v6 1/3] automation: Create Yocto docker images Bertrand Marquis
2022-12-01 21:09 ` Stefano Stabellini
2022-12-02 11:03 ` Michal Orzel
2022-12-01 14:25 ` [PATCH v6 2/3] automation: Add a clean rule for containers Bertrand Marquis
2022-12-01 21:10 ` Stefano Stabellini
2022-12-02 11:01 ` Michal Orzel
2022-12-01 14:25 ` [PATCH v6 3/3] automation: Add CI test jobs for Yocto Bertrand Marquis
2022-12-01 21:11 ` Stefano Stabellini
2022-12-01 21:11 ` [PATCH v6 0/3] Yocto Gitlab CI Stefano Stabellini
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.