[PATCH v2 0/3] automation: introduce an x86_64 Dom0/DomU test

[PATCH v2 0/3] automation: introduce an x86_64 Dom0/DomU test

[Stefano Stabellini]

Hi all,

This small patch series introduces a new QEMU-based test to Gitlab-CI.
It uses QEMU to emulate an x86_64 machine and run Xen, Dom0 and start a
DomU. It is very similar to the existing qemu-alpine-arm64-gcc test but
based on x86_64 rather than ARM64. I think it is important because all
the other x86_64 gitlab-ci tests we currently have are more narrow and
based on XTF. This would be the first end-to-end x86_64 test in
gitlab-ci.

To make it happen, we need an Alpine Linux rootfs and a Linux kernel.
The first two patches introduce them to gitlab-ci. Note that actually
nothing will get build during gitlab-ci runs, it has already been done
beforehand and uploaded as containers. They only import *existing*
containers and binaries into a gitlab-ci run, thus, they cannot fail.
The risk to the release of the first two patches is as close to zero as
possible.

The last patch is the one introducing a new test. This one can fail.
However, it is a new test at the end of the pipeline: it doesn't impact
the existing tests. In the worst case, the test will fail and the whole
pipeline will be marked as "failed" but looking at the jobs all the
other will continue to be marked as successful. In short, if it fails,
we can simply ignore it. Also, at the moment it is actually succeeding.

Cheers,

Stefano


Stefano Stabellini (3):
      automation: add x86_64 alpine 3.12 test-artifact
      automation: Linux 5.10.74 test-artifact
      automation: add a QEMU based x86_64 Dom0/DomU test

 automation/gitlab-ci/build.yaml                    | 24 ++++++
 automation/gitlab-ci/test.yaml                     | 24 ++++++
 automation/scripts/qemu-alpine-x86_64.sh           | 95 ++++++++++++++++++++++
 automation/tests-artifacts/alpine/3.12.dockerfile  | 66 +++++++++++++++
 .../tests-artifacts/kernel/5.10.74.dockerfile      | 38 +++++++++
 5 files changed, 247 insertions(+)
 create mode 100755 automation/scripts/qemu-alpine-x86_64.sh
 create mode 100644 automation/tests-artifacts/alpine/3.12.dockerfile
 create mode 100644 automation/tests-artifacts/kernel/5.10.74.dockerfile

[PATCH v2 1/3] automation: add x86_64 alpine 3.12 test-artifact

[Stefano Stabellini]

From: Stefano Stabellini <stefano.stabellini@xilinx.com>

It is the same as the existing ARM64 alpine 3.12 test-artifact. It is
used to export an Alpine rootfs for Dom0 used for testing.

Also add the exporting job to build.yaml so that the binaries can be
used during gitlab-ci runs.

Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>
---
Changes in 2:
- remove gettext
---
 automation/gitlab-ci/build.yaml               | 13 ++++
 .../tests-artifacts/alpine/3.12.dockerfile    | 66 +++++++++++++++++++
 2 files changed, 79 insertions(+)
 create mode 100644 automation/tests-artifacts/alpine/3.12.dockerfile

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index d177da1710..76b73beead 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -596,3 +596,16 @@ qemu-system-aarch64-5.2.0-arm64-export:
   tags:
     - arm64
 
+
+# x86_64 test artifacts
+
+alpine-3.12-rootfs-export:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/tests-artifacts/alpine:3.12
+  script:
+    - mkdir binaries && cp /initrd.tar.gz binaries/initrd.tar.gz
+  artifacts:
+    paths:
+      - binaries/initrd.tar.gz
+  tags:
+    - x86_64
diff --git a/automation/tests-artifacts/alpine/3.12.dockerfile b/automation/tests-artifacts/alpine/3.12.dockerfile
new file mode 100644
index 0000000000..b3909996b4
--- /dev/null
+++ b/automation/tests-artifacts/alpine/3.12.dockerfile
@@ -0,0 +1,66 @@
+FROM alpine:3.12
+LABEL maintainer.name="The Xen Project" \
+      maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV USER root
+
+RUN mkdir /build
+WORKDIR /build
+
+RUN \
+  # apk
+  apk update && \
+  \
+  # xen runtime deps
+  apk add musl && \
+  apk add openrc && \
+  apk add busybox && \
+  apk add sudo && \
+  apk add dbus && \
+  apk add bash && \
+  apk add python2 && \
+  apk add zlib && \
+  apk add ncurses && \
+  apk add texinfo && \
+  apk add yajl && \
+  apk add libaio && \
+  apk add xz-dev && \
+  apk add util-linux && \
+  apk add argp-standalone && \
+  apk add libfdt && \
+  apk add glib && \
+  apk add pixman && \
+  apk add curl && \
+  apk add udev && \
+  \
+  # Xen
+  cd / && \
+  # Minimal ramdisk environment in case of cpio output
+  rc-update add udev && \
+  rc-update add udev-trigger && \
+  rc-update add udev-settle && \
+  rc-update add networking sysinit && \
+  rc-update add loopback sysinit && \
+  rc-update add bootmisc boot && \
+  rc-update add devfs sysinit && \
+  rc-update add dmesg sysinit && \
+  rc-update add hostname boot && \
+  rc-update add hwclock boot && \
+  rc-update add hwdrivers sysinit && \
+  rc-update add killprocs shutdown && \
+  rc-update add modloop sysinit && \
+  rc-update add modules boot && \
+  rc-update add mount-ro shutdown && \
+  rc-update add savecache shutdown && \
+  rc-update add sysctl boot && \
+  rc-update add local default && \
+  cp -a /sbin/init /init && \
+  echo "ttyS0" >> /etc/securetty && \
+  echo "hvc0" >> /etc/securetty && \
+  echo "ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100" >> /etc/inittab && \
+  echo "hvc0::respawn:/sbin/getty -L hvc0 115200 vt100" >> /etc/inittab && \
+  passwd -d "root" root && \
+  \
+  # Create rootfs
+  cd / && \
+  tar cvzf /initrd.tar.gz bin dev etc home init lib mnt opt root sbin usr var
-- 
2.17.1

[PATCH v2 2/3] automation: Linux 5.10.74 test-artifact

[Stefano Stabellini]

From: Stefano Stabellini <stefano.stabellini@xilinx.com>

Build a 5.10 kernel to be used as Dom0 and DomU kernel for testing. This
is almost the same as the existing ARM64 recipe for Linux 5.9, the
only differences are:
- upgrade to latest 5.10.x stable
- force Xen modules to built-in (on ARM it was already done by defconfig)

Also add the exporting job to build.yaml so that the binary can be used
during gitlab-ci runs.

Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>
---

Note that make tinyconfig cannot be used because it doesn't boot on QEMU

Changes in 2:
- use two RUNs
- remove apt-get cleanup earlier
---
 automation/gitlab-ci/build.yaml               | 11 ++++++
 .../tests-artifacts/kernel/5.10.74.dockerfile | 38 +++++++++++++++++++
 2 files changed, 49 insertions(+)
 create mode 100644 automation/tests-artifacts/kernel/5.10.74.dockerfile

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 76b73beead..0034c50950 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -609,3 +609,14 @@ alpine-3.12-rootfs-export:
       - binaries/initrd.tar.gz
   tags:
     - x86_64
+
+kernel-5.10.74-export:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/tests-artifacts/kernel:5.10.74
+  script:
+    - mkdir binaries && cp /bzImage binaries/bzImage
+  artifacts:
+    paths:
+      - binaries/bzImage
+  tags:
+    - x86_64
diff --git a/automation/tests-artifacts/kernel/5.10.74.dockerfile b/automation/tests-artifacts/kernel/5.10.74.dockerfile
new file mode 100644
index 0000000000..112e27fe45
--- /dev/null
+++ b/automation/tests-artifacts/kernel/5.10.74.dockerfile
@@ -0,0 +1,38 @@
+FROM debian:unstable
+LABEL maintainer.name="The Xen Project" \
+      maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV LINUX_VERSION=5.10.74
+ENV USER root
+
+RUN mkdir /build
+WORKDIR /build
+
+# build depends
+RUN apt-get update && \
+    apt-get --quiet --yes install \
+        build-essential \
+        libssl-dev \
+        bc \
+        curl \
+        flex \
+        bison \
+        libelf-dev \
+        && \
+    apt-get autoremove -y && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
+
+# Build the kernel
+RUN curl -fsSLO https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-"$LINUX_VERSION".tar.xz && \
+    tar xvJf linux-"$LINUX_VERSION".tar.xz && \
+    cd linux-"$LINUX_VERSION" && \
+    make defconfig && \
+    make xen.config && \
+    cp .config .config.orig && \
+    cat .config.orig | grep XEN | grep =m |sed 's/=m/=y/g' >> .config && \
+    make -j$(nproc) bzImage && \
+    cp arch/x86/boot/bzImage / && \
+    cd /build && \
+    rm -rf linux-"$LINUX_VERSION"*
-- 
2.17.1

[PATCH v2 3/3] automation: add a QEMU based x86_64 Dom0/DomU test

[Stefano Stabellini]

From: Stefano Stabellini <stefano.stabellini@xilinx.com>

Introduce a test based on QEMU to run Xen, Dom0 and start a DomU.
This is similar to the existing qemu-alpine-arm64.sh script and test.
The only differences are:
- use Debian's qemu-system-x86_64 (on ARM we build our own)
- use ipxe instead of u-boot and ImageBuilder

Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
---
apt-get is not entirely removed yet as we need cpio and busybox-static.
I am in favor of keeping them as apt-get packages because they are not
needed in the build container. At the same time I see that being small
they wouldn't increase the build container by much.

Changes in v2:
- remove curl
- add comments
- add set -x
- use debian:stretch and remove qemu
---
 automation/gitlab-ci/test.yaml           | 24 ++++++
 automation/scripts/qemu-alpine-x86_64.sh | 95 ++++++++++++++++++++++++
 2 files changed, 119 insertions(+)
 create mode 100755 automation/scripts/qemu-alpine-x86_64.sh

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 91a10febbf..43d248a604 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -47,6 +47,30 @@ qemu-alpine-arm64-gcc:
     - /^coverity-tested\/.*/
     - /^stable-.*/
 
+qemu-alpine-x86_64-gcc:
+  stage: test
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  variables:
+    CONTAINER: debian:stretch
+  script:
+    - ./automation/scripts/qemu-alpine-x86_64.sh 2>&1 | tee qemu-smoke-arm64.log
+  dependencies:
+    - alpine-3.12-gcc
+    - alpine-3.12-rootfs-export
+    - kernel-5.10.74-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - x86_64
+  except:
+    - master
+    - smoke
+    - /^coverity-tested\/.*/
+    - /^stable-.*/
+
 qemu-smoke-arm64-gcc:
   stage: test
   image: registry.gitlab.com/xen-project/xen/${CONTAINER}
diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
new file mode 100755
index 0000000000..2e9625109c
--- /dev/null
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -0,0 +1,95 @@
+#!/bin/bash
+
+set -ex
+
+apt-get -qy update
+apt-get -qy install --no-install-recommends cpio \
+                                            busybox-static
+
+# DomU Busybox
+cd binaries
+mkdir -p initrd
+mkdir -p initrd/bin
+mkdir -p initrd/sbin
+mkdir -p initrd/etc
+mkdir -p initrd/dev
+mkdir -p initrd/proc
+mkdir -p initrd/sys
+mkdir -p initrd/lib
+mkdir -p initrd/var
+mkdir -p initrd/mnt
+cp /bin/busybox initrd/bin/busybox
+initrd/bin/busybox --install initrd/bin
+echo "#!/bin/sh
+
+mount -t proc proc /proc
+mount -t sysfs sysfs /sys
+mount -t devtmpfs devtmpfs /dev
+/bin/sh" > initrd/init
+chmod +x initrd/init
+# DomU rootfs
+cd initrd
+find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
+cd ..
+
+# initrd.tar.gz is Dom0 rootfs
+mkdir -p rootfs
+cd rootfs
+tar xvzf ../initrd.tar.gz
+mkdir proc
+mkdir run
+mkdir srv
+mkdir sys
+rm var/run
+cp -ar ../dist/install/* .
+mv ../initrd.cpio.gz ./root
+cp ../bzImage ./root
+echo "name=\"test\"
+memory=512
+vcpus=1
+kernel=\"/root/bzImage\"
+ramdisk=\"/root/initrd.cpio.gz\"
+extra=\"console=hvc0 root=/dev/ram0 rdinit=/bin/sh\"
+" > root/test.cfg
+echo "#!/bin/bash
+
+set -x
+
+export LD_LIBRARY_PATH=/usr/local/lib
+bash /etc/init.d/xencommons start
+
+xl list
+
+xl create -c /root/test.cfg
+
+" > etc/local.d/xen.start
+chmod +x etc/local.d/xen.start
+echo "rc_verbose=yes" >> etc/rc.conf
+# rebuild Dom0 rootfs
+find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
+cd ../..
+
+cat >> binaries/pxelinux.0 << EOF
+#!ipxe
+
+kernel xen console=com1
+module bzImage console=hvc0
+module xen-rootfs.cpio.gz
+boot
+EOF
+
+# Run the test
+rm -f smoke.serial
+set +e
+timeout -k 1 720 \
+qemu-system-x86_64 \
+    -cpu qemu64,+svm \
+    -m 2G -smp 2 \
+    -monitor none -serial stdio \
+    -nographic \
+    -device virtio-net-pci,netdev=n0 \
+    -netdev user,id=n0,tftp=binaries,bootfile=/pxelinux.0 |& tee smoke.serial
+
+set -e
+(grep -q "Domain-0" smoke.serial && grep -q "BusyBox" smoke.serial) || exit 1
+exit 0
-- 
2.17.1

Re: [PATCH v2 3/3] automation: add a QEMU based x86_64 Dom0/DomU test

[Anthony PERARD]

On Mon, Oct 25, 2021 at 06:42:00PM -0700, Stefano Stabellini wrote:
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> 
> Introduce a test based on QEMU to run Xen, Dom0 and start a DomU.
> This is similar to the existing qemu-alpine-arm64.sh script and test.
> The only differences are:
> - use Debian's qemu-system-x86_64 (on ARM we build our own)
> - use ipxe instead of u-boot and ImageBuilder
> 
> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> ---
>  automation/gitlab-ci/test.yaml           | 24 ++++++
>  automation/scripts/qemu-alpine-x86_64.sh | 95 ++++++++++++++++++++++++
>  2 files changed, 119 insertions(+)
>  create mode 100755 automation/scripts/qemu-alpine-x86_64.sh
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 91a10febbf..43d248a604 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -47,6 +47,30 @@ qemu-alpine-arm64-gcc:
>      - /^coverity-tested\/.*/
>      - /^stable-.*/
>  
> +qemu-alpine-x86_64-gcc:
> +  stage: test
> +  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> +  variables:
> +    CONTAINER: debian:stretch
> +  script:
> +    - ./automation/scripts/qemu-alpine-x86_64.sh 2>&1 | tee qemu-smoke-arm64.log

Maybe the log file should be called "qemu-smoke-x86_64.log" ? Or just
"qemu-smoke.log"?


I'm still not happy about the `apt install` in the middle of the test,
but I guess that will do for now:

Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>

Thanks,

-- 
Anthony PERARD

Re: [PATCH v2 3/3] automation: add a QEMU based x86_64 Dom0/DomU test

[Stefano Stabellini]

On Wed, 27 Oct 2021, Anthony PERARD wrote:
> On Mon, Oct 25, 2021 at 06:42:00PM -0700, Stefano Stabellini wrote:
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > 
> > Introduce a test based on QEMU to run Xen, Dom0 and start a DomU.
> > This is similar to the existing qemu-alpine-arm64.sh script and test.
> > The only differences are:
> > - use Debian's qemu-system-x86_64 (on ARM we build our own)
> > - use ipxe instead of u-boot and ImageBuilder
> > 
> > Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > ---
> >  automation/gitlab-ci/test.yaml           | 24 ++++++
> >  automation/scripts/qemu-alpine-x86_64.sh | 95 ++++++++++++++++++++++++
> >  2 files changed, 119 insertions(+)
> >  create mode 100755 automation/scripts/qemu-alpine-x86_64.sh
> > 
> > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > index 91a10febbf..43d248a604 100644
> > --- a/automation/gitlab-ci/test.yaml
> > +++ b/automation/gitlab-ci/test.yaml
> > @@ -47,6 +47,30 @@ qemu-alpine-arm64-gcc:
> >      - /^coverity-tested\/.*/
> >      - /^stable-.*/
> >  
> > +qemu-alpine-x86_64-gcc:
> > +  stage: test
> > +  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> > +  variables:
> > +    CONTAINER: debian:stretch
> > +  script:
> > +    - ./automation/scripts/qemu-alpine-x86_64.sh 2>&1 | tee qemu-smoke-arm64.log
> 
> Maybe the log file should be called "qemu-smoke-x86_64.log" ? Or just
> "qemu-smoke.log"?

Good catch! This could be fixed on commit.

 
> I'm still not happy about the `apt install` in the middle of the test,
> but I guess that will do for now:
> 
> Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>

Thanks!

Now the series is fully acked and Ian gave his reviewed-by. If people
are OK with that I could commit it in a couple of days (and by the end
of this week).