[PATCH v3 0/2] automation: qemu32 smoke test

[PATCH v3 0/2] automation: qemu32 smoke test

[Stefano Stabellini]

Hi all,

This small series adds a simple Xen + Dom0 boot arm32 test to gitlab-ci
using QEMU, similar to the existing tests for arm64 and x86.

Cheers,

Stefano


Stefano Stabellini (2):
      gitlab-ci: add qemu-system-arm to the existing tests-artifacts container
      gitlab-ci: add an ARM32 qemu-based smoke test

 automation/gitlab-ci/build.yaml                    | 14 ++++-
 automation/gitlab-ci/test.yaml                     | 27 +++++++-
 automation/scripts/qemu-smoke-arm32.sh             | 72 ++++++++++++++++++++++
 ...arm64v8.dockerfile => 6.0.0-arm64v8.dockerfile} |  5 +-
 4 files changed, 112 insertions(+), 6 deletions(-)
 create mode 100755 automation/scripts/qemu-smoke-arm32.sh
 rename automation/tests-artifacts/qemu-system-aarch64/{5.2.0-arm64v8.dockerfile => 6.0.0-arm64v8.dockerfile} (95%)

[PATCH v3 1/2] gitlab-ci: add qemu-system-arm to the existing tests-artifacts container

[Stefano Stabellini]

From: Stefano Stabellini <stefano.stabellini@xilinx.com>

Add qemu-system-arm to the existing test-artifacts qemu container (which
doesn't get build for every iteration but only updated once in a while.)

With qemu-system-arm available, we'll be able to run ARM32 tests.

This patch also bumps the QEMU version to v6.0.0 for both arm32 and
arm64 (the test-artifacts container is one, shared for both).

Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>
---
 automation/gitlab-ci/build.yaml                    | 14 ++++++++++++--
 automation/gitlab-ci/test.yaml                     |  4 ++--
 ...arm64v8.dockerfile => 6.0.0-arm64v8.dockerfile} |  5 +++--
 3 files changed, 17 insertions(+), 6 deletions(-)
 rename automation/tests-artifacts/qemu-system-aarch64/{5.2.0-arm64v8.dockerfile => 6.0.0-arm64v8.dockerfile} (95%)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index cc36428cf5..72f2a317ac 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -591,9 +591,9 @@ kernel-5.9.9-arm64-export:
   tags:
     - arm64
 
-qemu-system-aarch64-5.2.0-arm64-export:
+qemu-system-aarch64-6.0.0-arm64-export:
   stage: build
-  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:5.2.0-arm64v8
+  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8
   script:
     - mkdir binaries && cp /qemu-system-aarch64 binaries/qemu-system-aarch64
   artifacts:
@@ -602,6 +602,16 @@ qemu-system-aarch64-5.2.0-arm64-export:
   tags:
     - arm64
 
+qemu-system-aarch64-6.0.0-arm32-export:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8
+  script:
+    - mkdir binaries && cp /qemu-system-arm binaries/qemu-system-arm
+  artifacts:
+    paths:
+      - binaries/qemu-system-arm
+  tags:
+    - arm64
 
 # x86_64 test artifacts
 
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 47e8704df3..ec2a2e1607 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -33,7 +33,7 @@ qemu-alpine-arm64-gcc:
     - alpine-3.12-gcc-arm64
     - alpine-3.12-arm64-rootfs-export
     - kernel-5.9.9-arm64-export
-    - qemu-system-aarch64-5.2.0-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
   artifacts:
     paths:
       - smoke.serial
@@ -81,7 +81,7 @@ qemu-smoke-arm64-gcc:
   dependencies:
     - debian-unstable-gcc-arm64
     - kernel-5.9.9-arm64-export
-    - qemu-system-aarch64-5.2.0-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
   artifacts:
     paths:
       - smoke.serial
diff --git a/automation/tests-artifacts/qemu-system-aarch64/5.2.0-arm64v8.dockerfile b/automation/tests-artifacts/qemu-system-aarch64/6.0.0-arm64v8.dockerfile
similarity index 95%
rename from automation/tests-artifacts/qemu-system-aarch64/5.2.0-arm64v8.dockerfile
rename to automation/tests-artifacts/qemu-system-aarch64/6.0.0-arm64v8.dockerfile
index e105a1c636..793432d40b 100644
--- a/automation/tests-artifacts/qemu-system-aarch64/5.2.0-arm64v8.dockerfile
+++ b/automation/tests-artifacts/qemu-system-aarch64/6.0.0-arm64v8.dockerfile
@@ -3,7 +3,7 @@ LABEL maintainer.name="The Xen Project" \
       maintainer.email="xen-devel@lists.xenproject.org"
 
 ENV DEBIAN_FRONTEND=noninteractive
-ENV QEMU_VERSION=5.2.0
+ENV QEMU_VERSION=6.0.0
 ENV USER root
 
 RUN mkdir /build
@@ -25,7 +25,7 @@ RUN apt-get update && \
     tar xvJf qemu-"$QEMU_VERSION".tar.xz && \
     cd qemu-"$QEMU_VERSION" && \
     ./configure                \
-        --target-list=aarch64-softmmu \
+        --target-list=arm-softmmu,aarch64-softmmu \
         --enable-system        \
         --disable-blobs        \
         --disable-bsd-user     \
@@ -68,6 +68,7 @@ RUN apt-get update && \
     && \
     make -j$(nproc) && \
     cp ./build/qemu-system-aarch64 / && \
+    cp ./build/qemu-system-arm / && \
     cd /build && \
     rm -rf qemu-"$QEMU_VERSION"* && \
     apt-get autoremove -y && \
-- 
2.25.1

[PATCH v3 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by
the test-artifacts qemu container. The minimal test simply boots Xen
(built from previous build stages) and Dom0. The test is fetching the
Dom0 kernel and initrd from Debian Jessie: they work just fine and this
way we don't have to maintain a build for them too.

Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>

---
Changes in v3:
- use long options for curl
- use curl --output

Changes in v2:
- improve comments
- don't limit dom0 cpus
- decrease mem to 1024M and dom0_mem=512M
- use Debian Bullseye instead of Jessie
---
 automation/gitlab-ci/test.yaml         | 23 ++++++++
 automation/scripts/qemu-smoke-arm32.sh | 72 ++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100755 automation/scripts/qemu-smoke-arm32.sh

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index ec2a2e1607..42cd725a12 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -95,6 +95,29 @@ qemu-smoke-arm64-gcc:
     - /^coverity-tested\/.*/
     - /^stable-.*/
 
+qemu-smoke-arm32-gcc:
+  stage: test
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
+  dependencies:
+    - debian-unstable-gcc-arm32
+    - qemu-system-aarch64-6.0.0-arm32-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+  except:
+    - master
+    - smoke
+    - /^coverity-tested\/.*/
+    - /^stable-.*/
+
 qemu-smoke-x86-64-gcc:
   stage: test
   image: registry.gitlab.com/xen-project/xen/${CONTAINER}
diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
new file mode 100755
index 0000000000..162922ace5
--- /dev/null
+++ b/automation/scripts/qemu-smoke-arm32.sh
@@ -0,0 +1,72 @@
+#!/bin/bash
+
+set -ex
+
+export DEBIAN_FRONTENT=noninteractive
+apt-get -qy update
+apt-get -qy install --no-install-recommends device-tree-compiler \
+                                            curl
+
+cd binaries
+curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
+curl --fail --silent --show-error --location --output initrd.gz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
+
+kernel=`stat -L --printf="%s" vmlinuz`
+initrd=`stat -L --printf="%s" initrd.gz`
+
+# For Xen, we need a couple of more node. Dump the DT from QEMU and add them
+# XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
+curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
+./qemu-system-arm \
+   -machine virt-6.0 \
+   -machine virtualization=true \
+   -smp 4 \
+   -m 1024 \
+   -serial stdio \
+   -monitor none \
+   -display none \
+   -machine dumpdtb=virt.dtb
+
+dtc -I dtb -O dts virt.dtb > virt.dts
+
+cat >> virt.dts << EOF
+/ {
+	chosen {
+		#address-cells = <0x2>;
+		#size-cells = <0x2>;
+		stdout-path = "/pl011@9000000";
+        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=512M bootscrub=0";
+		xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh init=/bin/sh";
+		dom0 {
+			compatible = "xen,linux-zimage", "xen,multiboot-module";
+			reg = <0x0 0x1000000 0x0 $kernel>;
+		};
+        dom0-ramdisk {
+			compatible = "xen,linux-initrd", "xen,multiboot-module";
+			reg = <0x0 0x3200000 0x0 $initrd>;
+		};
+	};
+};
+EOF
+dtc -I dts -O dtb virt.dts > virt.dtb
+
+rm -f smoke.serial
+set +e
+timeout -k 1 240 \
+./qemu-system-arm \
+   -machine virt-6.0 \
+   -machine virtualization=true \
+   -smp 4 \
+   -m 1024 \
+   -serial stdio \
+   -monitor none \
+   -display none \
+   -dtb virt.dtb \
+   -no-reboot \
+   -kernel ./xen \
+   -device loader,file=./vmlinuz,addr=0x1000000 \
+   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
+
+set -e
+(grep -q "^BusyBox" smoke.serial) || exit 1
+exit 0
-- 
2.25.1

Re: [PATCH v3 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

On Fri, 18 Mar 2022, Stefano Stabellini wrote:
> Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by
> the test-artifacts qemu container. The minimal test simply boots Xen
> (built from previous build stages) and Dom0. The test is fetching the
> Dom0 kernel and initrd from Debian Jessie: they work just fine and this
> way we don't have to maintain a build for them too.


Thanks to the Xen fix recently submitted
(https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to
update this script to use Debian Bullseye. I am thinking of merging the
below directly with this patch.


---

automation: upgrade Debian to Bullseye for testing Xen aarch32

Also change initrd. As the new netboot initrd from Debian Bullseye is
huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB).

Also note that the huge Debian Bullseye initrd would cause QEMU to
crash due to the -device loader parameter.

Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>

diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
index 162922ace5..d554de7939 100755
--- a/automation/scripts/qemu-smoke-arm32.sh
+++ b/automation/scripts/qemu-smoke-arm32.sh
@@ -5,11 +5,20 @@ set -ex
 export DEBIAN_FRONTENT=noninteractive
 apt-get -qy update
 apt-get -qy install --no-install-recommends device-tree-compiler \
-                                            curl
+                                            curl \
+                                            cpio
 
 cd binaries
-curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
-curl --fail --silent --show-error --location --output initrd.gz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
+# Use the kernel from Debian
+curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/bullseye/main/installer-armhf/current/images/netboot/vmlinuz
+# Use a tiny initrd based on busybox from Alpine Linux
+curl --fail --silent --show-error --location --output initrd.tar.gz https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/armhf/alpine-minirootfs-3.15.1-armhf.tar.gz
+
+mkdir rootfs
+cd rootfs
+tar xvzf ../initrd.tar.gz
+find . | cpio -H newc -o | gzip > ../initrd.gz
+cd ..
 
 kernel=`stat -L --printf="%s" vmlinuz`
 initrd=`stat -L --printf="%s" initrd.gz`
@@ -68,5 +77,5 @@ timeout -k 1 240 \
    -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
 
 set -e
-(grep -q "^BusyBox" smoke.serial) || exit 1
+(grep -q "^/ #" smoke.serial) || exit 1
 exit 0

Re: [PATCH v3 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Julien Grall]

Hi,

On 20/03/2022 01:46, Stefano Stabellini wrote:
> On Fri, 18 Mar 2022, Stefano Stabellini wrote:
>> Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by
>> the test-artifacts qemu container. The minimal test simply boots Xen
>> (built from previous build stages) and Dom0. The test is fetching the
>> Dom0 kernel and initrd from Debian Jessie: they work just fine and this
>> way we don't have to maintain a build for them too.
> 
> 
> Thanks to the Xen fix recently submitted
> (https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to
> update this script to use Debian Bullseye. I am thinking of merging the
> below directly with this patch.
> 
> 
> ---
> 
> automation: upgrade Debian to Bullseye for testing Xen aarch32
> 
> Also change initrd. As the new netboot initrd from Debian Bullseye is
> huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB).

This is sounds odd to me. So we are going to use Bullseye but not really 
because we want to use a different initrd.

Why can't you get everything from the same place?

> 
> Also note that the huge Debian Bullseye initrd would cause QEMU to
> crash due to the -device loader parameter.

Can you provide more details? Was this reported to QEMU?

> 
> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> 
> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
> index 162922ace5..d554de7939 100755
> --- a/automation/scripts/qemu-smoke-arm32.sh
> +++ b/automation/scripts/qemu-smoke-arm32.sh
> @@ -5,11 +5,20 @@ set -ex
>   export DEBIAN_FRONTENT=noninteractive
>   apt-get -qy update
>   apt-get -qy install --no-install-recommends device-tree-compiler \
> -                                            curl
> +                                            curl \
> +                                            cpio
>   
>   cd binaries
> -curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
> -curl --fail --silent --show-error --location --output initrd.gz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
> +# Use the kernel from Debian
> +curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/bullseye/main/installer-armhf/current/images/netboot/vmlinuz
> +# Use a tiny initrd based on busybox from Alpine Linux
> +curl --fail --silent --show-error --location --output initrd.tar.gz https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/armhf/alpine-minirootfs-3.15.1-armhf.tar.gz
> +
> +mkdir rootfs
> +cd rootfs
> +tar xvzf ../initrd.tar.gz
> +find . | cpio -H newc -o | gzip > ../initrd.gz
> +cd ..
>   
>   kernel=`stat -L --printf="%s" vmlinuz`
>   initrd=`stat -L --printf="%s" initrd.gz`
> @@ -68,5 +77,5 @@ timeout -k 1 240 \
>      -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
>   
>   set -e
> -(grep -q "^BusyBox" smoke.serial) || exit 1
> +(grep -q "^/ #" smoke.serial) || exit 1
>   exit 0
> 

Cheers,

-- 
Julien Grall

Re: [PATCH v3 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

On Sun, 20 Mar 2022, Julien Grall wrote:
> On 20/03/2022 01:46, Stefano Stabellini wrote:
> > On Fri, 18 Mar 2022, Stefano Stabellini wrote:
> > > Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by
> > > the test-artifacts qemu container. The minimal test simply boots Xen
> > > (built from previous build stages) and Dom0. The test is fetching the
> > > Dom0 kernel and initrd from Debian Jessie: they work just fine and this
> > > way we don't have to maintain a build for them too.
> > 
> > 
> > Thanks to the Xen fix recently submitted
> > (https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to
> > update this script to use Debian Bullseye. I am thinking of merging the
> > below directly with this patch.
> > 
> > 
> > ---
> > 
> > automation: upgrade Debian to Bullseye for testing Xen aarch32
> > 
> > Also change initrd. As the new netboot initrd from Debian Bullseye is
> > huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB).
> 
> This is sounds odd to me. So we are going to use Bullseye but not really
> because we want to use a different initrd.
> 
> Why can't you get everything from the same place?

Because it doesn't work :-(


> > Also note that the huge Debian Bullseye initrd would cause QEMU to
> > crash due to the -device loader parameter.
> 
> Can you provide more details? Was this reported to QEMU?

QEMU core dumps when provided with the Debian Bullseye initrd binary to
load. I guessed it was due to the size and tried with a smaller size.
Everything worked with a smaller initrd. I also think that it is not a
good idea to use a 22MB initrd anyway so decided against the Debian
Bullseye initrd. (For reference 22MB is basically the size of a fully
featured Yocto-build rootfs.) I did not file a bug to qemu-devel yet and
didn't investigate further on the QEMU side as I ran out of time.

Alpine Linux provides a very nice 2.5MB initrd. I tried to use both
kernel and initrd from Alpine Linux but unfortunately the Alpine Linux
kernel doesn't boot. I don't know why but I think it is because it might
be missing the console driver. I am not sure. There are a lot of
combinations that don't work and it is time consuming to investigate
them all. I have been trying to investigate only the most critical
things -- they are too many! 

I should add that the Debian initrd is not the ideal initrd because it
is made to start the Debian installer. Here we just want a tiny busybox
initrd.

In general, I think it would be better if we could use the kernel and
initrd from the same source but I couldn't find one that works. I could
build one myself but it would be one more thing to maintain in
gitlab-ci. Also using u-boot might solve the problem of loading the
binary but again we would have to maintain a u-boot arm32 build in
gitlab-ci.

So in order of preference best to worst in my opinion:

1) kernel and initrd from the same source
2) kernel and initrd from different sources
3) build your own kernel/initrd/u-boot

So I ended up doing 2). I tested it and it is sufficient to get the test
up and running.

 
> > Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > 
> > diff --git a/automation/scripts/qemu-smoke-arm32.sh
> > b/automation/scripts/qemu-smoke-arm32.sh
> > index 162922ace5..d554de7939 100755
> > --- a/automation/scripts/qemu-smoke-arm32.sh
> > +++ b/automation/scripts/qemu-smoke-arm32.sh
> > @@ -5,11 +5,20 @@ set -ex
> >   export DEBIAN_FRONTENT=noninteractive
> >   apt-get -qy update
> >   apt-get -qy install --no-install-recommends device-tree-compiler \
> > -                                            curl
> > +                                            curl \
> > +                                            cpio
> >     cd binaries
> > -curl --fail --silent --show-error --location --output vmlinuz
> > http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
> > -curl --fail --silent --show-error --location --output initrd.gz
> > http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
> > +# Use the kernel from Debian
> > +curl --fail --silent --show-error --location --output vmlinuz
> > http://http.us.debian.org/debian/dists/bullseye/main/installer-armhf/current/images/netboot/vmlinuz
> > +# Use a tiny initrd based on busybox from Alpine Linux
> > +curl --fail --silent --show-error --location --output initrd.tar.gz
> > https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/armhf/alpine-minirootfs-3.15.1-armhf.tar.gz
> > +
> > +mkdir rootfs
> > +cd rootfs
> > +tar xvzf ../initrd.tar.gz
> > +find . | cpio -H newc -o | gzip > ../initrd.gz
> > +cd ..
> >     kernel=`stat -L --printf="%s" vmlinuz`
> >   initrd=`stat -L --printf="%s" initrd.gz`
> > @@ -68,5 +77,5 @@ timeout -k 1 240 \
> >      -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
> >     set -e
> > -(grep -q "^BusyBox" smoke.serial) || exit 1
> > +(grep -q "^/ #" smoke.serial) || exit 1
> >   exit 0

Re: [PATCH v3 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Julien Grall]

Hi Stefano,

On 21/03/2022 19:47, Stefano Stabellini wrote:
> On Sun, 20 Mar 2022, Julien Grall wrote:
>> On 20/03/2022 01:46, Stefano Stabellini wrote:
>>> On Fri, 18 Mar 2022, Stefano Stabellini wrote:
>>>> Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by
>>>> the test-artifacts qemu container. The minimal test simply boots Xen
>>>> (built from previous build stages) and Dom0. The test is fetching the
>>>> Dom0 kernel and initrd from Debian Jessie: they work just fine and this
>>>> way we don't have to maintain a build for them too.
>>>
>>>
>>> Thanks to the Xen fix recently submitted
>>> (https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to
>>> update this script to use Debian Bullseye. I am thinking of merging the
>>> below directly with this patch.
>>>
>>>
>>> ---
>>>
>>> automation: upgrade Debian to Bullseye for testing Xen aarch32
>>>
>>> Also change initrd. As the new netboot initrd from Debian Bullseye is
>>> huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB).
>>
>> This is sounds odd to me. So we are going to use Bullseye but not really
>> because we want to use a different initrd.
>>
>> Why can't you get everything from the same place?
> 
> Because it doesn't work :-(
> 
> 
>>> Also note that the huge Debian Bullseye initrd would cause QEMU to
>>> crash due to the -device loader parameter.
>>
>> Can you provide more details? Was this reported to QEMU?
> 
> QEMU core dumps when provided with the Debian Bullseye initrd binary to
> load. I guessed it was due to the size and tried with a smaller size.
> Everything worked with a smaller initrd. I also think that it is not a
> good idea to use a 22MB initrd anyway so decided against the Debian
> Bullseye initrd. 

Why is it a bad idea? In general, bigger file allows us to test corner 
cases.

> (For reference 22MB is basically the size of a fully
> featured Yocto-build rootfs.) I did not file a bug to qemu-devel yet and
> didn't investigate further on the QEMU side as I ran out of time.
> 
> Alpine Linux provides a very nice 2.5MB initrd. I tried to use both
> kernel and initrd from Alpine Linux but unfortunately the Alpine Linux
> kernel doesn't boot. I don't know why but I think it is because it might
> be missing the console driver. I am not sure. There are a lot of
> combinations that don't work and it is time consuming to investigate
> them all. I have been trying to investigate only the most critical
> things -- they are too many!
> 
> I should add that the Debian initrd is not the ideal initrd because it
> is made to start the Debian installer. Here we just want a tiny busybox
> initrd.
> 
> In general, I think it would be better if we could use the kernel and
> initrd from the same source but I couldn't find one that works. I could
> build one myself but it would be one more thing to maintain in
> gitlab-ci. Also using u-boot might solve the problem of loading the
> binary but again we would have to maintain a u-boot arm32 build in
> gitlab-ci.
> 
> So in order of preference best to worst in my opinion:
> 
> 1) kernel and initrd from the same source
> 2) kernel and initrd from different sources
> 3) build your own kernel/initrd/u-boot
> 
> So I ended up doing 2). I tested it and it is sufficient to get the test
> up and running.

Thanks for the explanation. So I think we should not call that an 
"Upgrade to Bullseye" because you are not using Debian. Instead, you 
borrowed a working kernel that happens to have everything you need built-in.

Also, can you update the commit message and provide a summary of this 
discussion?

Cheers,

-- 
Julien Grall

Re: [PATCH v3 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

On Mon, 21 Mar 2022, Julien Grall wrote:
> On 21/03/2022 19:47, Stefano Stabellini wrote:
> > On Sun, 20 Mar 2022, Julien Grall wrote:
> > > On 20/03/2022 01:46, Stefano Stabellini wrote:
> > > > On Fri, 18 Mar 2022, Stefano Stabellini wrote:
> > > > > Add a minimal ARM32 smoke test based on qemu-system-arm, as provided
> > > > > by
> > > > > the test-artifacts qemu container. The minimal test simply boots Xen
> > > > > (built from previous build stages) and Dom0. The test is fetching the
> > > > > Dom0 kernel and initrd from Debian Jessie: they work just fine and
> > > > > this
> > > > > way we don't have to maintain a build for them too.
> > > > 
> > > > 
> > > > Thanks to the Xen fix recently submitted
> > > > (https://marc.info/?l=xen-devel&m=164774063802402) I'll be able to
> > > > update this script to use Debian Bullseye. I am thinking of merging the
> > > > below directly with this patch.
> > > > 
> > > > 
> > > > ---
> > > > 
> > > > automation: upgrade Debian to Bullseye for testing Xen aarch32
> > > > 
> > > > Also change initrd. As the new netboot initrd from Debian Bullseye is
> > > > huge (22MB), use a tiny initrd from Alpine Linux instead (only 2.5MB).
> > > 
> > > This is sounds odd to me. So we are going to use Bullseye but not really
> > > because we want to use a different initrd.
> > > 
> > > Why can't you get everything from the same place?
> > 
> > Because it doesn't work :-(
> > 
> > 
> > > > Also note that the huge Debian Bullseye initrd would cause QEMU to
> > > > crash due to the -device loader parameter.
> > > 
> > > Can you provide more details? Was this reported to QEMU?
> > 
> > QEMU core dumps when provided with the Debian Bullseye initrd binary to
> > load. I guessed it was due to the size and tried with a smaller size.
> > Everything worked with a smaller initrd. I also think that it is not a
> > good idea to use a 22MB initrd anyway so decided against the Debian
> > Bullseye initrd. 
> 
> Why is it a bad idea? In general, bigger file allows us to test corner cases.

That is also true.

This test is minimal, there is only dom0 booting, no domUs. To me, it
makes sense that also the initrd is small. In general 20-25MB is the
regular full size of a Linux arm64 rootfs. I think it makes sense to
stay below 10-15MB for arm32 if we can.

We could go up in size if we added the Xen tools and a domU to the
initrd and tested xl create. There is a test like that for arm64.

We can add more tests like that.


> > (For reference 22MB is basically the size of a fully
> > featured Yocto-build rootfs.) I did not file a bug to qemu-devel yet and
> > didn't investigate further on the QEMU side as I ran out of time.
> > 
> > Alpine Linux provides a very nice 2.5MB initrd. I tried to use both
> > kernel and initrd from Alpine Linux but unfortunately the Alpine Linux
> > kernel doesn't boot. I don't know why but I think it is because it might
> > be missing the console driver. I am not sure. There are a lot of
> > combinations that don't work and it is time consuming to investigate
> > them all. I have been trying to investigate only the most critical
> > things -- they are too many!
> > 
> > I should add that the Debian initrd is not the ideal initrd because it
> > is made to start the Debian installer. Here we just want a tiny busybox
> > initrd.
> > 
> > In general, I think it would be better if we could use the kernel and
> > initrd from the same source but I couldn't find one that works. I could
> > build one myself but it would be one more thing to maintain in
> > gitlab-ci. Also using u-boot might solve the problem of loading the
> > binary but again we would have to maintain a u-boot arm32 build in
> > gitlab-ci.
> > 
> > So in order of preference best to worst in my opinion:
> > 
> > 1) kernel and initrd from the same source
> > 2) kernel and initrd from different sources
> > 3) build your own kernel/initrd/u-boot
> > 
> > So I ended up doing 2). I tested it and it is sufficient to get the test
> > up and running.
> 
> Thanks for the explanation. So I think we should not call that an "Upgrade to
> Bullseye" because you are not using Debian. Instead, you borrowed a working
> kernel that happens to have everything you need built-in.
> 
> Also, can you update the commit message and provide a summary of this
> discussion?

Yes I can do that, thanks for the review!