[PATCH RESEND 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[PATCH RESEND 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by
the test-artifacts qemu container. The minimal test simply boots Xen
(built from previous build stages) and Dom0. The test is fetching the
Dom0 kernel and initrd from Debian Jessie: they work just fine and this
way we don't have to maintain a build for them too.

Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
---
 automation/gitlab-ci/test.yaml         | 23 ++++++++
 automation/scripts/qemu-smoke-arm32.sh | 72 ++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100755 automation/scripts/qemu-smoke-arm32.sh

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index d3b670b156..7f942dd64b 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -71,6 +71,29 @@ qemu-smoke-arm64-gcc:
     - /^coverity-tested\/.*/
     - /^stable-.*/
 
+qemu-smoke-arm32-gcc:
+  stage: test
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
+  dependencies:
+    - debian-unstable-gcc-arm32
+    - qemu-system-aarch64-6.0.0-arm32-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+  except:
+    - master
+    - smoke
+    - /^coverity-tested\/.*/
+    - /^stable-.*/
+
 qemu-smoke-x86-64-gcc:
   stage: test
   image: registry.gitlab.com/xen-project/xen/${CONTAINER}
diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
new file mode 100755
index 0000000000..4126be2042
--- /dev/null
+++ b/automation/scripts/qemu-smoke-arm32.sh
@@ -0,0 +1,72 @@
+#!/bin/bash
+
+set -ex
+
+# Install QEMU
+export DEBIAN_FRONTENT=noninteractive
+apt-get -qy update
+apt-get -qy install --no-install-recommends device-tree-compiler \
+                                            curl
+
+cd binaries
+curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
+curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
+
+kernel=`stat -L --printf="%s" vmlinuz`
+initrd=`stat -L --printf="%s" initrd.gz`
+
+# XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
+curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
+./qemu-system-arm \
+   -machine virt-6.0 \
+   -machine virtualization=true \
+   -smp 4 \
+   -m 2048 \
+   -serial stdio \
+   -monitor none \
+   -display none \
+   -machine dumpdtb=virt.dtb
+
+dtc -I dtb -O dts virt.dtb > virt.dts
+
+cat >> virt.dts << EOF
+/ {
+	chosen {
+		#address-cells = <0x2>;
+		#size-cells = <0x2>;
+		stdout-path = "/pl011@9000000";
+        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=1G dom0_max_vcpus=1 bootscrub=0";
+		xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh init=/bin/sh";
+		dom0 {
+			compatible = "xen,linux-zimage", "xen,multiboot-module";
+			reg = <0x0 0x1000000 0x0 $kernel>;
+		};
+        dom0-ramdisk {
+			compatible = "xen,linux-initrd", "xen,multiboot-module";
+			reg = <0x0 0x3200000 0x0 $initrd>;
+		};
+	};
+};
+EOF
+dtc -I dts -O dtb virt.dts > virt.dtb
+
+rm -f smoke.serial
+set +e
+timeout -k 1 240 \
+./qemu-system-arm \
+   -machine virt-6.0 \
+   -machine virtualization=true \
+   -smp 4 \
+   -m 2048 \
+   -serial stdio \
+   -monitor none \
+   -display none \
+   -dtb virt.dtb \
+   -no-reboot \
+   -kernel ./xen \
+   -device loader,file=./vmlinuz,addr=0x1000000 \
+   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
+
+set -e
+(grep -q "^BusyBox" smoke.serial) || exit 1
+exit 0
-- 
2.17.1

Re: [PATCH RESEND 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Julien Grall]

Hi Stefano,

On 14/07/2021 01:48, Stefano Stabellini wrote:
> Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by
> the test-artifacts qemu container. The minimal test simply boots Xen
> (built from previous build stages) and Dom0. The test is fetching the
> Dom0 kernel and initrd from Debian Jessie: they work just fine and this
> way we don't have to maintain a build for them too.

Debian Jessie is not in LTS anymore. I am entirely sure how long this 
will still be available (older release seems to have disappeared). I 
would suggest to use Bullseye or at least Buster.

> 
> Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> ---
>   automation/gitlab-ci/test.yaml         | 23 ++++++++
>   automation/scripts/qemu-smoke-arm32.sh | 72 ++++++++++++++++++++++++++
>   2 files changed, 95 insertions(+)
>   create mode 100755 automation/scripts/qemu-smoke-arm32.sh
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index d3b670b156..7f942dd64b 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -71,6 +71,29 @@ qemu-smoke-arm64-gcc:
>       - /^coverity-tested\/.*/
>       - /^stable-.*/
>   
> +qemu-smoke-arm32-gcc:
> +  stage: test
> +  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
> +  dependencies:
> +    - debian-unstable-gcc-arm32
> +    - qemu-system-aarch64-6.0.0-arm32-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +  except:
> +    - master
> +    - smoke
> +    - /^coverity-tested\/.*/
> +    - /^stable-.*/
> +
>   qemu-smoke-x86-64-gcc:
>     stage: test
>     image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
> new file mode 100755
> index 0000000000..4126be2042
> --- /dev/null
> +++ b/automation/scripts/qemu-smoke-arm32.sh
> @@ -0,0 +1,72 @@
> +#!/bin/bash
> +
> +set -ex
> +
> +# Install QEMU

The bits below doesn't seem to install QEMU. Do you mean "prepare the 
image"?

> +export DEBIAN_FRONTENT=noninteractive
> +apt-get -qy update
> +apt-get -qy install --no-install-recommends device-tree-compiler \
> +                                            curl
> +
> +cd binaries
> +curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
> +curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz

OOI, does this mean all our instances are running in the US?


> +
> +kernel=`stat -L --printf="%s" vmlinuz`
> +initrd=`stat -L --printf="%s" initrd.gz`
> +

NIT: I would suggest to add a comment similar to:

# For Xen, we need a couple of more node. Dump the DT from QEMU and add them

> +# XXX QEMU looks for "efi-virtio.rom" even if it is unneeded

Is this a problem with the package?

> +curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
> +./qemu-system-arm \
> +   -machine virt-6.0 \
> +   -machine virtualization=true \
> +   -smp 4 \
> +   -m 2048 \
> +   -serial stdio \
> +   -monitor none \
> +   -display none \
> +   -machine dumpdtb=virt.dtb
> +
> +dtc -I dtb -O dts virt.dtb > virt.dts

IIRC, Alex was working on add multiboot support in QEMU. Do you know 
what's the state?

> +
> +cat >> virt.dts << EOF
> +/ {
> +	chosen {
> +		#address-cells = <0x2>;
> +		#size-cells = <0x2>;
> +		stdout-path = "/pl011@9000000";
> +        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=1G dom0_max_vcpus=1 bootscrub=0";

Given that QEMU will expose 4 CPUs, any reason to limit dom0 to 1?

> +		xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh init=/bin/sh";
> +		dom0 {
> +			compatible = "xen,linux-zimage", "xen,multiboot-module";
> +			reg = <0x0 0x1000000 0x0 $kernel>;
> +		};
> +        dom0-ramdisk {
> +			compatible = "xen,linux-initrd", "xen,multiboot-module";
> +			reg = <0x0 0x3200000 0x0 $initrd>;
> +		};
> +	};
> +};
> +EOF
> +dtc -I dts -O dtb virt.dts > virt.dtb
> +
> +rm -f smoke.serial
> +set +e
> +timeout -k 1 240 \
> +./qemu-system-arm \
> +   -machine virt-6.0 \
> +   -machine virtualization=true \
> +   -smp 4 \
> +   -m 2048 \

Given seem to be a bit wasteful (and will definitely slow down Xen 
boot). As this is a smoke, can we consider to drop to 512?

> +   -serial stdio \
> +   -monitor none \
> +   -display none \
> +   -dtb virt.dtb \
> +   -no-reboot \
> +   -kernel ./xen \
> +   -device loader,file=./vmlinuz,addr=0x1000000 \
> +   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
> +
> +set -e
> +(grep -q "^BusyBox" smoke.serial) || exit 1

So this look for "BusyBox" anywhere in the log. What if there is a crash 
just after "BusyBox"? Is there something different we can grep?

> +exit 0 >

Cheers,

-- 
Julien Grall

Re: [PATCH RESEND 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

+Anthony, Bertrand


Apologies, this feel through the cracks!

Also considering the recent arm32 xen breakage, which could have been
caught by gitlab-ci before commit, I am keen on getting this short
series in.  Better late than never :-)

I forward-ported the series to staging with minimal changes and it still
works! I'll send out an update shortly to xen-devel. I am also replying
here because there is an interesting open question about Debian Jessie
below.

One note: in case you didn't notice, we already have a couple of arm64
and one x86 test in gitlab-ci using QEMU and they all follow the same
pattern, with similar scripts to the one added here, see for instance
automation/scripts/qemu-smoke-arm64.sh.


On Wed, 21 Jul 2021, Julien Grall wrote:
> Hi Stefano,
> 
> On 14/07/2021 01:48, Stefano Stabellini wrote:
> > Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by
> > the test-artifacts qemu container. The minimal test simply boots Xen
> > (built from previous build stages) and Dom0. The test is fetching the
> > Dom0 kernel and initrd from Debian Jessie: they work just fine and this
> > way we don't have to maintain a build for them too.
> 
> Debian Jessie is not in LTS anymore. I am entirely sure how long this will
> still be available (older release seems to have disappeared). I would suggest
> to use Bullseye or at least Buster.

This is a good point and unfortunately I don't have a good answer. I
tried to use Bullseye and Buster kernels but neither of them work. I
don't know why. I tried to use a couple of kernels from Alpine Linux,
but they also don't work.

Curious I started bisecting kernels using "make multi_v7_defconfig +
CONFIG_XEN" and discovered that something broke with the 4.9 release.
Up until 4.8 they work fine in the default config. I have not
investigated further yet.

We still have a few months of support in Debian Jessie. My suggestion is
to go with Debian Jessie for now in this series. Then depending on the
result of the kernel investigation we can decide what to do.

I could also move the fetching of the kernel and initrd from
qemu-smoke-arm32.sh to the containiner build so that if Debian Jessie
goes offline it wouldn't affect us immediately. Not required as of now,
but could be good to do.

 
> > Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > ---
> >   automation/gitlab-ci/test.yaml         | 23 ++++++++
> >   automation/scripts/qemu-smoke-arm32.sh | 72 ++++++++++++++++++++++++++
> >   2 files changed, 95 insertions(+)
> >   create mode 100755 automation/scripts/qemu-smoke-arm32.sh
> > 
> > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > index d3b670b156..7f942dd64b 100644
> > --- a/automation/gitlab-ci/test.yaml
> > +++ b/automation/gitlab-ci/test.yaml
> > @@ -71,6 +71,29 @@ qemu-smoke-arm64-gcc:
> >       - /^coverity-tested\/.*/
> >       - /^stable-.*/
> >   +qemu-smoke-arm32-gcc:
> > +  stage: test
> > +  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> > +  variables:
> > +    CONTAINER: debian:unstable-arm64v8
> > +  script:
> > +    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee
> > qemu-smoke-arm32.log
> > +  dependencies:
> > +    - debian-unstable-gcc-arm32
> > +    - qemu-system-aarch64-6.0.0-arm32-export
> > +  artifacts:
> > +    paths:
> > +      - smoke.serial
> > +      - '*.log'
> > +    when: always
> > +  tags:
> > +    - arm64
> > +  except:
> > +    - master
> > +    - smoke
> > +    - /^coverity-tested\/.*/
> > +    - /^stable-.*/
> > +
> >   qemu-smoke-x86-64-gcc:
> >     stage: test
> >     image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> > diff --git a/automation/scripts/qemu-smoke-arm32.sh
> > b/automation/scripts/qemu-smoke-arm32.sh
> > new file mode 100755
> > index 0000000000..4126be2042
> > --- /dev/null
> > +++ b/automation/scripts/qemu-smoke-arm32.sh
> > @@ -0,0 +1,72 @@
> > +#!/bin/bash
> > +
> > +set -ex
> > +
> > +# Install QEMU
> 
> The bits below doesn't seem to install QEMU. Do you mean "prepare the image"?

I updated the comment


> > +export DEBIAN_FRONTENT=noninteractive
> > +apt-get -qy update
> > +apt-get -qy install --no-install-recommends device-tree-compiler \
> > +                                            curl
> > +
> > +cd binaries
> > +curl -fsSLO
> > http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
> > +curl -fsSLO
> > http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
> 
> OOI, does this mean all our instances are running in the US?

Yes. However, I wanted to replace this link with one of those links that
serves you the best mirror available at the moment. But I couldn't find
any. So I left it as is.

 
> > +
> > +kernel=`stat -L --printf="%s" vmlinuz`
> > +initrd=`stat -L --printf="%s" initrd.gz`
> > +
> 
> NIT: I would suggest to add a comment similar to:
> 
> # For Xen, we need a couple of more node. Dump the DT from QEMU and add them

I updated the comment


> > +# XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
> 
> Is this a problem with the package?

Yes it is a problem with both QEMU (which shouldn't be looking for
efi-virtio.rom when calling dumpdtb) and also the package (that doesn't
provide it).

We have the same workaround in all the qemu scripts under automation.


> > +curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
> > +./qemu-system-arm \
> > +   -machine virt-6.0 \
> > +   -machine virtualization=true \
> > +   -smp 4 \
> > +   -m 2048 \
> > +   -serial stdio \
> > +   -monitor none \
> > +   -display none \
> > +   -machine dumpdtb=virt.dtb
> > +
> > +dtc -I dtb -O dts virt.dtb > virt.dts
> 
> IIRC, Alex was working on add multiboot support in QEMU. Do you know what's
> the state?

I don't know the latest


> > +
> > +cat >> virt.dts << EOF
> > +/ {
> > +	chosen {
> > +		#address-cells = <0x2>;
> > +		#size-cells = <0x2>;
> > +		stdout-path = "/pl011@9000000";
> > +        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000
> > dom0_mem=1G dom0_max_vcpus=1 bootscrub=0";
> 
> Given that QEMU will expose 4 CPUs, any reason to limit dom0 to 1?

Good idea, I did that


> > +		xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk
> > clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh init=/bin/sh";
> > +		dom0 {
> > +			compatible = "xen,linux-zimage",
> > "xen,multiboot-module";
> > +			reg = <0x0 0x1000000 0x0 $kernel>;
> > +		};
> > +        dom0-ramdisk {
> > +			compatible = "xen,linux-initrd",
> > "xen,multiboot-module";
> > +			reg = <0x0 0x3200000 0x0 $initrd>;
> > +		};
> > +	};
> > +};
> > +EOF
> > +dtc -I dts -O dtb virt.dts > virt.dtb
> > +
> > +rm -f smoke.serial
> > +set +e
> > +timeout -k 1 240 \
> > +./qemu-system-arm \
> > +   -machine virt-6.0 \
> > +   -machine virtualization=true \
> > +   -smp 4 \
> > +   -m 2048 \
> 
> Given seem to be a bit wasteful (and will definitely slow down Xen boot). As
> this is a smoke, can we consider to drop to 512?

Yes, also good idea. I dropped dom0_mem to 512 and here -m 1024. Less
than that, it leads to a boot error (I haven't investigated).


> > +   -serial stdio \
> > +   -monitor none \
> > +   -display none \
> > +   -dtb virt.dtb \
> > +   -no-reboot \
> > +   -kernel ./xen \
> > +   -device loader,file=./vmlinuz,addr=0x1000000 \
> > +   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
> > +
> > +set -e
> > +(grep -q "^BusyBox" smoke.serial) || exit 1
> 
> So this look for "BusyBox" anywhere in the log. What if there is a crash just
> after "BusyBox"? Is there something different we can grep?

I am not sure, there isn't much after "BusyBox" because the initrd is
basically empty (appended below the last few lines for your reference).
Also keeping in mind that all the QEMU tests today check for "BusyBox",
I would leave it as is.

---
BusyBox v1.22.1 (Debian 1:1.22.0-9+deb8u1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
/bin/sh: can't access tty; job control turned off
~ #

Re: [PATCH RESEND 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Anthony PERARD]

On Wed, Mar 16, 2022 at 06:38:53PM -0700, Stefano Stabellini wrote:
> Also considering the recent arm32 xen breakage, which could have been
> caught by gitlab-ci before commit,

I'm not sure that's true. I think the commits you are speaking about
also break the build on x86, which was caught by the gitlab ci.

Anyway, some arm32 smoke tests on gitlab should be useful.

Cheers,

-- 
Anthony PERARD

Re: [PATCH RESEND 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

On Fri, 18 Mar 2022, Anthony PERARD wrote:
> On Wed, Mar 16, 2022 at 06:38:53PM -0700, Stefano Stabellini wrote:
> > Also considering the recent arm32 xen breakage, which could have been
> > caught by gitlab-ci before commit,
> 
> I'm not sure that's true. I think the commits you are speaking about
> also break the build on x86, which was caught by the gitlab ci.
> 
> Anyway, some arm32 smoke tests on gitlab should be useful.

I think we are probably talking about different breakages :-)

Ayan recently broke Xen on ARM32 (run-time not build-time) with the
commit 9e5a68a66 and fef5531fd. I verified that the QEMU32 test in this
series actually catches the failure.

Can I get an ack on this? :-)

Re: [PATCH RESEND 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Anthony PERARD]

On Fri, Mar 18, 2022 at 05:15:06PM -0700, Stefano Stabellini wrote:
> On Fri, 18 Mar 2022, Anthony PERARD wrote:
> > On Wed, Mar 16, 2022 at 06:38:53PM -0700, Stefano Stabellini wrote:
> > > Also considering the recent arm32 xen breakage, which could have been
> > > caught by gitlab-ci before commit,
> > 
> > I'm not sure that's true. I think the commits you are speaking about
> > also break the build on x86, which was caught by the gitlab ci.
> > 
> > Anyway, some arm32 smoke tests on gitlab should be useful.
> 
> I think we are probably talking about different breakages :-)
> 
> Ayan recently broke Xen on ARM32 (run-time not build-time) with the
> commit 9e5a68a66 and fef5531fd. I verified that the QEMU32 test in this
> series actually catches the failure.

See the pipeline on this commit:
https://gitlab.com/xen-project/xen/-/commit/fef5531fd
https://gitlab.com/xen-project/xen/-/pipelines/491963118

;-)

-- 
Anthony PERARD

Re: [PATCH RESEND 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

On Mon, 21 Mar 2022, Anthony PERARD wrote:
> On Fri, Mar 18, 2022 at 05:15:06PM -0700, Stefano Stabellini wrote:
> > On Fri, 18 Mar 2022, Anthony PERARD wrote:
> > > On Wed, Mar 16, 2022 at 06:38:53PM -0700, Stefano Stabellini wrote:
> > > > Also considering the recent arm32 xen breakage, which could have been
> > > > caught by gitlab-ci before commit,
> > > 
> > > I'm not sure that's true. I think the commits you are speaking about
> > > also break the build on x86, which was caught by the gitlab ci.
> > > 
> > > Anyway, some arm32 smoke tests on gitlab should be useful.
> > 
> > I think we are probably talking about different breakages :-)
> > 
> > Ayan recently broke Xen on ARM32 (run-time not build-time) with the
> > commit 9e5a68a66 and fef5531fd. I verified that the QEMU32 test in this
> > series actually catches the failure.
> 
> See the pipeline on this commit:
> https://gitlab.com/xen-project/xen/-/commit/fef5531fd
> https://gitlab.com/xen-project/xen/-/pipelines/491963118
> 
> ;-)

Ah! Now I understand, thank you!