[PATCH 0/9] GitLab CI cleanup & improvements for Arm

[PATCH 0/9] GitLab CI cleanup & improvements for Arm

[Michal Orzel]

This patch series performs necessary cleanup and improvements in our
GitLab CI automation for Arm. This is crucial so that in the future
we can focus on adding new tests instead of spending time to fix
issues, making the behavior consistent, removing ambiguity, etc.

With the increased interest in gitlab-ci and using it locally, there
is still some work to do mainly to improve documentation. This however,
will be handled in the future series.

Michal Orzel (9):
  automation: Use custom build jobs when extra config options are needed
  automation: Add randconfig build jobs for arm64 alpine container
  automation: Add debug versions of Arm tests
  automation: Add Arm containers to containerize script
  automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder
  automation: qemu-alpine-arm64: Cleanup and fixes
  automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh
  automation: Rename qemu-alpine-arm64.sh to qemu-smoke-dom0-arm64.sh
  automation: Rename qemu-smoke-arm32.sh to qemu-smoke-dom0-arm32.sh

 automation/gitlab-ci/build.yaml               |  43 +++++++
 automation/gitlab-ci/test.yaml                | 117 ++++++++++++++++--
 automation/scripts/build                      |   8 +-
 automation/scripts/containerize               |   3 +
 ...moke-arm32.sh => qemu-smoke-dom0-arm32.sh} |  57 ++++-----
 ...pine-arm64.sh => qemu-smoke-dom0-arm64.sh} |  14 +--
 ...-arm64.sh => qemu-smoke-dom0less-arm64.sh} |   6 +-
 7 files changed, 190 insertions(+), 58 deletions(-)
 rename automation/scripts/{qemu-smoke-arm32.sh => qemu-smoke-dom0-arm32.sh} (55%)
 rename automation/scripts/{qemu-alpine-arm64.sh => qemu-smoke-dom0-arm64.sh} (88%)
 rename automation/scripts/{qemu-smoke-arm64.sh => qemu-smoke-dom0less-arm64.sh} (97%)

-- 
2.25.1

[PATCH 1/9] automation: Use custom build jobs when extra config options are needed

[Michal Orzel]

Currently, all the arm64 defconfig build jobs, regardless of the
container used, end up building Xen with the extra config options
specified in the main build script (e.g. CONFIG_EXPERT,
CONFIG_STATIC_MEMORY). Because these options are only needed for
specific test jobs, the current behavior of the CI is incorrect
as we add the extra options to all the defconfig builds. This means
that on arm64 there is not a single job performing proper defconfig build.

To fix this issue, add custom build jobs each time there is a need for
building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
variable to be used by these jobs to store the required options. This
variable will be then read by the main build script to modify the .config
file. This will also help users to understand what is needed to run specific
test.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
This patch could actually be consider to be taken for 4.17 release.
The reason why is because currently our CI for arm64 does not even
peform clean defconfig build which is quite crucial target to be tested.
Performing builds always with EXPERT and UNSUPPORTED is not something so
beneficial for release tests. This is up to the release manager.
---
 automation/gitlab-ci/build.yaml | 15 +++++++++++++++
 automation/gitlab-ci/test.yaml  |  4 ++--
 automation/scripts/build        |  8 ++------
 3 files changed, 19 insertions(+), 8 deletions(-)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 720ce6e07ba0..a39ed72aac6d 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -566,6 +566,21 @@ alpine-3.12-gcc-debug-arm64:
   variables:
     CONTAINER: alpine:3.12-arm64v8
 
+alpine-3.12-gcc-arm64-staticmem:
+  extends: .gcc-arm64-build
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    EXTRA_XEN_CONFIG: |
+      CONFIG_EXPERT=y
+      CONFIG_UNSUPPORTED=y
+      CONFIG_STATIC_MEMORY=y
+
+alpine-3.12-gcc-arm64-boot-cpupools:
+  extends: .gcc-arm64-build
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    EXTRA_XEN_CONFIG: |
+      CONFIG_BOOT_TIME_CPUPOOLS=y
 
 ## Test artifacts common
 
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index d899b3bdbf7a..4f96e6e322de 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -88,7 +88,7 @@ qemu-smoke-arm64-gcc-staticmem:
   script:
     - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
   needs:
-    - alpine-3.12-gcc-arm64
+    - alpine-3.12-gcc-arm64-staticmem
     - alpine-3.12-arm64-rootfs-export
     - kernel-5.19-arm64-export
     - qemu-system-aarch64-6.0.0-arm64-export
@@ -107,7 +107,7 @@ qemu-smoke-arm64-gcc-boot-cpupools:
   script:
     - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
   needs:
-    - alpine-3.12-gcc-arm64
+    - alpine-3.12-gcc-arm64-boot-cpupools
     - alpine-3.12-arm64-rootfs-export
     - kernel-5.19-arm64-export
     - qemu-system-aarch64-6.0.0-arm64-export
diff --git a/automation/scripts/build b/automation/scripts/build
index 2f15ab3198e6..7d441cedb4ae 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -15,12 +15,8 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
     make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
     hypervisor_only="y"
 else
-    if [[ "${XEN_TARGET_ARCH}" = "arm64" ]]; then
-        echo "
-CONFIG_EXPERT=y
-CONFIG_UNSUPPORTED=y
-CONFIG_STATIC_MEMORY=y
-CONFIG_BOOT_TIME_CPUPOOLS=y" > xen/.config
+    if [ -n "${EXTRA_XEN_CONFIG}" ]; then
+        echo "${EXTRA_XEN_CONFIG}" > xen/.config
         make -j$(nproc) -C xen olddefconfig
     else
         make -j$(nproc) -C xen defconfig
-- 
2.25.1

[PATCH 2/9] automation: Add randconfig build jobs for arm64 alpine container

[Michal Orzel]

For arm64 we perform builds using debian and alpine containers.
We are missing the randconfig build jobs for the latter, so add them.
This way for each container we have 4 fundamental build jobs:
- defconfig non-debug/debug
- randconfig non-debug/debug

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
 automation/gitlab-ci/build.yaml | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index a39ed72aac6d..d931441b910a 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -566,6 +566,18 @@ alpine-3.12-gcc-debug-arm64:
   variables:
     CONTAINER: alpine:3.12-arm64v8
 
+alpine-3.12-gcc-arm64-randconfig:
+  extends: .gcc-arm64-build
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    RANDCONFIG: y
+
+alpine-3.12-gcc-debug-arm64-randconfig:
+  extends: .gcc-arm64-build-debug
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    RANDCONFIG: y
+
 alpine-3.12-gcc-arm64-staticmem:
   extends: .gcc-arm64-build
   variables:
-- 
2.25.1

[PATCH 3/9] automation: Add debug versions of Arm tests

[Michal Orzel]

At the moment, all the tests are executed on non-debug Xen builds.
To improve the coverage (e.g. we might catch some asserts), add new
test jobs using debug Xen builds.

Take the opportunity to increase RAM size for QEMU from 1GB to 2GB
in qemu-smoke-arm64.sh as the debug builds take more space and we might
end up in a situation when there is not enough free space (especially
during a static memory test that reserves some region in the middle).

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
 automation/gitlab-ci/build.yaml        | 16 +++++
 automation/gitlab-ci/test.yaml         | 93 ++++++++++++++++++++++++++
 automation/scripts/qemu-smoke-arm64.sh |  6 +-
 3 files changed, 112 insertions(+), 3 deletions(-)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index d931441b910a..6ce92531f61f 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -587,6 +587,15 @@ alpine-3.12-gcc-arm64-staticmem:
       CONFIG_UNSUPPORTED=y
       CONFIG_STATIC_MEMORY=y
 
+alpine-3.12-gcc-debug-arm64-staticmem:
+  extends: .gcc-arm64-build-debug
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    EXTRA_XEN_CONFIG: |
+      CONFIG_EXPERT=y
+      CONFIG_UNSUPPORTED=y
+      CONFIG_STATIC_MEMORY=y
+
 alpine-3.12-gcc-arm64-boot-cpupools:
   extends: .gcc-arm64-build
   variables:
@@ -594,6 +603,13 @@ alpine-3.12-gcc-arm64-boot-cpupools:
     EXTRA_XEN_CONFIG: |
       CONFIG_BOOT_TIME_CPUPOOLS=y
 
+alpine-3.12-gcc-debug-arm64-boot-cpupools:
+  extends: .gcc-arm64-build-debug
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    EXTRA_XEN_CONFIG: |
+      CONFIG_BOOT_TIME_CPUPOOLS=y
+
 ## Test artifacts common
 
 .test-jobs-artifact-common:
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 4f96e6e322de..1b51030c6175 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -44,6 +44,25 @@ qemu-alpine-arm64-gcc:
   tags:
     - arm64
 
+qemu-alpine-arm64-gcc-debug:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+  needs:
+    - alpine-3.12-gcc-debug-arm64
+    - alpine-3.12-arm64-rootfs-export
+    - kernel-5.19-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-alpine-x86_64-gcc:
   extends: .test-jobs-common
   variables:
@@ -81,6 +100,25 @@ qemu-smoke-arm64-gcc:
   tags:
     - arm64
 
+qemu-smoke-arm64-gcc-debug:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+  needs:
+    - alpine-3.12-gcc-debug-arm64
+    - alpine-3.12-arm64-rootfs-export
+    - kernel-5.19-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-smoke-arm64-gcc-staticmem:
   extends: .test-jobs-common
   variables:
@@ -100,6 +138,25 @@ qemu-smoke-arm64-gcc-staticmem:
   tags:
     - arm64
 
+qemu-smoke-arm64-gcc-debug-staticmem:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
+  needs:
+    - alpine-3.12-gcc-debug-arm64-staticmem
+    - alpine-3.12-arm64-rootfs-export
+    - kernel-5.19-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-smoke-arm64-gcc-boot-cpupools:
   extends: .test-jobs-common
   variables:
@@ -119,6 +176,25 @@ qemu-smoke-arm64-gcc-boot-cpupools:
   tags:
     - arm64
 
+qemu-smoke-arm64-gcc-debug-boot-cpupools:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
+  needs:
+    - alpine-3.12-gcc-debug-arm64-boot-cpupools
+    - alpine-3.12-arm64-rootfs-export
+    - kernel-5.19-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-smoke-arm32-gcc:
   extends: .test-jobs-common
   variables:
@@ -136,6 +212,23 @@ qemu-smoke-arm32-gcc:
   tags:
     - arm64
 
+qemu-smoke-arm32-gcc-debug:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
+  needs:
+    - debian-unstable-gcc-arm32-debug
+    - qemu-system-aarch64-6.0.0-arm32-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-smoke-x86-64-gcc:
   extends: .test-jobs-common
   variables:
diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-arm64.sh
index f803835779f4..dea26c6c0a8f 100755
--- a/automation/scripts/qemu-smoke-arm64.sh
+++ b/automation/scripts/qemu-smoke-arm64.sh
@@ -52,7 +52,7 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
 ./binaries/qemu-system-aarch64 \
    -machine virtualization=true \
    -cpu cortex-a57 -machine type=virt \
-   -m 1024 -smp 2 -display none \
+   -m 2048 -smp 2 -display none \
    -machine dumpdtb=binaries/virt-gicv2.dtb
 
 # XXX disable pl061 to avoid Linux crash
@@ -117,7 +117,7 @@ cd ..
 
 # ImageBuilder
 echo 'MEMORY_START="0x40000000"
-MEMORY_END="0x80000000"
+MEMORY_END="0xC0000000"
 
 DEVICE_TREE="virt-gicv2.dtb"
 XEN="xen"
@@ -158,7 +158,7 @@ timeout -k 1 240 \
 ./binaries/qemu-system-aarch64 \
     -machine virtualization=true \
     -cpu cortex-a57 -machine type=virt \
-    -m 1024 -monitor none -serial stdio \
+    -m 2048 -monitor none -serial stdio \
     -smp 2 \
     -no-reboot \
     -device virtio-net-pci,netdev=n0 \
-- 
2.25.1

[PATCH 4/9] automation: Add Arm containers to containerize script

[Michal Orzel]

Script automation/scripts/containerize makes it easy to build Xen within
predefined containers from gitlab container registry. However, it is
currently not possible to use it with Arm containers because they are not
listed in the script. Populate the necessary entries.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
 automation/scripts/containerize | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 9d4beca4fa4b..0f4645c4cccb 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -25,6 +25,7 @@ die() {
 BASE="registry.gitlab.com/xen-project/xen"
 case "_${CONTAINER}" in
     _alpine) CONTAINER="${BASE}/alpine:3.12" ;;
+    _alpine-arm64v8) CONTAINER="${BASE}/alpine:3.12-arm64v8" ;;
     _archlinux|_arch) CONTAINER="${BASE}/archlinux:current" ;;
     _riscv64) CONTAINER="${BASE}/archlinux:riscv64" ;;
     _centos7) CONTAINER="${BASE}/centos:7" ;;
@@ -35,6 +36,8 @@ case "_${CONTAINER}" in
     _stretch|_) CONTAINER="${BASE}/debian:stretch" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _unstable|_) CONTAINER="${BASE}/debian:unstable" ;;
+    _unstable-arm32-gcc) CONTAINER="${BASE}/debian:unstable-arm32-gcc" ;;
+    _unstable-arm64v8) CONTAINER="${BASE}/debian:unstable-arm64v8" ;;
     _trusty) CONTAINER="${BASE}/ubuntu:trusty" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
-- 
2.25.1

[PATCH 5/9] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder

[Michal Orzel]

Take an example from arm64 qemu test scripts and use ImageBuilder
to generate u-boot script automatically. Calculating the addresses
manually is quite error prone and also we will be able to benefit
from using ImageBuilder when adding domUs to this test in the future.

Install and use u-boot from the debian package.
Modify the script so that binaries are loaded from u-boot via tftp.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
 automation/scripts/qemu-smoke-arm32.sh | 57 ++++++++++++--------------
 1 file changed, 27 insertions(+), 30 deletions(-)

diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
index 530f3892fdd3..765facbe4d66 100755
--- a/automation/scripts/qemu-smoke-arm32.sh
+++ b/automation/scripts/qemu-smoke-arm32.sh
@@ -4,7 +4,9 @@ set -ex
 
 export DEBIAN_FRONTENT=noninteractive
 apt-get -qy update
-apt-get -qy install --no-install-recommends device-tree-compiler \
+apt-get -qy install --no-install-recommends u-boot-qemu \
+                                            u-boot-tools \
+                                            device-tree-compiler \
                                             curl \
                                             cpio
 
@@ -20,10 +22,6 @@ tar xvzf ../initrd.tar.gz
 find . | cpio -H newc -o | gzip > ../initrd.gz
 cd ..
 
-kernel=`stat -L --printf="%s" vmlinuz`
-initrd=`stat -L --printf="%s" initrd.gz`
-
-# For Xen, we need a couple of more node. Dump the DT from QEMU and add them
 # XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
 curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
 ./qemu-system-arm \
@@ -36,31 +34,31 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
    -display none \
    -machine dumpdtb=virt.dtb
 
-dtc -I dtb -O dts virt.dtb > virt.dts
+# ImageBuilder
+echo 'MEMORY_START="0x40000000"
+MEMORY_END="0x80000000"
+
+DEVICE_TREE="virt.dtb"
+XEN="xen"
+DOM0_KERNEL="vmlinuz"
+DOM0_RAMDISK="initrd.gz"
+DOM0_CMD="console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh"
+XEN_CMD="console=dtuart dom0_mem=512M bootscrub=0"
+
+NUM_DOMUS=0
+
+LOAD_CMD="tftpb"
+BOOT_CMD="bootm"
+UBOOT_SOURCE="boot.source"
+UBOOT_SCRIPT="boot.scr"' > config
 
-cat >> virt.dts << EOF
-/ {
-	chosen {
-		#address-cells = <0x2>;
-		#size-cells = <0x2>;
-		stdout-path = "/pl011@9000000";
-        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=512M bootscrub=0";
-		xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh";
-		dom0 {
-			compatible = "xen,linux-zimage", "xen,multiboot-module";
-			reg = <0x0 0x1000000 0x0 $kernel>;
-		};
-        dom0-ramdisk {
-			compatible = "xen,linux-initrd", "xen,multiboot-module";
-			reg = <0x0 0x3200000 0x0 $initrd>;
-		};
-	};
-};
-EOF
-dtc -I dts -O dtb virt.dts > virt.dtb
+rm -rf imagebuilder
+git clone https://gitlab.com/ViryaOS/imagebuilder
+bash imagebuilder/scripts/uboot-script-gen -t tftp -d . -c config
 
 rm -f smoke.serial
 set +e
+echo "  virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x40000000"| \
 timeout -k 1 240 \
 ./qemu-system-arm \
    -machine virt \
@@ -70,11 +68,10 @@ timeout -k 1 240 \
    -serial stdio \
    -monitor none \
    -display none \
-   -dtb virt.dtb \
    -no-reboot \
-   -kernel ./xen \
-   -device loader,file=./vmlinuz,addr=0x1000000 \
-   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
+   -device virtio-net-pci,netdev=n0 \
+   -netdev user,id=n0,tftp=./ \
+   -bios /usr/lib/u-boot/qemu_arm/u-boot.bin |& tee smoke.serial
 
 set -e
 (grep -q "^/ #" smoke.serial) || exit 1
-- 
2.25.1

[PATCH 6/9] automation: qemu-alpine-arm64: Cleanup and fixes

[Michal Orzel]

Perform the following cleanup:
- rename the device tree from virt-gicv3 to virt-gicv2 as the GIC version
  used in this test is v2,
- use fdtput to perform modifications on the dtb,
- use DEBIAN_FRONTENT=noninteractive to prevent interactive prompt being
  stuck waiting for answer other than "yes",
- fix the number of cpus in the device tree because currently we generate
  it with a single cpu and try to run QEMU with two,
- fix the memory size we pass when generating QEMU device tree as it does
  not match the memory size with what we run QEMU.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
 automation/scripts/qemu-alpine-arm64.sh | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/automation/scripts/qemu-alpine-arm64.sh b/automation/scripts/qemu-alpine-arm64.sh
index f4ac2d856fa0..7b52d77d3c84 100755
--- a/automation/scripts/qemu-alpine-arm64.sh
+++ b/automation/scripts/qemu-alpine-arm64.sh
@@ -2,6 +2,7 @@
 
 set -ex
 
+export DEBIAN_FRONTENT=noninteractive
 apt-get -qy update
 apt-get -qy install --no-install-recommends u-boot-qemu \
                                             u-boot-tools \
@@ -73,18 +74,17 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
 ./binaries/qemu-system-aarch64 \
    -machine virtualization=true \
    -cpu cortex-a57 -machine type=virt \
-   -m 1024 -display none \
-   -machine dumpdtb=binaries/virt-gicv3.dtb
+   -m 2048 -smp 2 -display none \
+   -machine dumpdtb=binaries/virt-gicv2.dtb
+
 # XXX disable pl061 to avoid Linux crash
-dtc -I dtb -O dts binaries/virt-gicv3.dtb > binaries/virt-gicv3.dts
-sed 's/compatible = "arm,pl061.*/status = "disabled";/g' binaries/virt-gicv3.dts > binaries/virt-gicv3-edited.dts
-dtc -I dts -O dtb binaries/virt-gicv3-edited.dts > binaries/virt-gicv3.dtb
+fdtput binaries/virt-gicv2.dtb -p -t s /pl061@9030000 status disabled
 
 # ImageBuilder
 echo 'MEMORY_START="0x40000000"
-MEMORY_END="0x80000000"
+MEMORY_END="0xC0000000"
 
-DEVICE_TREE="virt-gicv3.dtb"
+DEVICE_TREE="virt-gicv2.dtb"
 XEN="xen"
 DOM0_KERNEL="Image"
 DOM0_RAMDISK="xen-rootfs.cpio.gz"
-- 
2.25.1

[PATCH 7/9] automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh

[Michal Orzel]

Testing arm64 is done using the qemu-alpine-arm64.sh and
qemu-smoke-arm64.sh scripts. These scripts are executed with exactly
the same artifacts (container, rootfs, kernel, qemu) and the only
difference is that the former is used to perform dom0 based testing
and the latter - dom0less based testing.

Because the current naming is quite umbiguous, rename qemu-smoke-arm64.sh
script to qemu-smoke-dom0less-arm64.sh to reflect its usage.

qemu-alpine-arm64.sh will be renamed in the follow-up patch.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
 automation/gitlab-ci/test.yaml                | 24 +++++++++----------
 ...-arm64.sh => qemu-smoke-dom0less-arm64.sh} |  0
 2 files changed, 12 insertions(+), 12 deletions(-)
 rename automation/scripts/{qemu-smoke-arm64.sh => qemu-smoke-dom0less-arm64.sh} (100%)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 1b51030c6175..f620622671f8 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -81,12 +81,12 @@ qemu-alpine-x86_64-gcc:
   tags:
     - x86_64
 
-qemu-smoke-arm64-gcc:
+qemu-smoke-dom0less-arm64-gcc:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-arm64
     - alpine-3.12-arm64-rootfs-export
@@ -100,12 +100,12 @@ qemu-smoke-arm64-gcc:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-debug:
+qemu-smoke-dom0less-arm64-gcc-debug:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-debug-arm64
     - alpine-3.12-arm64-rootfs-export
@@ -119,12 +119,12 @@ qemu-smoke-arm64-gcc-debug:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-staticmem:
+qemu-smoke-dom0less-arm64-gcc-staticmem:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-arm64-staticmem
     - alpine-3.12-arm64-rootfs-export
@@ -138,12 +138,12 @@ qemu-smoke-arm64-gcc-staticmem:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-debug-staticmem:
+qemu-smoke-dom0less-arm64-gcc-debug-staticmem:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-debug-arm64-staticmem
     - alpine-3.12-arm64-rootfs-export
@@ -157,12 +157,12 @@ qemu-smoke-arm64-gcc-debug-staticmem:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-boot-cpupools:
+qemu-smoke-dom0less-arm64-gcc-boot-cpupools:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-arm64-boot-cpupools
     - alpine-3.12-arm64-rootfs-export
@@ -176,12 +176,12 @@ qemu-smoke-arm64-gcc-boot-cpupools:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-debug-boot-cpupools:
+qemu-smoke-dom0less-arm64-gcc-debug-boot-cpupools:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-debug-arm64-boot-cpupools
     - alpine-3.12-arm64-rootfs-export
diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
similarity index 100%
rename from automation/scripts/qemu-smoke-arm64.sh
rename to automation/scripts/qemu-smoke-dom0less-arm64.sh
-- 
2.25.1

[PATCH 8/9] automation: Rename qemu-alpine-arm64.sh to qemu-smoke-dom0-arm64.sh

[Michal Orzel]

qemu-alpine-arm64.sh script is used to perform dom0 based testing.
Rename this script to qemu-smoke-dom0-arm64.sh to reflect its usage.
Also rename the corresponding test jobs.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
 automation/gitlab-ci/test.yaml                            | 8 ++++----
 .../{qemu-alpine-arm64.sh => qemu-smoke-dom0-arm64.sh}    | 0
 2 files changed, 4 insertions(+), 4 deletions(-)
 rename automation/scripts/{qemu-alpine-arm64.sh => qemu-smoke-dom0-arm64.sh} (100%)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index f620622671f8..3b147c88ab08 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -25,12 +25,12 @@ build-each-commit-gcc:
   tags:
     - x86_64
 
-qemu-alpine-arm64-gcc:
+qemu-smoke-dom0-arm64-gcc:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-arm64
     - alpine-3.12-arm64-rootfs-export
@@ -44,12 +44,12 @@ qemu-alpine-arm64-gcc:
   tags:
     - arm64
 
-qemu-alpine-arm64-gcc-debug:
+qemu-smoke-dom0-arm64-gcc-debug:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-debug-arm64
     - alpine-3.12-arm64-rootfs-export
diff --git a/automation/scripts/qemu-alpine-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
similarity index 100%
rename from automation/scripts/qemu-alpine-arm64.sh
rename to automation/scripts/qemu-smoke-dom0-arm64.sh
-- 
2.25.1

[PATCH 9/9] automation: Rename qemu-smoke-arm32.sh to qemu-smoke-dom0-arm32.sh

[Michal Orzel]

After qemu arm64 test scripts had been renamed to reflect their
usage, do the same for the qemu arm32 test script. Currently it only
boots dom0, so we can assume that this script will be used to perform
dom0 based testing. In the future we will be able to create corresponding
script qemu-smoke-dom0less-arm32.sh to perform dom0less based testing.
This is the last step to get rid of ambiguity with regards to naming
of Arm test scripts.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
 automation/gitlab-ci/test.yaml                            | 8 ++++----
 .../{qemu-smoke-arm32.sh => qemu-smoke-dom0-arm32.sh}     | 0
 2 files changed, 4 insertions(+), 4 deletions(-)
 rename automation/scripts/{qemu-smoke-arm32.sh => qemu-smoke-dom0-arm32.sh} (100%)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 3b147c88ab08..92e0a1f7c510 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -195,12 +195,12 @@ qemu-smoke-dom0less-arm64-gcc-debug-boot-cpupools:
   tags:
     - arm64
 
-qemu-smoke-arm32-gcc:
+qemu-smoke-dom0-arm32-gcc:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
+    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
   needs:
     - debian-unstable-gcc-arm32
     - qemu-system-aarch64-6.0.0-arm32-export
@@ -212,12 +212,12 @@ qemu-smoke-arm32-gcc:
   tags:
     - arm64
 
-qemu-smoke-arm32-gcc-debug:
+qemu-smoke-dom0-arm32-gcc-debug:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
+    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
   needs:
     - debian-unstable-gcc-arm32-debug
     - qemu-system-aarch64-6.0.0-arm32-export
diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-dom0-arm32.sh
similarity index 100%
rename from automation/scripts/qemu-smoke-arm32.sh
rename to automation/scripts/qemu-smoke-dom0-arm32.sh
-- 
2.25.1

Re: [PATCH 1/9] automation: Use custom build jobs when extra config options are needed

[Stefano Stabellini]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> Currently, all the arm64 defconfig build jobs, regardless of the
> container used, end up building Xen with the extra config options
> specified in the main build script (e.g. CONFIG_EXPERT,
> CONFIG_STATIC_MEMORY). Because these options are only needed for
> specific test jobs, the current behavior of the CI is incorrect
> as we add the extra options to all the defconfig builds. This means
> that on arm64 there is not a single job performing proper defconfig build.
> 
> To fix this issue, add custom build jobs each time there is a need for
> building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
> variable to be used by these jobs to store the required options. This
> variable will be then read by the main build script to modify the .config
> file. This will also help users to understand what is needed to run specific
> test.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> ---
> This patch could actually be consider to be taken for 4.17 release.
> The reason why is because currently our CI for arm64 does not even
> peform clean defconfig build which is quite crucial target to be tested.
> Performing builds always with EXPERT and UNSUPPORTED is not something so
> beneficial for release tests. This is up to the release manager.

+ Henry

I agree this should go in 4.17, so that gitlab-ci can test non-DEBUG
builds on ARM.


> ---
>  automation/gitlab-ci/build.yaml | 15 +++++++++++++++
>  automation/gitlab-ci/test.yaml  |  4 ++--
>  automation/scripts/build        |  8 ++------
>  3 files changed, 19 insertions(+), 8 deletions(-)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 720ce6e07ba0..a39ed72aac6d 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -566,6 +566,21 @@ alpine-3.12-gcc-debug-arm64:
>    variables:
>      CONTAINER: alpine:3.12-arm64v8
>  
> +alpine-3.12-gcc-arm64-staticmem:
> +  extends: .gcc-arm64-build
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |

Why the "|" ?

I was trying to look for its documentation in the gitlab yaml docs but
couldn't find it.


> +      CONFIG_EXPERT=y
> +      CONFIG_UNSUPPORTED=y
> +      CONFIG_STATIC_MEMORY=y
> +
> +alpine-3.12-gcc-arm64-boot-cpupools:
> +  extends: .gcc-arm64-build
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_BOOT_TIME_CPUPOOLS=y
>  
>  ## Test artifacts common
>  
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index d899b3bdbf7a..4f96e6e322de 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -88,7 +88,7 @@ qemu-smoke-arm64-gcc-staticmem:
>    script:
>      - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
>    needs:
> -    - alpine-3.12-gcc-arm64
> +    - alpine-3.12-gcc-arm64-staticmem
>      - alpine-3.12-arm64-rootfs-export
>      - kernel-5.19-arm64-export
>      - qemu-system-aarch64-6.0.0-arm64-export
> @@ -107,7 +107,7 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>    script:
>      - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
>    needs:
> -    - alpine-3.12-gcc-arm64
> +    - alpine-3.12-gcc-arm64-boot-cpupools
>      - alpine-3.12-arm64-rootfs-export
>      - kernel-5.19-arm64-export
>      - qemu-system-aarch64-6.0.0-arm64-export
> diff --git a/automation/scripts/build b/automation/scripts/build
> index 2f15ab3198e6..7d441cedb4ae 100755
> --- a/automation/scripts/build
> +++ b/automation/scripts/build
> @@ -15,12 +15,8 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
>      make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
>      hypervisor_only="y"
>  else
> -    if [[ "${XEN_TARGET_ARCH}" = "arm64" ]]; then
> -        echo "
> -CONFIG_EXPERT=y
> -CONFIG_UNSUPPORTED=y
> -CONFIG_STATIC_MEMORY=y
> -CONFIG_BOOT_TIME_CPUPOOLS=y" > xen/.config
> +    if [ -n "${EXTRA_XEN_CONFIG}" ]; then

NIT: for uniformity with rest of the file use

  if [[ -n "${EXTRA_XEN_CONFIG}" ]]; then


> +        echo "${EXTRA_XEN_CONFIG}" > xen/.config
>          make -j$(nproc) -C xen olddefconfig
>      else
>          make -j$(nproc) -C xen defconfig
> -- 
> 2.25.1
> 

Re: [PATCH 2/9] automation: Add randconfig build jobs for arm64 alpine container

[Stefano Stabellini]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> For arm64 we perform builds using debian and alpine containers.
> We are missing the randconfig build jobs for the latter, so add them.
> This way for each container we have 4 fundamental build jobs:
> - defconfig non-debug/debug
> - randconfig non-debug/debug

Should we have them also for Debian? The patch below looks fine.


> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> ---
>  automation/gitlab-ci/build.yaml | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index a39ed72aac6d..d931441b910a 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -566,6 +566,18 @@ alpine-3.12-gcc-debug-arm64:
>    variables:
>      CONTAINER: alpine:3.12-arm64v8
>  
> +alpine-3.12-gcc-arm64-randconfig:
> +  extends: .gcc-arm64-build
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    RANDCONFIG: y
> +
> +alpine-3.12-gcc-debug-arm64-randconfig:
> +  extends: .gcc-arm64-build-debug
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    RANDCONFIG: y
> +
>  alpine-3.12-gcc-arm64-staticmem:
>    extends: .gcc-arm64-build
>    variables:
> -- 
> 2.25.1
> 

Re: [PATCH 3/9] automation: Add debug versions of Arm tests

[Stefano Stabellini]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> At the moment, all the tests are executed on non-debug Xen builds.
> To improve the coverage (e.g. we might catch some asserts), add new
> test jobs using debug Xen builds.

This is interesting. On one hand, I agree it is good to test DEBUG and
non-DEBUG builds in runtime tests too, on the other hand I am worried
about our test capacity.

I take you have tried a few pipelines -- are the additional tests
causing the pipeline to take longer? Do we still have "room" because x86
is slower? What's your thinking?

One idea would be to only do:
- qemu-alpine-arm64-gcc-debug
- qemu-smoke-arm64-gcc-debug

and not the others


> Take the opportunity to increase RAM size for QEMU from 1GB to 2GB
> in qemu-smoke-arm64.sh as the debug builds take more space and we might
> end up in a situation when there is not enough free space (especially
> during a static memory test that reserves some region in the middle).
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> ---
>  automation/gitlab-ci/build.yaml        | 16 +++++
>  automation/gitlab-ci/test.yaml         | 93 ++++++++++++++++++++++++++
>  automation/scripts/qemu-smoke-arm64.sh |  6 +-
>  3 files changed, 112 insertions(+), 3 deletions(-)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index d931441b910a..6ce92531f61f 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -587,6 +587,15 @@ alpine-3.12-gcc-arm64-staticmem:
>        CONFIG_UNSUPPORTED=y
>        CONFIG_STATIC_MEMORY=y
>  
> +alpine-3.12-gcc-debug-arm64-staticmem:
> +  extends: .gcc-arm64-build-debug
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_EXPERT=y
> +      CONFIG_UNSUPPORTED=y
> +      CONFIG_STATIC_MEMORY=y
> +
>  alpine-3.12-gcc-arm64-boot-cpupools:
>    extends: .gcc-arm64-build
>    variables:
> @@ -594,6 +603,13 @@ alpine-3.12-gcc-arm64-boot-cpupools:
>      EXTRA_XEN_CONFIG: |
>        CONFIG_BOOT_TIME_CPUPOOLS=y
>  
> +alpine-3.12-gcc-debug-arm64-boot-cpupools:
> +  extends: .gcc-arm64-build-debug
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_BOOT_TIME_CPUPOOLS=y
> +
>  ## Test artifacts common
>  
>  .test-jobs-artifact-common:
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 4f96e6e322de..1b51030c6175 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -44,6 +44,25 @@ qemu-alpine-arm64-gcc:
>    tags:
>      - arm64
>  
> +qemu-alpine-arm64-gcc-debug:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-alpine-x86_64-gcc:
>    extends: .test-jobs-common
>    variables:
> @@ -81,6 +100,25 @@ qemu-smoke-arm64-gcc:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm64-gcc-debug:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-arm64-gcc-staticmem:
>    extends: .test-jobs-common
>    variables:
> @@ -100,6 +138,25 @@ qemu-smoke-arm64-gcc-staticmem:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm64-gcc-debug-staticmem:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64-staticmem
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-arm64-gcc-boot-cpupools:
>    extends: .test-jobs-common
>    variables:
> @@ -119,6 +176,25 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm64-gcc-debug-boot-cpupools:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64-boot-cpupools
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-arm32-gcc:
>    extends: .test-jobs-common
>    variables:
> @@ -136,6 +212,23 @@ qemu-smoke-arm32-gcc:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm32-gcc-debug:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
> +  needs:
> +    - debian-unstable-gcc-arm32-debug
> +    - qemu-system-aarch64-6.0.0-arm32-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-x86-64-gcc:
>    extends: .test-jobs-common
>    variables:
> diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-arm64.sh
> index f803835779f4..dea26c6c0a8f 100755
> --- a/automation/scripts/qemu-smoke-arm64.sh
> +++ b/automation/scripts/qemu-smoke-arm64.sh
> @@ -52,7 +52,7 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
>  ./binaries/qemu-system-aarch64 \
>     -machine virtualization=true \
>     -cpu cortex-a57 -machine type=virt \
> -   -m 1024 -smp 2 -display none \
> +   -m 2048 -smp 2 -display none \
>     -machine dumpdtb=binaries/virt-gicv2.dtb
>  
>  # XXX disable pl061 to avoid Linux crash
> @@ -117,7 +117,7 @@ cd ..
>  
>  # ImageBuilder
>  echo 'MEMORY_START="0x40000000"
> -MEMORY_END="0x80000000"
> +MEMORY_END="0xC0000000"
>  
>  DEVICE_TREE="virt-gicv2.dtb"
>  XEN="xen"
> @@ -158,7 +158,7 @@ timeout -k 1 240 \
>  ./binaries/qemu-system-aarch64 \
>      -machine virtualization=true \
>      -cpu cortex-a57 -machine type=virt \
> -    -m 1024 -monitor none -serial stdio \
> +    -m 2048 -monitor none -serial stdio \
>      -smp 2 \
>      -no-reboot \
>      -device virtio-net-pci,netdev=n0 \

The changes to qemu-smoke-arm64.sh make sense, but could you move them
to a separate patch in case we need to do any backports?

Also I would change MEMORY_END to 0xC0000000 in qemu-alpine-arm64.sh too
for uniformity.

Re: [PATCH 4/9] automation: Add Arm containers to containerize script

[Stefano Stabellini]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> Script automation/scripts/containerize makes it easy to build Xen within
> predefined containers from gitlab container registry. However, it is
> currently not possible to use it with Arm containers because they are not
> listed in the script. Populate the necessary entries.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/scripts/containerize | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/automation/scripts/containerize b/automation/scripts/containerize
> index 9d4beca4fa4b..0f4645c4cccb 100755
> --- a/automation/scripts/containerize
> +++ b/automation/scripts/containerize
> @@ -25,6 +25,7 @@ die() {
>  BASE="registry.gitlab.com/xen-project/xen"
>  case "_${CONTAINER}" in
>      _alpine) CONTAINER="${BASE}/alpine:3.12" ;;
> +    _alpine-arm64v8) CONTAINER="${BASE}/alpine:3.12-arm64v8" ;;
>      _archlinux|_arch) CONTAINER="${BASE}/archlinux:current" ;;
>      _riscv64) CONTAINER="${BASE}/archlinux:riscv64" ;;
>      _centos7) CONTAINER="${BASE}/centos:7" ;;
> @@ -35,6 +36,8 @@ case "_${CONTAINER}" in
>      _stretch|_) CONTAINER="${BASE}/debian:stretch" ;;
>      _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
>      _unstable|_) CONTAINER="${BASE}/debian:unstable" ;;
> +    _unstable-arm32-gcc) CONTAINER="${BASE}/debian:unstable-arm32-gcc" ;;
> +    _unstable-arm64v8) CONTAINER="${BASE}/debian:unstable-arm64v8" ;;
>      _trusty) CONTAINER="${BASE}/ubuntu:trusty" ;;
>      _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
>      _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
> -- 
> 2.25.1
> 

Re: [PATCH 5/9] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder

[Stefano Stabellini]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> Take an example from arm64 qemu test scripts and use ImageBuilder
> to generate u-boot script automatically. Calculating the addresses
> manually is quite error prone and also we will be able to benefit
> from using ImageBuilder when adding domUs to this test in the future.
> 
> Install and use u-boot from the debian package.
> Modify the script so that binaries are loaded from u-boot via tftp.

Great patch! It makes the test a lot better!


> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> ---
>  automation/scripts/qemu-smoke-arm32.sh | 57 ++++++++++++--------------
>  1 file changed, 27 insertions(+), 30 deletions(-)
> 
> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
> index 530f3892fdd3..765facbe4d66 100755
> --- a/automation/scripts/qemu-smoke-arm32.sh
> +++ b/automation/scripts/qemu-smoke-arm32.sh
> @@ -4,7 +4,9 @@ set -ex
>  
>  export DEBIAN_FRONTENT=noninteractive
>  apt-get -qy update
> -apt-get -qy install --no-install-recommends device-tree-compiler \
> +apt-get -qy install --no-install-recommends u-boot-qemu \
> +                                            u-boot-tools \
> +                                            device-tree-compiler \
>                                              curl \
>                                              cpio
>  
> @@ -20,10 +22,6 @@ tar xvzf ../initrd.tar.gz
>  find . | cpio -H newc -o | gzip > ../initrd.gz
>  cd ..
>  
> -kernel=`stat -L --printf="%s" vmlinuz`
> -initrd=`stat -L --printf="%s" initrd.gz`
> -
> -# For Xen, we need a couple of more node. Dump the DT from QEMU and add them
>  # XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
>  curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
>  ./qemu-system-arm \
> @@ -36,31 +34,31 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
>     -display none \
>     -machine dumpdtb=virt.dtb
>  
> -dtc -I dtb -O dts virt.dtb > virt.dts
> +# ImageBuilder
> +echo 'MEMORY_START="0x40000000"
> +MEMORY_END="0x80000000"
> +
> +DEVICE_TREE="virt.dtb"
> +XEN="xen"
> +DOM0_KERNEL="vmlinuz"
> +DOM0_RAMDISK="initrd.gz"
> +DOM0_CMD="console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh"
> +XEN_CMD="console=dtuart dom0_mem=512M bootscrub=0"

This is missing dtuart=/pl011@9000000 compared to the original


> +NUM_DOMUS=0
> +
> +LOAD_CMD="tftpb"
> +BOOT_CMD="bootm"

"bootm" because "booti" is not available on arm32, right?


> +UBOOT_SOURCE="boot.source"
> +UBOOT_SCRIPT="boot.scr"' > config
>  
> -cat >> virt.dts << EOF
> -/ {
> -	chosen {
> -		#address-cells = <0x2>;
> -		#size-cells = <0x2>;
> -		stdout-path = "/pl011@9000000";
> -        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=512M bootscrub=0";
> -		xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh";
> -		dom0 {
> -			compatible = "xen,linux-zimage", "xen,multiboot-module";
> -			reg = <0x0 0x1000000 0x0 $kernel>;
> -		};
> -        dom0-ramdisk {
> -			compatible = "xen,linux-initrd", "xen,multiboot-module";
> -			reg = <0x0 0x3200000 0x0 $initrd>;
> -		};
> -	};
> -};
> -EOF
> -dtc -I dts -O dtb virt.dts > virt.dtb
> +rm -rf imagebuilder
> +git clone https://gitlab.com/ViryaOS/imagebuilder
> +bash imagebuilder/scripts/uboot-script-gen -t tftp -d . -c config
>  
>  rm -f smoke.serial
>  set +e
> +echo "  virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x40000000"| \
>  timeout -k 1 240 \
>  ./qemu-system-arm \
>     -machine virt \
> @@ -70,11 +68,10 @@ timeout -k 1 240 \
>     -serial stdio \
>     -monitor none \
>     -display none \
> -   -dtb virt.dtb \
>     -no-reboot \
> -   -kernel ./xen \
> -   -device loader,file=./vmlinuz,addr=0x1000000 \
> -   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
> +   -device virtio-net-pci,netdev=n0 \
> +   -netdev user,id=n0,tftp=./ \
> +   -bios /usr/lib/u-boot/qemu_arm/u-boot.bin |& tee smoke.serial
>  
>  set -e
>  (grep -q "^/ #" smoke.serial) || exit 1
> -- 
> 2.25.1
> 

Re: [PATCH 6/9] automation: qemu-alpine-arm64: Cleanup and fixes

[Stefano Stabellini]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> Perform the following cleanup:
> - rename the device tree from virt-gicv3 to virt-gicv2 as the GIC version
>   used in this test is v2,
> - use fdtput to perform modifications on the dtb,
> - use DEBIAN_FRONTENT=noninteractive to prevent interactive prompt being
>   stuck waiting for answer other than "yes",
> - fix the number of cpus in the device tree because currently we generate
>   it with a single cpu and try to run QEMU with two,
> - fix the memory size we pass when generating QEMU device tree as it does
>   not match the memory size with what we run QEMU.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/scripts/qemu-alpine-arm64.sh | 14 +++++++-------
>  1 file changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git a/automation/scripts/qemu-alpine-arm64.sh b/automation/scripts/qemu-alpine-arm64.sh
> index f4ac2d856fa0..7b52d77d3c84 100755
> --- a/automation/scripts/qemu-alpine-arm64.sh
> +++ b/automation/scripts/qemu-alpine-arm64.sh
> @@ -2,6 +2,7 @@
>  
>  set -ex
>  
> +export DEBIAN_FRONTENT=noninteractive
>  apt-get -qy update
>  apt-get -qy install --no-install-recommends u-boot-qemu \
>                                              u-boot-tools \
> @@ -73,18 +74,17 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
>  ./binaries/qemu-system-aarch64 \
>     -machine virtualization=true \
>     -cpu cortex-a57 -machine type=virt \
> -   -m 1024 -display none \
> -   -machine dumpdtb=binaries/virt-gicv3.dtb
> +   -m 2048 -smp 2 -display none \
> +   -machine dumpdtb=binaries/virt-gicv2.dtb
> +
>  # XXX disable pl061 to avoid Linux crash
> -dtc -I dtb -O dts binaries/virt-gicv3.dtb > binaries/virt-gicv3.dts
> -sed 's/compatible = "arm,pl061.*/status = "disabled";/g' binaries/virt-gicv3.dts > binaries/virt-gicv3-edited.dts
> -dtc -I dts -O dtb binaries/virt-gicv3-edited.dts > binaries/virt-gicv3.dtb
> +fdtput binaries/virt-gicv2.dtb -p -t s /pl061@9030000 status disabled
>  
>  # ImageBuilder
>  echo 'MEMORY_START="0x40000000"
> -MEMORY_END="0x80000000"
> +MEMORY_END="0xC0000000"
>  
> -DEVICE_TREE="virt-gicv3.dtb"
> +DEVICE_TREE="virt-gicv2.dtb"
>  XEN="xen"
>  DOM0_KERNEL="Image"
>  DOM0_RAMDISK="xen-rootfs.cpio.gz"
> -- 
> 2.25.1
> 

Re: [PATCH 7/9] automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh

[Stefano Stabellini]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> Testing arm64 is done using the qemu-alpine-arm64.sh and
> qemu-smoke-arm64.sh scripts. These scripts are executed with exactly
> the same artifacts (container, rootfs, kernel, qemu) and the only
> difference is that the former is used to perform dom0 based testing
> and the latter - dom0less based testing.
> 
> Because the current naming is quite umbiguous, rename qemu-smoke-arm64.sh
> script to qemu-smoke-dom0less-arm64.sh to reflect its usage.
> 
> qemu-alpine-arm64.sh will be renamed in the follow-up patch.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/gitlab-ci/test.yaml                | 24 +++++++++----------
>  ...-arm64.sh => qemu-smoke-dom0less-arm64.sh} |  0
>  2 files changed, 12 insertions(+), 12 deletions(-)
>  rename automation/scripts/{qemu-smoke-arm64.sh => qemu-smoke-dom0less-arm64.sh} (100%)
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 1b51030c6175..f620622671f8 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -81,12 +81,12 @@ qemu-alpine-x86_64-gcc:
>    tags:
>      - x86_64
>  
> -qemu-smoke-arm64-gcc:
> +qemu-smoke-dom0less-arm64-gcc:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
>    needs:
>      - alpine-3.12-gcc-arm64
>      - alpine-3.12-arm64-rootfs-export
> @@ -100,12 +100,12 @@ qemu-smoke-arm64-gcc:
>    tags:
>      - arm64
>  
> -qemu-smoke-arm64-gcc-debug:
> +qemu-smoke-dom0less-arm64-gcc-debug:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
>    needs:
>      - alpine-3.12-gcc-debug-arm64
>      - alpine-3.12-arm64-rootfs-export
> @@ -119,12 +119,12 @@ qemu-smoke-arm64-gcc-debug:
>    tags:
>      - arm64
>  
> -qemu-smoke-arm64-gcc-staticmem:
> +qemu-smoke-dom0less-arm64-gcc-staticmem:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
>    needs:
>      - alpine-3.12-gcc-arm64-staticmem
>      - alpine-3.12-arm64-rootfs-export
> @@ -138,12 +138,12 @@ qemu-smoke-arm64-gcc-staticmem:
>    tags:
>      - arm64
>  
> -qemu-smoke-arm64-gcc-debug-staticmem:
> +qemu-smoke-dom0less-arm64-gcc-debug-staticmem:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
>    needs:
>      - alpine-3.12-gcc-debug-arm64-staticmem
>      - alpine-3.12-arm64-rootfs-export
> @@ -157,12 +157,12 @@ qemu-smoke-arm64-gcc-debug-staticmem:
>    tags:
>      - arm64
>  
> -qemu-smoke-arm64-gcc-boot-cpupools:
> +qemu-smoke-dom0less-arm64-gcc-boot-cpupools:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
>    needs:
>      - alpine-3.12-gcc-arm64-boot-cpupools
>      - alpine-3.12-arm64-rootfs-export
> @@ -176,12 +176,12 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>    tags:
>      - arm64
>  
> -qemu-smoke-arm64-gcc-debug-boot-cpupools:
> +qemu-smoke-dom0less-arm64-gcc-debug-boot-cpupools:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
>    needs:
>      - alpine-3.12-gcc-debug-arm64-boot-cpupools
>      - alpine-3.12-arm64-rootfs-export
> diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
> similarity index 100%
> rename from automation/scripts/qemu-smoke-arm64.sh
> rename to automation/scripts/qemu-smoke-dom0less-arm64.sh
> -- 
> 2.25.1
> 

Re: [PATCH 8/9] automation: Rename qemu-alpine-arm64.sh to qemu-smoke-dom0-arm64.sh

[Stefano Stabellini]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> qemu-alpine-arm64.sh script is used to perform dom0 based testing.
> Rename this script to qemu-smoke-dom0-arm64.sh to reflect its usage.
> Also rename the corresponding test jobs.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/gitlab-ci/test.yaml                            | 8 ++++----
>  .../{qemu-alpine-arm64.sh => qemu-smoke-dom0-arm64.sh}    | 0
>  2 files changed, 4 insertions(+), 4 deletions(-)
>  rename automation/scripts/{qemu-alpine-arm64.sh => qemu-smoke-dom0-arm64.sh} (100%)
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index f620622671f8..3b147c88ab08 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -25,12 +25,12 @@ build-each-commit-gcc:
>    tags:
>      - x86_64
>  
> -qemu-alpine-arm64-gcc:
> +qemu-smoke-dom0-arm64-gcc:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
>    needs:
>      - alpine-3.12-gcc-arm64
>      - alpine-3.12-arm64-rootfs-export
> @@ -44,12 +44,12 @@ qemu-alpine-arm64-gcc:
>    tags:
>      - arm64
>  
> -qemu-alpine-arm64-gcc-debug:
> +qemu-smoke-dom0-arm64-gcc-debug:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
>    needs:
>      - alpine-3.12-gcc-debug-arm64
>      - alpine-3.12-arm64-rootfs-export
> diff --git a/automation/scripts/qemu-alpine-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
> similarity index 100%
> rename from automation/scripts/qemu-alpine-arm64.sh
> rename to automation/scripts/qemu-smoke-dom0-arm64.sh
> -- 
> 2.25.1
> 

Re: [PATCH 9/9] automation: Rename qemu-smoke-arm32.sh to qemu-smoke-dom0-arm32.sh

[Stefano Stabellini]

On Thu, 22 Sep 2022, Michal Orzel wrote:
> After qemu arm64 test scripts had been renamed to reflect their
> usage, do the same for the qemu arm32 test script. Currently it only
> boots dom0, so we can assume that this script will be used to perform
> dom0 based testing. In the future we will be able to create corresponding
> script qemu-smoke-dom0less-arm32.sh to perform dom0less based testing.
> This is the last step to get rid of ambiguity with regards to naming
> of Arm test scripts.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/gitlab-ci/test.yaml                            | 8 ++++----
>  .../{qemu-smoke-arm32.sh => qemu-smoke-dom0-arm32.sh}     | 0
>  2 files changed, 4 insertions(+), 4 deletions(-)
>  rename automation/scripts/{qemu-smoke-arm32.sh => qemu-smoke-dom0-arm32.sh} (100%)
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 3b147c88ab08..92e0a1f7c510 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -195,12 +195,12 @@ qemu-smoke-dom0less-arm64-gcc-debug-boot-cpupools:
>    tags:
>      - arm64
>  
> -qemu-smoke-arm32-gcc:
> +qemu-smoke-dom0-arm32-gcc:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
> +    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
>    needs:
>      - debian-unstable-gcc-arm32
>      - qemu-system-aarch64-6.0.0-arm32-export
> @@ -212,12 +212,12 @@ qemu-smoke-arm32-gcc:
>    tags:
>      - arm64
>  
> -qemu-smoke-arm32-gcc-debug:
> +qemu-smoke-dom0-arm32-gcc-debug:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
>    script:
> -    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
> +    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
>    needs:
>      - debian-unstable-gcc-arm32-debug
>      - qemu-system-aarch64-6.0.0-arm32-export
> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-dom0-arm32.sh
> similarity index 100%
> rename from automation/scripts/qemu-smoke-arm32.sh
> rename to automation/scripts/qemu-smoke-dom0-arm32.sh
> -- 
> 2.25.1
> 

RE: [PATCH 1/9] automation: Use custom build jobs when extra config options are needed

[Henry Wang]

Hi Stefano,

> -----Original Message-----
> From: Stefano Stabellini <sstabellini@kernel.org>
> Subject: Re: [PATCH 1/9] automation: Use custom build jobs when extra
> config options are needed
> 
> On Thu, 22 Sep 2022, Michal Orzel wrote:
> > Currently, all the arm64 defconfig build jobs, regardless of the
> > container used, end up building Xen with the extra config options
> > specified in the main build script (e.g. CONFIG_EXPERT,
> > CONFIG_STATIC_MEMORY). Because these options are only needed for
> > specific test jobs, the current behavior of the CI is incorrect
> > as we add the extra options to all the defconfig builds. This means
> > that on arm64 there is not a single job performing proper defconfig build.
> >
> > To fix this issue, add custom build jobs each time there is a need for
> > building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
> > variable to be used by these jobs to store the required options. This
> > variable will be then read by the main build script to modify the .config
> > file. This will also help users to understand what is needed to run specific
> > test.
> >
> > Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> > ---
> > This patch could actually be consider to be taken for 4.17 release.
> > The reason why is because currently our CI for arm64 does not even
> > peform clean defconfig build which is quite crucial target to be tested.
> > Performing builds always with EXPERT and UNSUPPORTED is not something
> so
> > beneficial for release tests. This is up to the release manager.
> 
> + Henry
> 
> I agree this should go in 4.17, so that gitlab-ci can test non-DEBUG
> builds on ARM.

Yes sure, I agree once this series is properly reviewed, this can be
merged to 4.17.

Kind regards,
Henry

Re: [PATCH 1/9] automation: Use custom build jobs when extra config options are needed

[Michal Orzel]

Hi Stefano,

On 22/09/2022 23:40, Stefano Stabellini wrote:
> 
> 
> On Thu, 22 Sep 2022, Michal Orzel wrote:
>> Currently, all the arm64 defconfig build jobs, regardless of the
>> container used, end up building Xen with the extra config options
>> specified in the main build script (e.g. CONFIG_EXPERT,
>> CONFIG_STATIC_MEMORY). Because these options are only needed for
>> specific test jobs, the current behavior of the CI is incorrect
>> as we add the extra options to all the defconfig builds. This means
>> that on arm64 there is not a single job performing proper defconfig build.
>>
>> To fix this issue, add custom build jobs each time there is a need for
>> building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
>> variable to be used by these jobs to store the required options. This
>> variable will be then read by the main build script to modify the .config
>> file. This will also help users to understand what is needed to run specific
>> test.
>>
>> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
>> ---
>> This patch could actually be consider to be taken for 4.17 release.
>> The reason why is because currently our CI for arm64 does not even
>> peform clean defconfig build which is quite crucial target to be tested.
>> Performing builds always with EXPERT and UNSUPPORTED is not something so
>> beneficial for release tests. This is up to the release manager.
> 
> + Henry
> 
> I agree this should go in 4.17, so that gitlab-ci can test non-DEBUG
> builds on ARM.
> 
Do you mean the whole series should go in?
I'm ok with that, even though I only marked this patch as the one that should go in
as it can be seen as a fix. But I can also see the benefits of merging the whole series.

> 
>> ---
>>  automation/gitlab-ci/build.yaml | 15 +++++++++++++++
>>  automation/gitlab-ci/test.yaml  |  4 ++--
>>  automation/scripts/build        |  8 ++------
>>  3 files changed, 19 insertions(+), 8 deletions(-)
>>
>> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
>> index 720ce6e07ba0..a39ed72aac6d 100644
>> --- a/automation/gitlab-ci/build.yaml
>> +++ b/automation/gitlab-ci/build.yaml
>> @@ -566,6 +566,21 @@ alpine-3.12-gcc-debug-arm64:
>>    variables:
>>      CONTAINER: alpine:3.12-arm64v8
>>
>> +alpine-3.12-gcc-arm64-staticmem:
>> +  extends: .gcc-arm64-build
>> +  variables:
>> +    CONTAINER: alpine:3.12-arm64v8
>> +    EXTRA_XEN_CONFIG: |
> 
> Why the "|" ?
> 
> I was trying to look for its documentation in the gitlab yaml docs but
> couldn't find it.
> 
By default gitlab variables are one liners so that they can store one key:value pair.
If you want to define a variable storing multiple values (in this case we want to
store multi-line string because of .config format) you need to use |. You can check [1].

> 
>> +      CONFIG_EXPERT=y
>> +      CONFIG_UNSUPPORTED=y
>> +      CONFIG_STATIC_MEMORY=y
>> +
>> +alpine-3.12-gcc-arm64-boot-cpupools:
>> +  extends: .gcc-arm64-build
>> +  variables:
>> +    CONTAINER: alpine:3.12-arm64v8
>> +    EXTRA_XEN_CONFIG: |
>> +      CONFIG_BOOT_TIME_CPUPOOLS=y
>>
>>  ## Test artifacts common
>>
>> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
>> index d899b3bdbf7a..4f96e6e322de 100644
>> --- a/automation/gitlab-ci/test.yaml
>> +++ b/automation/gitlab-ci/test.yaml
>> @@ -88,7 +88,7 @@ qemu-smoke-arm64-gcc-staticmem:
>>    script:
>>      - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
>>    needs:
>> -    - alpine-3.12-gcc-arm64
>> +    - alpine-3.12-gcc-arm64-staticmem
>>      - alpine-3.12-arm64-rootfs-export
>>      - kernel-5.19-arm64-export
>>      - qemu-system-aarch64-6.0.0-arm64-export
>> @@ -107,7 +107,7 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>>    script:
>>      - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
>>    needs:
>> -    - alpine-3.12-gcc-arm64
>> +    - alpine-3.12-gcc-arm64-boot-cpupools
>>      - alpine-3.12-arm64-rootfs-export
>>      - kernel-5.19-arm64-export
>>      - qemu-system-aarch64-6.0.0-arm64-export
>> diff --git a/automation/scripts/build b/automation/scripts/build
>> index 2f15ab3198e6..7d441cedb4ae 100755
>> --- a/automation/scripts/build
>> +++ b/automation/scripts/build
>> @@ -15,12 +15,8 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
>>      make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
>>      hypervisor_only="y"
>>  else
>> -    if [[ "${XEN_TARGET_ARCH}" = "arm64" ]]; then
>> -        echo "
>> -CONFIG_EXPERT=y
>> -CONFIG_UNSUPPORTED=y
>> -CONFIG_STATIC_MEMORY=y
>> -CONFIG_BOOT_TIME_CPUPOOLS=y" > xen/.config
>> +    if [ -n "${EXTRA_XEN_CONFIG}" ]; then
> 
> NIT: for uniformity with rest of the file use
> 
>   if [[ -n "${EXTRA_XEN_CONFIG}" ]]; then
> 
Ok, will do in v2.

> 
>> +        echo "${EXTRA_XEN_CONFIG}" > xen/.config
>>          make -j$(nproc) -C xen olddefconfig
>>      else
>>          make -j$(nproc) -C xen defconfig
>> --
>> 2.25.1
>>

~Michal

[1] https://docs.gitlab.com/ee/ci/variables/#store-multiple-values-in-one-variable

Re: [PATCH 2/9] automation: Add randconfig build jobs for arm64 alpine container

[Michal Orzel]

Hi Stefano,

On 22/09/2022 23:42, Stefano Stabellini wrote:
> 
> 
> On Thu, 22 Sep 2022, Michal Orzel wrote:
>> For arm64 we perform builds using debian and alpine containers.
>> We are missing the randconfig build jobs for the latter, so add them.
>> This way for each container we have 4 fundamental build jobs:
>> - defconfig non-debug/debug
>> - randconfig non-debug/debug
> 
> Should we have them also for Debian? The patch below looks fine.
> 

We already have them for Debian. We were missing only the ones for Alpine.
That is why I wrote that after this patch for each container we have 4 fundamental build jobs:
DEBIAN:
 debian-unstable-gcc-arm64
 debian-unstable-gcc-debug-arm64
 debian-unstable-gcc-arm64-randconfig
 debian-unstable-gcc-debug-arm64-randconfig
ALPINE:
 alpine-3.12-gcc-arm64
 alpine-3.12-gcc-debug-arm64
 alpine-3.12-gcc-arm64-randconfig
 alpine-3.12-gcc-debug-arm64-randconfig

> 
>> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
>> ---
>>  automation/gitlab-ci/build.yaml | 12 ++++++++++++
>>  1 file changed, 12 insertions(+)
>>
>> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
>> index a39ed72aac6d..d931441b910a 100644
>> --- a/automation/gitlab-ci/build.yaml
>> +++ b/automation/gitlab-ci/build.yaml
>> @@ -566,6 +566,18 @@ alpine-3.12-gcc-debug-arm64:
>>    variables:
>>      CONTAINER: alpine:3.12-arm64v8
>>
>> +alpine-3.12-gcc-arm64-randconfig:
>> +  extends: .gcc-arm64-build
>> +  variables:
>> +    CONTAINER: alpine:3.12-arm64v8
>> +    RANDCONFIG: y
>> +
>> +alpine-3.12-gcc-debug-arm64-randconfig:
>> +  extends: .gcc-arm64-build-debug
>> +  variables:
>> +    CONTAINER: alpine:3.12-arm64v8
>> +    RANDCONFIG: y
>> +
>>  alpine-3.12-gcc-arm64-staticmem:
>>    extends: .gcc-arm64-build
>>    variables:
>> --
>> 2.25.1
>>

~Michal

Re: [PATCH 3/9] automation: Add debug versions of Arm tests

[Michal Orzel]

Hi Stefano,

On 22/09/2022 23:53, Stefano Stabellini wrote:
> 
> 
> On Thu, 22 Sep 2022, Michal Orzel wrote:
>> At the moment, all the tests are executed on non-debug Xen builds.
>> To improve the coverage (e.g. we might catch some asserts), add new
>> test jobs using debug Xen builds.
> 
> This is interesting. On one hand, I agree it is good to test DEBUG and
> non-DEBUG builds in runtime tests too, on the other hand I am worried
> about our test capacity.

We still have plenty of capacity. We can add many more Arm tests because
Arm machine is very powerful. I reckon that adding more x86 tests would
have a performance impact but not the other way around. See below.

> 
> I take you have tried a few pipelines -- are the additional tests
> causing the pipeline to take longer? Do we still have "room" because x86
> is slower? What's your thinking?
> 

With the additional tests introduced by this patch one cannot observe any
impact for the performance. I ran several tests and the pipeline finishes
in the same time or faster.

Performance comparison:
current master: 92 jobs (82 minutes, 19 seconds)
master + my series: 103 jobs (79 minutes, 40 seconds)

That said, let's improve the coverage and add these non-debug tests for Arm.

> One idea would be to only do:
> - qemu-alpine-arm64-gcc-debug
> - qemu-smoke-arm64-gcc-debug
> 
> and not the others
> 
> 
>> Take the opportunity to increase RAM size for QEMU from 1GB to 2GB
>> in qemu-smoke-arm64.sh as the debug builds take more space and we might
>> end up in a situation when there is not enough free space (especially
>> during a static memory test that reserves some region in the middle).
>>
>> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
>> ---
>>  automation/gitlab-ci/build.yaml        | 16 +++++
>>  automation/gitlab-ci/test.yaml         | 93 ++++++++++++++++++++++++++
>>  automation/scripts/qemu-smoke-arm64.sh |  6 +-
>>  3 files changed, 112 insertions(+), 3 deletions(-)
>>
>> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
>> index d931441b910a..6ce92531f61f 100644
>> --- a/automation/gitlab-ci/build.yaml
>> +++ b/automation/gitlab-ci/build.yaml
>> @@ -587,6 +587,15 @@ alpine-3.12-gcc-arm64-staticmem:
>>        CONFIG_UNSUPPORTED=y
>>        CONFIG_STATIC_MEMORY=y
>>
>> +alpine-3.12-gcc-debug-arm64-staticmem:
>> +  extends: .gcc-arm64-build-debug
>> +  variables:
>> +    CONTAINER: alpine:3.12-arm64v8
>> +    EXTRA_XEN_CONFIG: |
>> +      CONFIG_EXPERT=y
>> +      CONFIG_UNSUPPORTED=y
>> +      CONFIG_STATIC_MEMORY=y
>> +
>>  alpine-3.12-gcc-arm64-boot-cpupools:
>>    extends: .gcc-arm64-build
>>    variables:
>> @@ -594,6 +603,13 @@ alpine-3.12-gcc-arm64-boot-cpupools:
>>      EXTRA_XEN_CONFIG: |
>>        CONFIG_BOOT_TIME_CPUPOOLS=y
>>
>> +alpine-3.12-gcc-debug-arm64-boot-cpupools:
>> +  extends: .gcc-arm64-build-debug
>> +  variables:
>> +    CONTAINER: alpine:3.12-arm64v8
>> +    EXTRA_XEN_CONFIG: |
>> +      CONFIG_BOOT_TIME_CPUPOOLS=y
>> +
>>  ## Test artifacts common
>>
>>  .test-jobs-artifact-common:
>> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
>> index 4f96e6e322de..1b51030c6175 100644
>> --- a/automation/gitlab-ci/test.yaml
>> +++ b/automation/gitlab-ci/test.yaml
>> @@ -44,6 +44,25 @@ qemu-alpine-arm64-gcc:
>>    tags:
>>      - arm64
>>
>> +qemu-alpine-arm64-gcc-debug:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
>> +  needs:
>> +    - alpine-3.12-gcc-debug-arm64
>> +    - alpine-3.12-arm64-rootfs-export
>> +    - kernel-5.19-arm64-export
>> +    - qemu-system-aarch64-6.0.0-arm64-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-alpine-x86_64-gcc:
>>    extends: .test-jobs-common
>>    variables:
>> @@ -81,6 +100,25 @@ qemu-smoke-arm64-gcc:
>>    tags:
>>      - arm64
>>
>> +qemu-smoke-arm64-gcc-debug:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
>> +  needs:
>> +    - alpine-3.12-gcc-debug-arm64
>> +    - alpine-3.12-arm64-rootfs-export
>> +    - kernel-5.19-arm64-export
>> +    - qemu-system-aarch64-6.0.0-arm64-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-smoke-arm64-gcc-staticmem:
>>    extends: .test-jobs-common
>>    variables:
>> @@ -100,6 +138,25 @@ qemu-smoke-arm64-gcc-staticmem:
>>    tags:
>>      - arm64
>>
>> +qemu-smoke-arm64-gcc-debug-staticmem:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
>> +  needs:
>> +    - alpine-3.12-gcc-debug-arm64-staticmem
>> +    - alpine-3.12-arm64-rootfs-export
>> +    - kernel-5.19-arm64-export
>> +    - qemu-system-aarch64-6.0.0-arm64-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-smoke-arm64-gcc-boot-cpupools:
>>    extends: .test-jobs-common
>>    variables:
>> @@ -119,6 +176,25 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>>    tags:
>>      - arm64
>>
>> +qemu-smoke-arm64-gcc-debug-boot-cpupools:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
>> +  needs:
>> +    - alpine-3.12-gcc-debug-arm64-boot-cpupools
>> +    - alpine-3.12-arm64-rootfs-export
>> +    - kernel-5.19-arm64-export
>> +    - qemu-system-aarch64-6.0.0-arm64-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-smoke-arm32-gcc:
>>    extends: .test-jobs-common
>>    variables:
>> @@ -136,6 +212,23 @@ qemu-smoke-arm32-gcc:
>>    tags:
>>      - arm64
>>
>> +qemu-smoke-arm32-gcc-debug:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
>> +  needs:
>> +    - debian-unstable-gcc-arm32-debug
>> +    - qemu-system-aarch64-6.0.0-arm32-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-smoke-x86-64-gcc:
>>    extends: .test-jobs-common
>>    variables:
>> diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-arm64.sh
>> index f803835779f4..dea26c6c0a8f 100755
>> --- a/automation/scripts/qemu-smoke-arm64.sh
>> +++ b/automation/scripts/qemu-smoke-arm64.sh
>> @@ -52,7 +52,7 @@ curl -fsSLO https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fraw%2Fv5.2.0%2Fpc-bios%2Fefi-virtio.rom&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cf6b0b8fabb02478b009c08da9ce4eac9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994804254367719%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=jgGdO93kLoieg6UqajFWYAsufvvv5mZVYHD%2B7fUW7ME%3D&amp;reserved=0
>>  ./binaries/qemu-system-aarch64 \
>>     -machine virtualization=true \
>>     -cpu cortex-a57 -machine type=virt \
>> -   -m 1024 -smp 2 -display none \
>> +   -m 2048 -smp 2 -display none \
>>     -machine dumpdtb=binaries/virt-gicv2.dtb
>>
>>  # XXX disable pl061 to avoid Linux crash
>> @@ -117,7 +117,7 @@ cd ..
>>
>>  # ImageBuilder
>>  echo 'MEMORY_START="0x40000000"
>> -MEMORY_END="0x80000000"
>> +MEMORY_END="0xC0000000"
>>
>>  DEVICE_TREE="virt-gicv2.dtb"
>>  XEN="xen"
>> @@ -158,7 +158,7 @@ timeout -k 1 240 \
>>  ./binaries/qemu-system-aarch64 \
>>      -machine virtualization=true \
>>      -cpu cortex-a57 -machine type=virt \
>> -    -m 1024 -monitor none -serial stdio \
>> +    -m 2048 -monitor none -serial stdio \
>>      -smp 2 \
>>      -no-reboot \
>>      -device virtio-net-pci,netdev=n0 \
> 
> The changes to qemu-smoke-arm64.sh make sense, but could you move them
> to a separate patch in case we need to do any backports?

Ok, I will do a prerequisite patch for that one.

> 
> Also I would change MEMORY_END to 0xC0000000 in qemu-alpine-arm64.sh too
> for uniformity.

I already did that in patch [6/9] automation: qemu-alpine-arm64: Cleanup and fixes.

~Michal

Re: [PATCH 5/9] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder

[Michal Orzel]

Hi Stefano,

On 23/09/2022 00:13, Stefano Stabellini wrote:
> 
> 
> On Thu, 22 Sep 2022, Michal Orzel wrote:
>> Take an example from arm64 qemu test scripts and use ImageBuilder
>> to generate u-boot script automatically. Calculating the addresses
>> manually is quite error prone and also we will be able to benefit
>> from using ImageBuilder when adding domUs to this test in the future.
>>
>> Install and use u-boot from the debian package.
>> Modify the script so that binaries are loaded from u-boot via tftp.
> 
> Great patch! It makes the test a lot better!
> 
> 
>> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
>> ---
>>  automation/scripts/qemu-smoke-arm32.sh | 57 ++++++++++++--------------
>>  1 file changed, 27 insertions(+), 30 deletions(-)
>>
>> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
>> index 530f3892fdd3..765facbe4d66 100755
>> --- a/automation/scripts/qemu-smoke-arm32.sh
>> +++ b/automation/scripts/qemu-smoke-arm32.sh
>> @@ -4,7 +4,9 @@ set -ex
>>
>>  export DEBIAN_FRONTENT=noninteractive
>>  apt-get -qy update
>> -apt-get -qy install --no-install-recommends device-tree-compiler \
>> +apt-get -qy install --no-install-recommends u-boot-qemu \
>> +                                            u-boot-tools \
>> +                                            device-tree-compiler \
>>                                              curl \
>>                                              cpio
>>
>> @@ -20,10 +22,6 @@ tar xvzf ../initrd.tar.gz
>>  find . | cpio -H newc -o | gzip > ../initrd.gz
>>  cd ..
>>
>> -kernel=`stat -L --printf="%s" vmlinuz`
>> -initrd=`stat -L --printf="%s" initrd.gz`
>> -
>> -# For Xen, we need a couple of more node. Dump the DT from QEMU and add them
>>  # XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
>>  curl -fsSLO https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fraw%2Fv5.2.0%2Fpc-bios%2Fefi-virtio.rom&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd33d5c4c08934fac0cc208da9ce7a3fb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994815957091528%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=JuEapfYVLj5P3S5yY%2BAa47Nk4zgbdymjjsiUTmoTmyk%3D&amp;reserved=0
>>  ./qemu-system-arm \
>> @@ -36,31 +34,31 @@ curl -fsSLO https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fraw%2Fv5.2.0%2Fpc-bios%2Fefi-virtio.rom&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd33d5c4c08934fac0cc208da9ce7a3fb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994815957091528%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=JuEapfYVLj5P3S5yY%2BAa47Nk4zgbdymjjsiUTmoTmyk%3D&amp;reserved=0
>>     -display none \
>>     -machine dumpdtb=virt.dtb
>>
>> -dtc -I dtb -O dts virt.dtb > virt.dts
>> +# ImageBuilder
>> +echo 'MEMORY_START="0x40000000"
>> +MEMORY_END="0x80000000"
>> +
>> +DEVICE_TREE="virt.dtb"
>> +XEN="xen"
>> +DOM0_KERNEL="vmlinuz"
>> +DOM0_RAMDISK="initrd.gz"
>> +DOM0_CMD="console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh"
>> +XEN_CMD="console=dtuart dom0_mem=512M bootscrub=0"
> 
> This is missing dtuart=/pl011@9000000 compared to the original

Actually this is not needed because we always have stdout-path defined in chosen to point to /pl011@...
but I can add this.

> 
> 
>> +NUM_DOMUS=0
>> +
>> +LOAD_CMD="tftpb"
>> +BOOT_CMD="bootm"
> 
> "bootm" because "booti" is not available on arm32, right?

Exactly.
> 
> 
>> +UBOOT_SOURCE="boot.source"
>> +UBOOT_SCRIPT="boot.scr"' > config
>>
>> -cat >> virt.dts << EOF
>> -/ {
>> -     chosen {
>> -             #address-cells = <0x2>;
>> -             #size-cells = <0x2>;
>> -             stdout-path = "/pl011@9000000";
>> -        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=512M bootscrub=0";
>> -             xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh";
>> -             dom0 {
>> -                     compatible = "xen,linux-zimage", "xen,multiboot-module";
>> -                     reg = <0x0 0x1000000 0x0 $kernel>;
>> -             };
>> -        dom0-ramdisk {
>> -                     compatible = "xen,linux-initrd", "xen,multiboot-module";
>> -                     reg = <0x0 0x3200000 0x0 $initrd>;
>> -             };
>> -     };
>> -};
>> -EOF
>> -dtc -I dts -O dtb virt.dts > virt.dtb
>> +rm -rf imagebuilder
>> +git clone https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2FViryaOS%2Fimagebuilder&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd33d5c4c08934fac0cc208da9ce7a3fb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994815957091528%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=i5N8DXMjn%2F80mdXtc%2FwDGJw6ImUrUAjOg0SJp9CkRdQ%3D&amp;reserved=0
>> +bash imagebuilder/scripts/uboot-script-gen -t tftp -d . -c config
>>
>>  rm -f smoke.serial
>>  set +e
>> +echo "  virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x40000000"| \
>>  timeout -k 1 240 \
>>  ./qemu-system-arm \
>>     -machine virt \
>> @@ -70,11 +68,10 @@ timeout -k 1 240 \
>>     -serial stdio \
>>     -monitor none \
>>     -display none \
>> -   -dtb virt.dtb \
>>     -no-reboot \
>> -   -kernel ./xen \
>> -   -device loader,file=./vmlinuz,addr=0x1000000 \
>> -   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
>> +   -device virtio-net-pci,netdev=n0 \
>> +   -netdev user,id=n0,tftp=./ \
>> +   -bios /usr/lib/u-boot/qemu_arm/u-boot.bin |& tee smoke.serial
>>
>>  set -e
>>  (grep -q "^/ #" smoke.serial) || exit 1
>> --
>> 2.25.1
>>

~Michal

Re: [PATCH 4/9] automation: Add Arm containers to containerize script

[Anthony PERARD]

On Thu, Sep 22, 2022 at 03:40:53PM +0200, Michal Orzel wrote:
> Script automation/scripts/containerize makes it easy to build Xen within
> predefined containers from gitlab container registry. However, it is
> currently not possible to use it with Arm containers because they are not
> listed in the script. Populate the necessary entries.

FYI, those entry are just helper/shortcut/aliases, you can use any
arbitrary container with the script, it just more annoying.

Your patch here allows to write:
    CONTAINER=unstable-arm64v8 automation/scripts/containerize
but you could write the following instead, for the same result:
    CONTAINER=registry.gitlab.com/xen-project/xen/debian:unstable-arm64v8 automation/scripts/containerize

I wonder if the script could select the right container base on the
architecture of the host, because "alpine" and "alpine-arm64v8" will not
both work on the same machine. It might be nice to just choose "alpine"
and the script would select the x86 or arm container automagically. Just
an idea, no need to do anything about it.

(patch is fine otherwise, just the patch description is a bit
misleading)

Cheers,

-- 
Anthony PERARD

Re: [PATCH 4/9] automation: Add Arm containers to containerize script

[Michal Orzel]

Hi Anthony,

On 23/09/2022 15:56, Anthony PERARD wrote:
> 
> 
> On Thu, Sep 22, 2022 at 03:40:53PM +0200, Michal Orzel wrote:
>> Script automation/scripts/containerize makes it easy to build Xen within
>> predefined containers from gitlab container registry. However, it is
>> currently not possible to use it with Arm containers because they are not
>> listed in the script. Populate the necessary entries.
> 
> FYI, those entry are just helper/shortcut/aliases, you can use any
> arbitrary container with the script, it just more annoying.
> 
> Your patch here allows to write:
>     CONTAINER=unstable-arm64v8 automation/scripts/containerize
> but you could write the following instead, for the same result:
>     CONTAINER=registry.gitlab.com/xen-project/xen/debian:unstable-arm64v8 automation/scripts/containerize
> 
> I wonder if the script could select the right container base on the
> architecture of the host, because "alpine" and "alpine-arm64v8" will not
> both work on the same machine. It might be nice to just choose "alpine"
> and the script would select the x86 or arm container automagically. Just
> an idea, no need to do anything about it.
I'm not in favor of adding the automatic selection based on the host.
The reason is that on x86 you can run both x86 and e.g. Arm containers.
You just need to use register qemu-user-static [1] to perform emulation.
This is something widely used and I use that to test Arm images/containers on x86 host.
So you can run both alpine and alpine-arm64v8 on the same machine.

> 
> (patch is fine otherwise, just the patch description is a bit
> misleading)
> 
> Cheers,
> 
> --
> Anthony PERARD

~Michal

[1] https://github.com/multiarch/qemu-user-static

Re: [PATCH 1/9] automation: Use custom build jobs when extra config options are needed

[Stefano Stabellini]

On Fri, 23 Sep 2022, Michal Orzel wrote:
> Hi Stefano,
> 
> On 22/09/2022 23:40, Stefano Stabellini wrote:
> > 
> > 
> > On Thu, 22 Sep 2022, Michal Orzel wrote:
> >> Currently, all the arm64 defconfig build jobs, regardless of the
> >> container used, end up building Xen with the extra config options
> >> specified in the main build script (e.g. CONFIG_EXPERT,
> >> CONFIG_STATIC_MEMORY). Because these options are only needed for
> >> specific test jobs, the current behavior of the CI is incorrect
> >> as we add the extra options to all the defconfig builds. This means
> >> that on arm64 there is not a single job performing proper defconfig build.
> >>
> >> To fix this issue, add custom build jobs each time there is a need for
> >> building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
> >> variable to be used by these jobs to store the required options. This
> >> variable will be then read by the main build script to modify the .config
> >> file. This will also help users to understand what is needed to run specific
> >> test.
> >>
> >> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> >> ---
> >> This patch could actually be consider to be taken for 4.17 release.
> >> The reason why is because currently our CI for arm64 does not even
> >> peform clean defconfig build which is quite crucial target to be tested.
> >> Performing builds always with EXPERT and UNSUPPORTED is not something so
> >> beneficial for release tests. This is up to the release manager.
> > 
> > + Henry
> > 
> > I agree this should go in 4.17, so that gitlab-ci can test non-DEBUG
> > builds on ARM.
> > 
> Do you mean the whole series should go in?
> I'm ok with that, even though I only marked this patch as the one that should go in
> as it can be seen as a fix. But I can also see the benefits of merging the whole series.

I think only this patch should go in. I like this series but it is best
to stay on the safe side and push to staging the rest of the series
later after the release.

 
> >> ---
> >>  automation/gitlab-ci/build.yaml | 15 +++++++++++++++
> >>  automation/gitlab-ci/test.yaml  |  4 ++--
> >>  automation/scripts/build        |  8 ++------
> >>  3 files changed, 19 insertions(+), 8 deletions(-)
> >>
> >> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> >> index 720ce6e07ba0..a39ed72aac6d 100644
> >> --- a/automation/gitlab-ci/build.yaml
> >> +++ b/automation/gitlab-ci/build.yaml
> >> @@ -566,6 +566,21 @@ alpine-3.12-gcc-debug-arm64:
> >>    variables:
> >>      CONTAINER: alpine:3.12-arm64v8
> >>
> >> +alpine-3.12-gcc-arm64-staticmem:
> >> +  extends: .gcc-arm64-build
> >> +  variables:
> >> +    CONTAINER: alpine:3.12-arm64v8
> >> +    EXTRA_XEN_CONFIG: |
> > 
> > Why the "|" ?
> > 
> > I was trying to look for its documentation in the gitlab yaml docs but
> > couldn't find it.
> > 
> By default gitlab variables are one liners so that they can store one key:value pair.
> If you want to define a variable storing multiple values (in this case we want to
> store multi-line string because of .config format) you need to use |. You can check [1].

OK


> > 
> >> +      CONFIG_EXPERT=y
> >> +      CONFIG_UNSUPPORTED=y
> >> +      CONFIG_STATIC_MEMORY=y
> >> +
> >> +alpine-3.12-gcc-arm64-boot-cpupools:
> >> +  extends: .gcc-arm64-build
> >> +  variables:
> >> +    CONTAINER: alpine:3.12-arm64v8
> >> +    EXTRA_XEN_CONFIG: |
> >> +      CONFIG_BOOT_TIME_CPUPOOLS=y
> >>
> >>  ## Test artifacts common
> >>
> >> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> >> index d899b3bdbf7a..4f96e6e322de 100644
> >> --- a/automation/gitlab-ci/test.yaml
> >> +++ b/automation/gitlab-ci/test.yaml
> >> @@ -88,7 +88,7 @@ qemu-smoke-arm64-gcc-staticmem:
> >>    script:
> >>      - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
> >>    needs:
> >> -    - alpine-3.12-gcc-arm64
> >> +    - alpine-3.12-gcc-arm64-staticmem
> >>      - alpine-3.12-arm64-rootfs-export
> >>      - kernel-5.19-arm64-export
> >>      - qemu-system-aarch64-6.0.0-arm64-export
> >> @@ -107,7 +107,7 @@ qemu-smoke-arm64-gcc-boot-cpupools:
> >>    script:
> >>      - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
> >>    needs:
> >> -    - alpine-3.12-gcc-arm64
> >> +    - alpine-3.12-gcc-arm64-boot-cpupools
> >>      - alpine-3.12-arm64-rootfs-export
> >>      - kernel-5.19-arm64-export
> >>      - qemu-system-aarch64-6.0.0-arm64-export
> >> diff --git a/automation/scripts/build b/automation/scripts/build
> >> index 2f15ab3198e6..7d441cedb4ae 100755
> >> --- a/automation/scripts/build
> >> +++ b/automation/scripts/build
> >> @@ -15,12 +15,8 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
> >>      make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
> >>      hypervisor_only="y"
> >>  else
> >> -    if [[ "${XEN_TARGET_ARCH}" = "arm64" ]]; then
> >> -        echo "
> >> -CONFIG_EXPERT=y
> >> -CONFIG_UNSUPPORTED=y
> >> -CONFIG_STATIC_MEMORY=y
> >> -CONFIG_BOOT_TIME_CPUPOOLS=y" > xen/.config
> >> +    if [ -n "${EXTRA_XEN_CONFIG}" ]; then
> > 
> > NIT: for uniformity with rest of the file use
> > 
> >   if [[ -n "${EXTRA_XEN_CONFIG}" ]]; then
> > 
> Ok, will do in v2.
> 
> > 
> >> +        echo "${EXTRA_XEN_CONFIG}" > xen/.config
> >>          make -j$(nproc) -C xen olddefconfig
> >>      else
> >>          make -j$(nproc) -C xen defconfig
> >> --
> >> 2.25.1
> >>
> 
> ~Michal
> 
> [1] https://docs.gitlab.com/ee/ci/variables/#store-multiple-values-in-one-variable
> 

Re: [PATCH 3/9] automation: Add debug versions of Arm tests

[Stefano Stabellini]

On Fri, 23 Sep 2022, Michal Orzel wrote:
> Hi Stefano,
> 
> On 22/09/2022 23:53, Stefano Stabellini wrote:
> > 
> > 
> > On Thu, 22 Sep 2022, Michal Orzel wrote:
> >> At the moment, all the tests are executed on non-debug Xen builds.
> >> To improve the coverage (e.g. we might catch some asserts), add new
> >> test jobs using debug Xen builds.
> > 
> > This is interesting. On one hand, I agree it is good to test DEBUG and
> > non-DEBUG builds in runtime tests too, on the other hand I am worried
> > about our test capacity.
> 
> We still have plenty of capacity. We can add many more Arm tests because
> Arm machine is very powerful. I reckon that adding more x86 tests would
> have a performance impact but not the other way around. See below.
> 
> > 
> > I take you have tried a few pipelines -- are the additional tests
> > causing the pipeline to take longer? Do we still have "room" because x86
> > is slower? What's your thinking?
> > 
> 
> With the additional tests introduced by this patch one cannot observe any
> impact for the performance. I ran several tests and the pipeline finishes
> in the same time or faster.
> 
> Performance comparison:
> current master: 92 jobs (82 minutes, 19 seconds)
> master + my series: 103 jobs (79 minutes, 40 seconds)
> 
> That said, let's improve the coverage and add these non-debug tests for Arm.

ahah WOW OK then


> > One idea would be to only do:
> > - qemu-alpine-arm64-gcc-debug
> > - qemu-smoke-arm64-gcc-debug
> > 
> > and not the others
> > 
> > 
> >> Take the opportunity to increase RAM size for QEMU from 1GB to 2GB
> >> in qemu-smoke-arm64.sh as the debug builds take more space and we might
> >> end up in a situation when there is not enough free space (especially
> >> during a static memory test that reserves some region in the middle).
> >>
> >> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> >> ---
> >>  automation/gitlab-ci/build.yaml        | 16 +++++
> >>  automation/gitlab-ci/test.yaml         | 93 ++++++++++++++++++++++++++
> >>  automation/scripts/qemu-smoke-arm64.sh |  6 +-
> >>  3 files changed, 112 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> >> index d931441b910a..6ce92531f61f 100644
> >> --- a/automation/gitlab-ci/build.yaml
> >> +++ b/automation/gitlab-ci/build.yaml
> >> @@ -587,6 +587,15 @@ alpine-3.12-gcc-arm64-staticmem:
> >>        CONFIG_UNSUPPORTED=y
> >>        CONFIG_STATIC_MEMORY=y
> >>
> >> +alpine-3.12-gcc-debug-arm64-staticmem:
> >> +  extends: .gcc-arm64-build-debug
> >> +  variables:
> >> +    CONTAINER: alpine:3.12-arm64v8
> >> +    EXTRA_XEN_CONFIG: |
> >> +      CONFIG_EXPERT=y
> >> +      CONFIG_UNSUPPORTED=y
> >> +      CONFIG_STATIC_MEMORY=y
> >> +
> >>  alpine-3.12-gcc-arm64-boot-cpupools:
> >>    extends: .gcc-arm64-build
> >>    variables:
> >> @@ -594,6 +603,13 @@ alpine-3.12-gcc-arm64-boot-cpupools:
> >>      EXTRA_XEN_CONFIG: |
> >>        CONFIG_BOOT_TIME_CPUPOOLS=y
> >>
> >> +alpine-3.12-gcc-debug-arm64-boot-cpupools:
> >> +  extends: .gcc-arm64-build-debug
> >> +  variables:
> >> +    CONTAINER: alpine:3.12-arm64v8
> >> +    EXTRA_XEN_CONFIG: |
> >> +      CONFIG_BOOT_TIME_CPUPOOLS=y
> >> +
> >>  ## Test artifacts common
> >>
> >>  .test-jobs-artifact-common:
> >> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> >> index 4f96e6e322de..1b51030c6175 100644
> >> --- a/automation/gitlab-ci/test.yaml
> >> +++ b/automation/gitlab-ci/test.yaml
> >> @@ -44,6 +44,25 @@ qemu-alpine-arm64-gcc:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-alpine-arm64-gcc-debug:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> >> +  needs:
> >> +    - alpine-3.12-gcc-debug-arm64
> >> +    - alpine-3.12-arm64-rootfs-export
> >> +    - kernel-5.19-arm64-export
> >> +    - qemu-system-aarch64-6.0.0-arm64-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-alpine-x86_64-gcc:
> >>    extends: .test-jobs-common
> >>    variables:
> >> @@ -81,6 +100,25 @@ qemu-smoke-arm64-gcc:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-smoke-arm64-gcc-debug:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> >> +  needs:
> >> +    - alpine-3.12-gcc-debug-arm64
> >> +    - alpine-3.12-arm64-rootfs-export
> >> +    - kernel-5.19-arm64-export
> >> +    - qemu-system-aarch64-6.0.0-arm64-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-smoke-arm64-gcc-staticmem:
> >>    extends: .test-jobs-common
> >>    variables:
> >> @@ -100,6 +138,25 @@ qemu-smoke-arm64-gcc-staticmem:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-smoke-arm64-gcc-debug-staticmem:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
> >> +  needs:
> >> +    - alpine-3.12-gcc-debug-arm64-staticmem
> >> +    - alpine-3.12-arm64-rootfs-export
> >> +    - kernel-5.19-arm64-export
> >> +    - qemu-system-aarch64-6.0.0-arm64-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-smoke-arm64-gcc-boot-cpupools:
> >>    extends: .test-jobs-common
> >>    variables:
> >> @@ -119,6 +176,25 @@ qemu-smoke-arm64-gcc-boot-cpupools:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-smoke-arm64-gcc-debug-boot-cpupools:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
> >> +  needs:
> >> +    - alpine-3.12-gcc-debug-arm64-boot-cpupools
> >> +    - alpine-3.12-arm64-rootfs-export
> >> +    - kernel-5.19-arm64-export
> >> +    - qemu-system-aarch64-6.0.0-arm64-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-smoke-arm32-gcc:
> >>    extends: .test-jobs-common
> >>    variables:
> >> @@ -136,6 +212,23 @@ qemu-smoke-arm32-gcc:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-smoke-arm32-gcc-debug:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
> >> +  needs:
> >> +    - debian-unstable-gcc-arm32-debug
> >> +    - qemu-system-aarch64-6.0.0-arm32-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-smoke-x86-64-gcc:
> >>    extends: .test-jobs-common
> >>    variables:
> >> diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-arm64.sh
> >> index f803835779f4..dea26c6c0a8f 100755
> >> --- a/automation/scripts/qemu-smoke-arm64.sh
> >> +++ b/automation/scripts/qemu-smoke-arm64.sh
> >> @@ -52,7 +52,7 @@ curl -fsSLO https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fraw%2Fv5.2.0%2Fpc-bios%2Fefi-virtio.rom&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cf6b0b8fabb02478b009c08da9ce4eac9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994804254367719%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=jgGdO93kLoieg6UqajFWYAsufvvv5mZVYHD%2B7fUW7ME%3D&amp;reserved=0
> >>  ./binaries/qemu-system-aarch64 \
> >>     -machine virtualization=true \
> >>     -cpu cortex-a57 -machine type=virt \
> >> -   -m 1024 -smp 2 -display none \
> >> +   -m 2048 -smp 2 -display none \
> >>     -machine dumpdtb=binaries/virt-gicv2.dtb
> >>
> >>  # XXX disable pl061 to avoid Linux crash
> >> @@ -117,7 +117,7 @@ cd ..
> >>
> >>  # ImageBuilder
> >>  echo 'MEMORY_START="0x40000000"
> >> -MEMORY_END="0x80000000"
> >> +MEMORY_END="0xC0000000"
> >>
> >>  DEVICE_TREE="virt-gicv2.dtb"
> >>  XEN="xen"
> >> @@ -158,7 +158,7 @@ timeout -k 1 240 \
> >>  ./binaries/qemu-system-aarch64 \
> >>      -machine virtualization=true \
> >>      -cpu cortex-a57 -machine type=virt \
> >> -    -m 1024 -monitor none -serial stdio \
> >> +    -m 2048 -monitor none -serial stdio \
> >>      -smp 2 \
> >>      -no-reboot \
> >>      -device virtio-net-pci,netdev=n0 \
> > 
> > The changes to qemu-smoke-arm64.sh make sense, but could you move them
> > to a separate patch in case we need to do any backports?
> 
> Ok, I will do a prerequisite patch for that one.
> 
> > 
> > Also I would change MEMORY_END to 0xC0000000 in qemu-alpine-arm64.sh too
> > for uniformity.
> 
> I already did that in patch [6/9] automation: qemu-alpine-arm64: Cleanup and fixes.
> 
> ~Michal
> 

Re: [PATCH 5/9] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder

[Stefano Stabellini]

On Fri, 23 Sep 2022, Michal Orzel wrote:
> Hi Stefano,
> 
> On 23/09/2022 00:13, Stefano Stabellini wrote:
> > 
> > 
> > On Thu, 22 Sep 2022, Michal Orzel wrote:
> >> Take an example from arm64 qemu test scripts and use ImageBuilder
> >> to generate u-boot script automatically. Calculating the addresses
> >> manually is quite error prone and also we will be able to benefit
> >> from using ImageBuilder when adding domUs to this test in the future.
> >>
> >> Install and use u-boot from the debian package.
> >> Modify the script so that binaries are loaded from u-boot via tftp.
> > 
> > Great patch! It makes the test a lot better!
> > 
> > 
> >> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> >> ---
> >>  automation/scripts/qemu-smoke-arm32.sh | 57 ++++++++++++--------------
> >>  1 file changed, 27 insertions(+), 30 deletions(-)
> >>
> >> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
> >> index 530f3892fdd3..765facbe4d66 100755
> >> --- a/automation/scripts/qemu-smoke-arm32.sh
> >> +++ b/automation/scripts/qemu-smoke-arm32.sh
> >> @@ -4,7 +4,9 @@ set -ex
> >>
> >>  export DEBIAN_FRONTENT=noninteractive
> >>  apt-get -qy update
> >> -apt-get -qy install --no-install-recommends device-tree-compiler \
> >> +apt-get -qy install --no-install-recommends u-boot-qemu \
> >> +                                            u-boot-tools \
> >> +                                            device-tree-compiler \
> >>                                              curl \
> >>                                              cpio
> >>
> >> @@ -20,10 +22,6 @@ tar xvzf ../initrd.tar.gz
> >>  find . | cpio -H newc -o | gzip > ../initrd.gz
> >>  cd ..
> >>
> >> -kernel=`stat -L --printf="%s" vmlinuz`
> >> -initrd=`stat -L --printf="%s" initrd.gz`
> >> -
> >> -# For Xen, we need a couple of more node. Dump the DT from QEMU and add them
> >>  # XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
> >>  curl -fsSLO https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fraw%2Fv5.2.0%2Fpc-bios%2Fefi-virtio.rom&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd33d5c4c08934fac0cc208da9ce7a3fb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994815957091528%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=JuEapfYVLj5P3S5yY%2BAa47Nk4zgbdymjjsiUTmoTmyk%3D&amp;reserved=0
> >>  ./qemu-system-arm \
> >> @@ -36,31 +34,31 @@ curl -fsSLO https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fraw%2Fv5.2.0%2Fpc-bios%2Fefi-virtio.rom&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd33d5c4c08934fac0cc208da9ce7a3fb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994815957091528%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=JuEapfYVLj5P3S5yY%2BAa47Nk4zgbdymjjsiUTmoTmyk%3D&amp;reserved=0
> >>     -display none \
> >>     -machine dumpdtb=virt.dtb
> >>
> >> -dtc -I dtb -O dts virt.dtb > virt.dts
> >> +# ImageBuilder
> >> +echo 'MEMORY_START="0x40000000"
> >> +MEMORY_END="0x80000000"
> >> +
> >> +DEVICE_TREE="virt.dtb"
> >> +XEN="xen"
> >> +DOM0_KERNEL="vmlinuz"
> >> +DOM0_RAMDISK="initrd.gz"
> >> +DOM0_CMD="console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh"
> >> +XEN_CMD="console=dtuart dom0_mem=512M bootscrub=0"
> > 
> > This is missing dtuart=/pl011@9000000 compared to the original
> 
> Actually this is not needed because we always have stdout-path defined in chosen to point to /pl011@...

Ah OK, in that case just add a note to the commit message


> but I can add this.

no need


> > 
> > 
> >> +NUM_DOMUS=0
> >> +
> >> +LOAD_CMD="tftpb"
> >> +BOOT_CMD="bootm"
> > 
> > "bootm" because "booti" is not available on arm32, right?
> 
> Exactly.
> > 
> > 
> >> +UBOOT_SOURCE="boot.source"
> >> +UBOOT_SCRIPT="boot.scr"' > config
> >>
> >> -cat >> virt.dts << EOF
> >> -/ {
> >> -     chosen {
> >> -             #address-cells = <0x2>;
> >> -             #size-cells = <0x2>;
> >> -             stdout-path = "/pl011@9000000";
> >> -        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=512M bootscrub=0";
> >> -             xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh";
> >> -             dom0 {
> >> -                     compatible = "xen,linux-zimage", "xen,multiboot-module";
> >> -                     reg = <0x0 0x1000000 0x0 $kernel>;
> >> -             };
> >> -        dom0-ramdisk {
> >> -                     compatible = "xen,linux-initrd", "xen,multiboot-module";
> >> -                     reg = <0x0 0x3200000 0x0 $initrd>;
> >> -             };
> >> -     };
> >> -};
> >> -EOF
> >> -dtc -I dts -O dtb virt.dts > virt.dtb
> >> +rm -rf imagebuilder
> >> +git clone https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2FViryaOS%2Fimagebuilder&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cd33d5c4c08934fac0cc208da9ce7a3fb%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994815957091528%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=i5N8DXMjn%2F80mdXtc%2FwDGJw6ImUrUAjOg0SJp9CkRdQ%3D&amp;reserved=0
> >> +bash imagebuilder/scripts/uboot-script-gen -t tftp -d . -c config
> >>
> >>  rm -f smoke.serial
> >>  set +e
> >> +echo "  virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x40000000"| \
> >>  timeout -k 1 240 \
> >>  ./qemu-system-arm \
> >>     -machine virt \
> >> @@ -70,11 +68,10 @@ timeout -k 1 240 \
> >>     -serial stdio \
> >>     -monitor none \
> >>     -display none \
> >> -   -dtb virt.dtb \
> >>     -no-reboot \
> >> -   -kernel ./xen \
> >> -   -device loader,file=./vmlinuz,addr=0x1000000 \
> >> -   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
> >> +   -device virtio-net-pci,netdev=n0 \
> >> +   -netdev user,id=n0,tftp=./ \
> >> +   -bios /usr/lib/u-boot/qemu_arm/u-boot.bin |& tee smoke.serial
> >>
> >>  set -e
> >>  (grep -q "^/ #" smoke.serial) || exit 1
> >> --
> >> 2.25.1
> >>
> 
> ~Michal
> 

Re: [PATCH 4/9] automation: Add Arm containers to containerize script

[Stefano Stabellini]

On Fri, 23 Sep 2022, Michal Orzel wrote:
> Hi Anthony,
> 
> On 23/09/2022 15:56, Anthony PERARD wrote:
> > 
> > 
> > On Thu, Sep 22, 2022 at 03:40:53PM +0200, Michal Orzel wrote:
> >> Script automation/scripts/containerize makes it easy to build Xen within
> >> predefined containers from gitlab container registry. However, it is
> >> currently not possible to use it with Arm containers because they are not
> >> listed in the script. Populate the necessary entries.
> > 
> > FYI, those entry are just helper/shortcut/aliases, you can use any
> > arbitrary container with the script, it just more annoying.
> > 
> > Your patch here allows to write:
> >     CONTAINER=unstable-arm64v8 automation/scripts/containerize
> > but you could write the following instead, for the same result:
> >     CONTAINER=registry.gitlab.com/xen-project/xen/debian:unstable-arm64v8 automation/scripts/containerize
> > 
> > I wonder if the script could select the right container base on the
> > architecture of the host, because "alpine" and "alpine-arm64v8" will not
> > both work on the same machine. It might be nice to just choose "alpine"
> > and the script would select the x86 or arm container automagically. Just
> > an idea, no need to do anything about it.
> I'm not in favor of adding the automatic selection based on the host.
> The reason is that on x86 you can run both x86 and e.g. Arm containers.
> You just need to use register qemu-user-static [1] to perform emulation.
> This is something widely used and I use that to test Arm images/containers on x86 host.
> So you can run both alpine and alpine-arm64v8 on the same machine.

Yeah and modern Docker sets up qemu-user-static automatically without
the user having to do anything. Anthony, you can try it yourself: you
should be able to just:

  docker run -it registry.gitlab.com/xen-project/xen/debian:unstable-arm64v8

on your x86 host if you have a docker new enough

Re: [PATCH 4/9] automation: Add Arm containers to containerize script

[Anthony PERARD]

On Fri, Sep 23, 2022 at 03:23:26PM -0700, Stefano Stabellini wrote:
> On Fri, 23 Sep 2022, Michal Orzel wrote:
> > Hi Anthony,
> > 
> > On 23/09/2022 15:56, Anthony PERARD wrote:
> > > 
> > > 
> > > On Thu, Sep 22, 2022 at 03:40:53PM +0200, Michal Orzel wrote:
> > >> Script automation/scripts/containerize makes it easy to build Xen within
> > >> predefined containers from gitlab container registry. However, it is
> > >> currently not possible to use it with Arm containers because they are not
> > >> listed in the script. Populate the necessary entries.
> > > 
> > > FYI, those entry are just helper/shortcut/aliases, you can use any
> > > arbitrary container with the script, it just more annoying.
> > > 
> > > Your patch here allows to write:
> > >     CONTAINER=unstable-arm64v8 automation/scripts/containerize
> > > but you could write the following instead, for the same result:
> > >     CONTAINER=registry.gitlab.com/xen-project/xen/debian:unstable-arm64v8 automation/scripts/containerize
> > > 
> > > I wonder if the script could select the right container base on the
> > > architecture of the host, because "alpine" and "alpine-arm64v8" will not
> > > both work on the same machine. It might be nice to just choose "alpine"
> > > and the script would select the x86 or arm container automagically. Just
> > > an idea, no need to do anything about it.
> > I'm not in favor of adding the automatic selection based on the host.
> > The reason is that on x86 you can run both x86 and e.g. Arm containers.
> > You just need to use register qemu-user-static [1] to perform emulation.
> > This is something widely used and I use that to test Arm images/containers on x86 host.
> > So you can run both alpine and alpine-arm64v8 on the same machine.
> 
> Yeah and modern Docker sets up qemu-user-static automatically without
> the user having to do anything. Anthony, you can try it yourself: you
> should be able to just:
> 
>   docker run -it registry.gitlab.com/xen-project/xen/debian:unstable-arm64v8
> 
> on your x86 host if you have a docker new enough

No, it isn't so easy, there is at least one step that might be needed,
actually having qemu-user-static on the machine.

But thanks to both of you, I've learned about qemu-user-static, and that
it isn't necessarily complicated to setup. It is actually very easy to
setup once we know what to look for, I've just had to install a packaged
called "qemu-user-static" and that's all that was needed for the above
command line to work. Installing that package worked on two different
Linux distribution for me (on a derivative of Debian stable and on Arch
Linux) so it's likely to work in many cases, at least on x86.

Cheers,

-- 
Anthony PERARD