All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm
@ 2022-09-26 11:04 Michal Orzel
  2022-09-26 11:04 ` [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed Michal Orzel
                   ` (9 more replies)
  0 siblings, 10 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

This patch series performs necessary cleanup and improvements in our
GitLab CI automation for Arm. This is crucial so that in the future
we can focus on adding new tests instead of spending time to fix
issues, making the behavior consistent, removing ambiguity, etc.

With the increased interest in gitlab-ci and using it locally, there
is still some work to do mainly to improve documentation. This however,
will be handled in the future series.

Michal Orzel (10):
  automation: Use custom build jobs when extra config options are needed
  automation: Add randconfig build jobs for arm64 alpine container
  automation: qemu-smoke-arm64.sh: Increase RAM size
  automation: Add debug versions of Arm tests
  automation: Add Arm containers to containerize script
  automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder
  automation: qemu-alpine-arm64: Cleanup and fixes
  automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh
  automation: Rename qemu-alpine-arm64.sh to qemu-smoke-dom0-arm64.sh
  automation: Rename qemu-smoke-arm32.sh to qemu-smoke-dom0-arm32.sh

 automation/gitlab-ci/build.yaml               |  43 +++++++
 automation/gitlab-ci/test.yaml                | 117 ++++++++++++++++--
 automation/scripts/build                      |   8 +-
 automation/scripts/containerize               |   3 +
 ...moke-arm32.sh => qemu-smoke-dom0-arm32.sh} |  57 ++++-----
 ...pine-arm64.sh => qemu-smoke-dom0-arm64.sh} |  14 +--
 ...-arm64.sh => qemu-smoke-dom0less-arm64.sh} |   6 +-
 7 files changed, 190 insertions(+), 58 deletions(-)
 rename automation/scripts/{qemu-smoke-arm32.sh => qemu-smoke-dom0-arm32.sh} (55%)
 rename automation/scripts/{qemu-alpine-arm64.sh => qemu-smoke-dom0-arm64.sh} (88%)
 rename automation/scripts/{qemu-smoke-arm64.sh => qemu-smoke-dom0less-arm64.sh} (97%)

-- 
2.25.1



^ permalink raw reply	[flat|nested] 25+ messages in thread
* [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  2022-09-26 12:03   ` Luca Fancellu
                     ` (2 more replies)
  2022-09-26 11:04 ` [PATCH v2 02/10] automation: Add randconfig build jobs for arm64 alpine container Michal Orzel
                   ` (8 subsequent siblings)
  9 siblings, 3 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

Currently, all the arm64 defconfig build jobs, regardless of the
container used, end up building Xen with the extra config options
specified in the main build script (e.g. CONFIG_EXPERT,
CONFIG_STATIC_MEMORY). Because these options are only needed for
specific test jobs, the current behavior of the CI is incorrect
as we add the extra options to all the defconfig builds. This means
that on arm64 there is not a single job performing proper defconfig build.

To fix this issue, add custom build jobs each time there is a need for
building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
variable to be used by these jobs to store the required options. This
variable will be then read by the main build script to modify the .config
file. This will also help users to understand what is needed to run specific
test.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
This patch is supposed to be merged for 4.17. The release manager agreed
on that.

Changes in v2:
- replace [ with [[ to be consistent with other ifs in build script
---
 automation/gitlab-ci/build.yaml | 15 +++++++++++++++
 automation/gitlab-ci/test.yaml  |  4 ++--
 automation/scripts/build        |  8 ++------
 3 files changed, 19 insertions(+), 8 deletions(-)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 720ce6e07ba0..a39ed72aac6d 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -566,6 +566,21 @@ alpine-3.12-gcc-debug-arm64:
   variables:
     CONTAINER: alpine:3.12-arm64v8
 
+alpine-3.12-gcc-arm64-staticmem:
+  extends: .gcc-arm64-build
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    EXTRA_XEN_CONFIG: |
+      CONFIG_EXPERT=y
+      CONFIG_UNSUPPORTED=y
+      CONFIG_STATIC_MEMORY=y
+
+alpine-3.12-gcc-arm64-boot-cpupools:
+  extends: .gcc-arm64-build
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    EXTRA_XEN_CONFIG: |
+      CONFIG_BOOT_TIME_CPUPOOLS=y
 
 ## Test artifacts common
 
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index d899b3bdbf7a..4f96e6e322de 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -88,7 +88,7 @@ qemu-smoke-arm64-gcc-staticmem:
   script:
     - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
   needs:
-    - alpine-3.12-gcc-arm64
+    - alpine-3.12-gcc-arm64-staticmem
     - alpine-3.12-arm64-rootfs-export
     - kernel-5.19-arm64-export
     - qemu-system-aarch64-6.0.0-arm64-export
@@ -107,7 +107,7 @@ qemu-smoke-arm64-gcc-boot-cpupools:
   script:
     - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
   needs:
-    - alpine-3.12-gcc-arm64
+    - alpine-3.12-gcc-arm64-boot-cpupools
     - alpine-3.12-arm64-rootfs-export
     - kernel-5.19-arm64-export
     - qemu-system-aarch64-6.0.0-arm64-export
diff --git a/automation/scripts/build b/automation/scripts/build
index 2f15ab3198e6..bcfa6838f0bb 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -15,12 +15,8 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
     make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
     hypervisor_only="y"
 else
-    if [[ "${XEN_TARGET_ARCH}" = "arm64" ]]; then
-        echo "
-CONFIG_EXPERT=y
-CONFIG_UNSUPPORTED=y
-CONFIG_STATIC_MEMORY=y
-CONFIG_BOOT_TIME_CPUPOOLS=y" > xen/.config
+    if [[ -n "${EXTRA_XEN_CONFIG}" ]]; then
+        echo "${EXTRA_XEN_CONFIG}" > xen/.config
         make -j$(nproc) -C xen olddefconfig
     else
         make -j$(nproc) -C xen defconfig
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* [PATCH v2 02/10] automation: Add randconfig build jobs for arm64 alpine container
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
  2022-09-26 11:04 ` [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  2022-09-26 12:00   ` Luca Fancellu
  2022-09-26 22:50   ` Stefano Stabellini
  2022-09-26 11:04 ` [PATCH v2 03/10] automation: qemu-smoke-arm64.sh: Increase RAM size Michal Orzel
                   ` (7 subsequent siblings)
  9 siblings, 2 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

For arm64 we perform builds using debian and alpine containers.
We are missing the randconfig build jobs for the latter, so add them.
This way for each container we have 4 fundamental build jobs:
- defconfig non-debug/debug
- randconfig non-debug/debug

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
Changes in v2:
- none
---
 automation/gitlab-ci/build.yaml | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index a39ed72aac6d..d931441b910a 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -566,6 +566,18 @@ alpine-3.12-gcc-debug-arm64:
   variables:
     CONTAINER: alpine:3.12-arm64v8
 
+alpine-3.12-gcc-arm64-randconfig:
+  extends: .gcc-arm64-build
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    RANDCONFIG: y
+
+alpine-3.12-gcc-debug-arm64-randconfig:
+  extends: .gcc-arm64-build-debug
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    RANDCONFIG: y
+
 alpine-3.12-gcc-arm64-staticmem:
   extends: .gcc-arm64-build
   variables:
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* [PATCH v2 03/10] automation: qemu-smoke-arm64.sh: Increase RAM size
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
  2022-09-26 11:04 ` [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed Michal Orzel
  2022-09-26 11:04 ` [PATCH v2 02/10] automation: Add randconfig build jobs for arm64 alpine container Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  2022-09-26 12:02   ` Luca Fancellu
  2022-09-26 22:50   ` Stefano Stabellini
  2022-09-26 11:04 ` [PATCH v2 04/10] automation: Add debug versions of Arm tests Michal Orzel
                   ` (6 subsequent siblings)
  9 siblings, 2 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

In the follow-up patch we will add new jobs using debug Xen builds.
Because the debug builds take more space and we might end up in
a situation when there is not enough free space (especially during
a static memory test that reserves some region in the middle), increase
RAM size for QEMU from 1GB to 2GB.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
Changes in v2:
- new patch carved out from previous v1 [3/9] patch
---
 automation/scripts/qemu-smoke-arm64.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-arm64.sh
index f803835779f4..dea26c6c0a8f 100755
--- a/automation/scripts/qemu-smoke-arm64.sh
+++ b/automation/scripts/qemu-smoke-arm64.sh
@@ -52,7 +52,7 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
 ./binaries/qemu-system-aarch64 \
    -machine virtualization=true \
    -cpu cortex-a57 -machine type=virt \
-   -m 1024 -smp 2 -display none \
+   -m 2048 -smp 2 -display none \
    -machine dumpdtb=binaries/virt-gicv2.dtb
 
 # XXX disable pl061 to avoid Linux crash
@@ -117,7 +117,7 @@ cd ..
 
 # ImageBuilder
 echo 'MEMORY_START="0x40000000"
-MEMORY_END="0x80000000"
+MEMORY_END="0xC0000000"
 
 DEVICE_TREE="virt-gicv2.dtb"
 XEN="xen"
@@ -158,7 +158,7 @@ timeout -k 1 240 \
 ./binaries/qemu-system-aarch64 \
     -machine virtualization=true \
     -cpu cortex-a57 -machine type=virt \
-    -m 1024 -monitor none -serial stdio \
+    -m 2048 -monitor none -serial stdio \
     -smp 2 \
     -no-reboot \
     -device virtio-net-pci,netdev=n0 \
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* [PATCH v2 04/10] automation: Add debug versions of Arm tests
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
                   ` (2 preceding siblings ...)
  2022-09-26 11:04 ` [PATCH v2 03/10] automation: qemu-smoke-arm64.sh: Increase RAM size Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  2022-09-26 12:06   ` Luca Fancellu
  2022-09-26 22:50   ` Stefano Stabellini
  2022-09-26 11:04 ` [PATCH v2 05/10] automation: Add Arm containers to containerize script Michal Orzel
                   ` (5 subsequent siblings)
  9 siblings, 2 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

At the moment, all the tests are executed on non-debug Xen builds.
To improve the coverage (e.g. we might catch some asserts), add new
test jobs using debug Xen builds.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
Changes in v2:
- carve out the RAM size change to a separate patch [v2 03/10]
---
 automation/gitlab-ci/build.yaml | 16 ++++++
 automation/gitlab-ci/test.yaml  | 93 +++++++++++++++++++++++++++++++++
 2 files changed, 109 insertions(+)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index d931441b910a..6ce92531f61f 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -587,6 +587,15 @@ alpine-3.12-gcc-arm64-staticmem:
       CONFIG_UNSUPPORTED=y
       CONFIG_STATIC_MEMORY=y
 
+alpine-3.12-gcc-debug-arm64-staticmem:
+  extends: .gcc-arm64-build-debug
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    EXTRA_XEN_CONFIG: |
+      CONFIG_EXPERT=y
+      CONFIG_UNSUPPORTED=y
+      CONFIG_STATIC_MEMORY=y
+
 alpine-3.12-gcc-arm64-boot-cpupools:
   extends: .gcc-arm64-build
   variables:
@@ -594,6 +603,13 @@ alpine-3.12-gcc-arm64-boot-cpupools:
     EXTRA_XEN_CONFIG: |
       CONFIG_BOOT_TIME_CPUPOOLS=y
 
+alpine-3.12-gcc-debug-arm64-boot-cpupools:
+  extends: .gcc-arm64-build-debug
+  variables:
+    CONTAINER: alpine:3.12-arm64v8
+    EXTRA_XEN_CONFIG: |
+      CONFIG_BOOT_TIME_CPUPOOLS=y
+
 ## Test artifacts common
 
 .test-jobs-artifact-common:
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 4f96e6e322de..1b51030c6175 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -44,6 +44,25 @@ qemu-alpine-arm64-gcc:
   tags:
     - arm64
 
+qemu-alpine-arm64-gcc-debug:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+  needs:
+    - alpine-3.12-gcc-debug-arm64
+    - alpine-3.12-arm64-rootfs-export
+    - kernel-5.19-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-alpine-x86_64-gcc:
   extends: .test-jobs-common
   variables:
@@ -81,6 +100,25 @@ qemu-smoke-arm64-gcc:
   tags:
     - arm64
 
+qemu-smoke-arm64-gcc-debug:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+  needs:
+    - alpine-3.12-gcc-debug-arm64
+    - alpine-3.12-arm64-rootfs-export
+    - kernel-5.19-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-smoke-arm64-gcc-staticmem:
   extends: .test-jobs-common
   variables:
@@ -100,6 +138,25 @@ qemu-smoke-arm64-gcc-staticmem:
   tags:
     - arm64
 
+qemu-smoke-arm64-gcc-debug-staticmem:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
+  needs:
+    - alpine-3.12-gcc-debug-arm64-staticmem
+    - alpine-3.12-arm64-rootfs-export
+    - kernel-5.19-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-smoke-arm64-gcc-boot-cpupools:
   extends: .test-jobs-common
   variables:
@@ -119,6 +176,25 @@ qemu-smoke-arm64-gcc-boot-cpupools:
   tags:
     - arm64
 
+qemu-smoke-arm64-gcc-debug-boot-cpupools:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
+  needs:
+    - alpine-3.12-gcc-debug-arm64-boot-cpupools
+    - alpine-3.12-arm64-rootfs-export
+    - kernel-5.19-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-smoke-arm32-gcc:
   extends: .test-jobs-common
   variables:
@@ -136,6 +212,23 @@ qemu-smoke-arm32-gcc:
   tags:
     - arm64
 
+qemu-smoke-arm32-gcc-debug:
+  extends: .test-jobs-common
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
+  needs:
+    - debian-unstable-gcc-arm32-debug
+    - qemu-system-aarch64-6.0.0-arm32-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+
 qemu-smoke-x86-64-gcc:
   extends: .test-jobs-common
   variables:
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* [PATCH v2 05/10] automation: Add Arm containers to containerize script
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
                   ` (3 preceding siblings ...)
  2022-09-26 11:04 ` [PATCH v2 04/10] automation: Add debug versions of Arm tests Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  2022-09-26 11:04 ` [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder Michal Orzel
                   ` (4 subsequent siblings)
  9 siblings, 0 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

Script automation/scripts/containerize makes it easy to build Xen within
predefined containers from gitlab container registry. This script is
currently missing the helpers to select Arm containers, so populate the
necessary entries.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
---
Changes in v2:
- modify commit msg to reflect that we are missing helpers but in reality
  it could be possible to use Arm containers by specifying the full path
  to gitlab container registry. However, such usage is annoying.
---
 automation/scripts/containerize | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 9d4beca4fa4b..0f4645c4cccb 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -25,6 +25,7 @@ die() {
 BASE="registry.gitlab.com/xen-project/xen"
 case "_${CONTAINER}" in
     _alpine) CONTAINER="${BASE}/alpine:3.12" ;;
+    _alpine-arm64v8) CONTAINER="${BASE}/alpine:3.12-arm64v8" ;;
     _archlinux|_arch) CONTAINER="${BASE}/archlinux:current" ;;
     _riscv64) CONTAINER="${BASE}/archlinux:riscv64" ;;
     _centos7) CONTAINER="${BASE}/centos:7" ;;
@@ -35,6 +36,8 @@ case "_${CONTAINER}" in
     _stretch|_) CONTAINER="${BASE}/debian:stretch" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _unstable|_) CONTAINER="${BASE}/debian:unstable" ;;
+    _unstable-arm32-gcc) CONTAINER="${BASE}/debian:unstable-arm32-gcc" ;;
+    _unstable-arm64v8) CONTAINER="${BASE}/debian:unstable-arm64v8" ;;
     _trusty) CONTAINER="${BASE}/ubuntu:trusty" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
                   ` (4 preceding siblings ...)
  2022-09-26 11:04 ` [PATCH v2 05/10] automation: Add Arm containers to containerize script Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  2022-09-26 12:03   ` Luca Fancellu
  2022-09-26 14:46   ` Stewart Hildebrand
  2022-09-26 11:04 ` [PATCH v2 07/10] automation: qemu-alpine-arm64: Cleanup and fixes Michal Orzel
                   ` (3 subsequent siblings)
  9 siblings, 2 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

Take an example from arm64 qemu test scripts and use ImageBuilder
to generate u-boot script automatically. Calculating the addresses
manually is quite error prone and also we will be able to benefit
from using ImageBuilder when adding domUs to this test in the future.

Install and use u-boot from the debian package.
Modify the script so that binaries are loaded from u-boot via tftp.
Remove dtuart=/pl011@9000000 because stdout-path in QEMU dtb is always
set to pl011 node path.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
---
Changes in v2:
- add explanation in commit msg why we remove dtuart path
---
 automation/scripts/qemu-smoke-arm32.sh | 57 ++++++++++++--------------
 1 file changed, 27 insertions(+), 30 deletions(-)

diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
index 530f3892fdd3..765facbe4d66 100755
--- a/automation/scripts/qemu-smoke-arm32.sh
+++ b/automation/scripts/qemu-smoke-arm32.sh
@@ -4,7 +4,9 @@ set -ex
 
 export DEBIAN_FRONTENT=noninteractive
 apt-get -qy update
-apt-get -qy install --no-install-recommends device-tree-compiler \
+apt-get -qy install --no-install-recommends u-boot-qemu \
+                                            u-boot-tools \
+                                            device-tree-compiler \
                                             curl \
                                             cpio
 
@@ -20,10 +22,6 @@ tar xvzf ../initrd.tar.gz
 find . | cpio -H newc -o | gzip > ../initrd.gz
 cd ..
 
-kernel=`stat -L --printf="%s" vmlinuz`
-initrd=`stat -L --printf="%s" initrd.gz`
-
-# For Xen, we need a couple of more node. Dump the DT from QEMU and add them
 # XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
 curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
 ./qemu-system-arm \
@@ -36,31 +34,31 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
    -display none \
    -machine dumpdtb=virt.dtb
 
-dtc -I dtb -O dts virt.dtb > virt.dts
+# ImageBuilder
+echo 'MEMORY_START="0x40000000"
+MEMORY_END="0x80000000"
+
+DEVICE_TREE="virt.dtb"
+XEN="xen"
+DOM0_KERNEL="vmlinuz"
+DOM0_RAMDISK="initrd.gz"
+DOM0_CMD="console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh"
+XEN_CMD="console=dtuart dom0_mem=512M bootscrub=0"
+
+NUM_DOMUS=0
+
+LOAD_CMD="tftpb"
+BOOT_CMD="bootm"
+UBOOT_SOURCE="boot.source"
+UBOOT_SCRIPT="boot.scr"' > config
 
-cat >> virt.dts << EOF
-/ {
-	chosen {
-		#address-cells = <0x2>;
-		#size-cells = <0x2>;
-		stdout-path = "/pl011@9000000";
-        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=512M bootscrub=0";
-		xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh";
-		dom0 {
-			compatible = "xen,linux-zimage", "xen,multiboot-module";
-			reg = <0x0 0x1000000 0x0 $kernel>;
-		};
-        dom0-ramdisk {
-			compatible = "xen,linux-initrd", "xen,multiboot-module";
-			reg = <0x0 0x3200000 0x0 $initrd>;
-		};
-	};
-};
-EOF
-dtc -I dts -O dtb virt.dts > virt.dtb
+rm -rf imagebuilder
+git clone https://gitlab.com/ViryaOS/imagebuilder
+bash imagebuilder/scripts/uboot-script-gen -t tftp -d . -c config
 
 rm -f smoke.serial
 set +e
+echo "  virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x40000000"| \
 timeout -k 1 240 \
 ./qemu-system-arm \
    -machine virt \
@@ -70,11 +68,10 @@ timeout -k 1 240 \
    -serial stdio \
    -monitor none \
    -display none \
-   -dtb virt.dtb \
    -no-reboot \
-   -kernel ./xen \
-   -device loader,file=./vmlinuz,addr=0x1000000 \
-   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
+   -device virtio-net-pci,netdev=n0 \
+   -netdev user,id=n0,tftp=./ \
+   -bios /usr/lib/u-boot/qemu_arm/u-boot.bin |& tee smoke.serial
 
 set -e
 (grep -q "^/ #" smoke.serial) || exit 1
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* [PATCH v2 07/10] automation: qemu-alpine-arm64: Cleanup and fixes
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
                   ` (5 preceding siblings ...)
  2022-09-26 11:04 ` [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  2022-09-26 11:04 ` [PATCH v2 08/10] automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh Michal Orzel
                   ` (2 subsequent siblings)
  9 siblings, 0 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

Perform the following cleanup:
- rename the device tree from virt-gicv3 to virt-gicv2 as the GIC version
  used in this test is v2,
- use fdtput to perform modifications on the dtb,
- use DEBIAN_FRONTENT=noninteractive to prevent interactive prompt being
  stuck waiting for answer other than "yes",
- fix the number of cpus in the device tree because currently we generate
  it with a single cpu and try to run QEMU with two,
- fix the memory size we pass when generating QEMU device tree as it does
  not match the memory size with what we run QEMU.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
---
Changes in v2:
- none
---
 automation/scripts/qemu-alpine-arm64.sh | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/automation/scripts/qemu-alpine-arm64.sh b/automation/scripts/qemu-alpine-arm64.sh
index f4ac2d856fa0..7b52d77d3c84 100755
--- a/automation/scripts/qemu-alpine-arm64.sh
+++ b/automation/scripts/qemu-alpine-arm64.sh
@@ -2,6 +2,7 @@
 
 set -ex
 
+export DEBIAN_FRONTENT=noninteractive
 apt-get -qy update
 apt-get -qy install --no-install-recommends u-boot-qemu \
                                             u-boot-tools \
@@ -73,18 +74,17 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
 ./binaries/qemu-system-aarch64 \
    -machine virtualization=true \
    -cpu cortex-a57 -machine type=virt \
-   -m 1024 -display none \
-   -machine dumpdtb=binaries/virt-gicv3.dtb
+   -m 2048 -smp 2 -display none \
+   -machine dumpdtb=binaries/virt-gicv2.dtb
+
 # XXX disable pl061 to avoid Linux crash
-dtc -I dtb -O dts binaries/virt-gicv3.dtb > binaries/virt-gicv3.dts
-sed 's/compatible = "arm,pl061.*/status = "disabled";/g' binaries/virt-gicv3.dts > binaries/virt-gicv3-edited.dts
-dtc -I dts -O dtb binaries/virt-gicv3-edited.dts > binaries/virt-gicv3.dtb
+fdtput binaries/virt-gicv2.dtb -p -t s /pl061@9030000 status disabled
 
 # ImageBuilder
 echo 'MEMORY_START="0x40000000"
-MEMORY_END="0x80000000"
+MEMORY_END="0xC0000000"
 
-DEVICE_TREE="virt-gicv3.dtb"
+DEVICE_TREE="virt-gicv2.dtb"
 XEN="xen"
 DOM0_KERNEL="Image"
 DOM0_RAMDISK="xen-rootfs.cpio.gz"
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* [PATCH v2 08/10] automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
                   ` (6 preceding siblings ...)
  2022-09-26 11:04 ` [PATCH v2 07/10] automation: qemu-alpine-arm64: Cleanup and fixes Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  2022-09-26 15:12   ` Andrew Cooper
  2022-09-26 11:04 ` [PATCH v2 09/10] automation: Rename qemu-alpine-arm64.sh to qemu-smoke-dom0-arm64.sh Michal Orzel
  2022-09-26 11:04 ` [PATCH v2 10/10] automation: Rename qemu-smoke-arm32.sh to qemu-smoke-dom0-arm32.sh Michal Orzel
  9 siblings, 1 reply; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

Testing arm64 is done using the qemu-alpine-arm64.sh and
qemu-smoke-arm64.sh scripts. These scripts are executed with exactly
the same artifacts (container, rootfs, kernel, qemu) and the only
difference is that the former is used to perform dom0 based testing
and the latter - dom0less based testing.

Because the current naming is quite umbiguous, rename qemu-smoke-arm64.sh
script to qemu-smoke-dom0less-arm64.sh to reflect its usage.

qemu-alpine-arm64.sh will be renamed in the follow-up patch.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
---
Changes in v2:
- none
---
 automation/gitlab-ci/test.yaml                | 24 +++++++++----------
 ...-arm64.sh => qemu-smoke-dom0less-arm64.sh} |  0
 2 files changed, 12 insertions(+), 12 deletions(-)
 rename automation/scripts/{qemu-smoke-arm64.sh => qemu-smoke-dom0less-arm64.sh} (100%)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 1b51030c6175..f620622671f8 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -81,12 +81,12 @@ qemu-alpine-x86_64-gcc:
   tags:
     - x86_64
 
-qemu-smoke-arm64-gcc:
+qemu-smoke-dom0less-arm64-gcc:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-arm64
     - alpine-3.12-arm64-rootfs-export
@@ -100,12 +100,12 @@ qemu-smoke-arm64-gcc:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-debug:
+qemu-smoke-dom0less-arm64-gcc-debug:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-debug-arm64
     - alpine-3.12-arm64-rootfs-export
@@ -119,12 +119,12 @@ qemu-smoke-arm64-gcc-debug:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-staticmem:
+qemu-smoke-dom0less-arm64-gcc-staticmem:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-arm64-staticmem
     - alpine-3.12-arm64-rootfs-export
@@ -138,12 +138,12 @@ qemu-smoke-arm64-gcc-staticmem:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-debug-staticmem:
+qemu-smoke-dom0less-arm64-gcc-debug-staticmem:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-debug-arm64-staticmem
     - alpine-3.12-arm64-rootfs-export
@@ -157,12 +157,12 @@ qemu-smoke-arm64-gcc-debug-staticmem:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-boot-cpupools:
+qemu-smoke-dom0less-arm64-gcc-boot-cpupools:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-arm64-boot-cpupools
     - alpine-3.12-arm64-rootfs-export
@@ -176,12 +176,12 @@ qemu-smoke-arm64-gcc-boot-cpupools:
   tags:
     - arm64
 
-qemu-smoke-arm64-gcc-debug-boot-cpupools:
+qemu-smoke-dom0less-arm64-gcc-debug-boot-cpupools:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-debug-arm64-boot-cpupools
     - alpine-3.12-arm64-rootfs-export
diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
similarity index 100%
rename from automation/scripts/qemu-smoke-arm64.sh
rename to automation/scripts/qemu-smoke-dom0less-arm64.sh
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* [PATCH v2 09/10] automation: Rename qemu-alpine-arm64.sh to qemu-smoke-dom0-arm64.sh
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
                   ` (7 preceding siblings ...)
  2022-09-26 11:04 ` [PATCH v2 08/10] automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  2022-09-26 11:04 ` [PATCH v2 10/10] automation: Rename qemu-smoke-arm32.sh to qemu-smoke-dom0-arm32.sh Michal Orzel
  9 siblings, 0 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

qemu-alpine-arm64.sh script is used to perform dom0 based testing.
Rename this script to qemu-smoke-dom0-arm64.sh to reflect its usage.
Also rename the corresponding test jobs.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
---
Changes in v2:
- none
---
 automation/gitlab-ci/test.yaml                            | 8 ++++----
 .../{qemu-alpine-arm64.sh => qemu-smoke-dom0-arm64.sh}    | 0
 2 files changed, 4 insertions(+), 4 deletions(-)
 rename automation/scripts/{qemu-alpine-arm64.sh => qemu-smoke-dom0-arm64.sh} (100%)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index f620622671f8..3b147c88ab08 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -25,12 +25,12 @@ build-each-commit-gcc:
   tags:
     - x86_64
 
-qemu-alpine-arm64-gcc:
+qemu-smoke-dom0-arm64-gcc:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-arm64
     - alpine-3.12-arm64-rootfs-export
@@ -44,12 +44,12 @@ qemu-alpine-arm64-gcc:
   tags:
     - arm64
 
-qemu-alpine-arm64-gcc-debug:
+qemu-smoke-dom0-arm64-gcc-debug:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
+    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
   needs:
     - alpine-3.12-gcc-debug-arm64
     - alpine-3.12-arm64-rootfs-export
diff --git a/automation/scripts/qemu-alpine-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
similarity index 100%
rename from automation/scripts/qemu-alpine-arm64.sh
rename to automation/scripts/qemu-smoke-dom0-arm64.sh
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* [PATCH v2 10/10] automation: Rename qemu-smoke-arm32.sh to qemu-smoke-dom0-arm32.sh
  2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
                   ` (8 preceding siblings ...)
  2022-09-26 11:04 ` [PATCH v2 09/10] automation: Rename qemu-alpine-arm64.sh to qemu-smoke-dom0-arm64.sh Michal Orzel
@ 2022-09-26 11:04 ` Michal Orzel
  9 siblings, 0 replies; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 11:04 UTC (permalink / raw)
  To: xen-devel; +Cc: Michal Orzel, Doug Goldstein, Stefano Stabellini

After qemu arm64 test scripts had been renamed to reflect their
usage, do the same for the qemu arm32 test script. Currently it only
boots dom0, so we can assume that this script will be used to perform
dom0 based testing. In the future we will be able to create corresponding
script qemu-smoke-dom0less-arm32.sh to perform dom0less based testing.
This is the last step to get rid of ambiguity with regards to naming
of Arm test scripts.

Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
---
Changes in v2:
- none
---
 automation/gitlab-ci/test.yaml                            | 8 ++++----
 .../{qemu-smoke-arm32.sh => qemu-smoke-dom0-arm32.sh}     | 0
 2 files changed, 4 insertions(+), 4 deletions(-)
 rename automation/scripts/{qemu-smoke-arm32.sh => qemu-smoke-dom0-arm32.sh} (100%)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 3b147c88ab08..92e0a1f7c510 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -195,12 +195,12 @@ qemu-smoke-dom0less-arm64-gcc-debug-boot-cpupools:
   tags:
     - arm64
 
-qemu-smoke-arm32-gcc:
+qemu-smoke-dom0-arm32-gcc:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
+    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
   needs:
     - debian-unstable-gcc-arm32
     - qemu-system-aarch64-6.0.0-arm32-export
@@ -212,12 +212,12 @@ qemu-smoke-arm32-gcc:
   tags:
     - arm64
 
-qemu-smoke-arm32-gcc-debug:
+qemu-smoke-dom0-arm32-gcc-debug:
   extends: .test-jobs-common
   variables:
     CONTAINER: debian:unstable-arm64v8
   script:
-    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
+    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
   needs:
     - debian-unstable-gcc-arm32-debug
     - qemu-system-aarch64-6.0.0-arm32-export
diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-dom0-arm32.sh
similarity index 100%
rename from automation/scripts/qemu-smoke-arm32.sh
rename to automation/scripts/qemu-smoke-dom0-arm32.sh
-- 
2.25.1



^ permalink raw reply related	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 02/10] automation: Add randconfig build jobs for arm64 alpine container
  2022-09-26 11:04 ` [PATCH v2 02/10] automation: Add randconfig build jobs for arm64 alpine container Michal Orzel
@ 2022-09-26 12:00   ` Luca Fancellu
  2022-09-26 22:50   ` Stefano Stabellini
  1 sibling, 0 replies; 25+ messages in thread
From: Luca Fancellu @ 2022-09-26 12:00 UTC (permalink / raw)
  To: Michal Orzel; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini



> On 26 Sep 2022, at 12:04, Michal Orzel <michal.orzel@amd.com> wrote:
> 
> For arm64 we perform builds using debian and alpine containers.
> We are missing the randconfig build jobs for the latter, so add them.
> This way for each container we have 4 fundamental build jobs:
> - defconfig non-debug/debug
> - randconfig non-debug/debug
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Hi Michal,

This looks good to me

Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>





^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 03/10] automation: qemu-smoke-arm64.sh: Increase RAM size
  2022-09-26 11:04 ` [PATCH v2 03/10] automation: qemu-smoke-arm64.sh: Increase RAM size Michal Orzel
@ 2022-09-26 12:02   ` Luca Fancellu
  2022-09-26 22:50   ` Stefano Stabellini
  1 sibling, 0 replies; 25+ messages in thread
From: Luca Fancellu @ 2022-09-26 12:02 UTC (permalink / raw)
  To: Michal Orzel; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini



> On 26 Sep 2022, at 12:04, Michal Orzel <michal.orzel@amd.com> wrote:
> 
> In the follow-up patch we will add new jobs using debug Xen builds.
> Because the debug builds take more space and we might end up in
> a situation when there is not enough free space (especially during
> a static memory test that reserves some region in the middle), increase
> RAM size for QEMU from 1GB to 2GB.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Hi Michal,

This looks good to me

Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>





^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder
  2022-09-26 11:04 ` [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder Michal Orzel
@ 2022-09-26 12:03   ` Luca Fancellu
  2022-09-26 14:46   ` Stewart Hildebrand
  1 sibling, 0 replies; 25+ messages in thread
From: Luca Fancellu @ 2022-09-26 12:03 UTC (permalink / raw)
  To: Michal Orzel
  Cc: Xen development discussion, Doug Goldstein, Stefano Stabellini



> On 26 Sep 2022, at 12:04, Michal Orzel <michal.orzel@amd.com> wrote:
> 
> Take an example from arm64 qemu test scripts and use ImageBuilder
> to generate u-boot script automatically. Calculating the addresses
> manually is quite error prone and also we will be able to benefit
> from using ImageBuilder when adding domUs to this test in the future.
> 
> Install and use u-boot from the debian package.
> Modify the script so that binaries are loaded from u-boot via tftp.
> Remove dtuart=/pl011@9000000 because stdout-path in QEMU dtb is always
> set to pl011 node path.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> ---
> Changes in v2:
> - add explanation in commit msg why we remove dtuart path
> ---

Hi Michal,

This looks good to me

Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>





^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed
  2022-09-26 11:04 ` [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed Michal Orzel
@ 2022-09-26 12:03   ` Luca Fancellu
  2022-09-26 14:42   ` Ayan Kumar Halder
  2022-09-26 22:54   ` Stefano Stabellini
  2 siblings, 0 replies; 25+ messages in thread
From: Luca Fancellu @ 2022-09-26 12:03 UTC (permalink / raw)
  To: Michal Orzel
  Cc: Xen development discussion, Doug Goldstein, Stefano Stabellini



> On 26 Sep 2022, at 12:04, Michal Orzel <michal.orzel@amd.com> wrote:
> 
> Currently, all the arm64 defconfig build jobs, regardless of the
> container used, end up building Xen with the extra config options
> specified in the main build script (e.g. CONFIG_EXPERT,
> CONFIG_STATIC_MEMORY). Because these options are only needed for
> specific test jobs, the current behavior of the CI is incorrect
> as we add the extra options to all the defconfig builds. This means
> that on arm64 there is not a single job performing proper defconfig build.
> 
> To fix this issue, add custom build jobs each time there is a need for
> building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
> variable to be used by these jobs to store the required options. This
> variable will be then read by the main build script to modify the .config
> file. This will also help users to understand what is needed to run specific
> test.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> ---
> This patch is supposed to be merged for 4.17. The release manager agreed
> on that.
> 
> Changes in v2:
> - replace [ with [[ to be consistent with other ifs in build script
> ---

Hi Michal,

This looks good to me

Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>





^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 04/10] automation: Add debug versions of Arm tests
  2022-09-26 11:04 ` [PATCH v2 04/10] automation: Add debug versions of Arm tests Michal Orzel
@ 2022-09-26 12:06   ` Luca Fancellu
  2022-09-26 22:50   ` Stefano Stabellini
  1 sibling, 0 replies; 25+ messages in thread
From: Luca Fancellu @ 2022-09-26 12:06 UTC (permalink / raw)
  To: Michal Orzel
  Cc: Xen development discussion, Doug Goldstein, Stefano Stabellini



> On 26 Sep 2022, at 12:04, Michal Orzel <michal.orzel@amd.com> wrote:
> 
> At the moment, all the tests are executed on non-debug Xen builds.
> To improve the coverage (e.g. we might catch some asserts), add new
> test jobs using debug Xen builds.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> ---
> Changes in v2:
> - carve out the RAM size change to a separate patch [v2 03/10]
> ---

Hi Michal,

This looks good to me

Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>





^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed
  2022-09-26 11:04 ` [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed Michal Orzel
  2022-09-26 12:03   ` Luca Fancellu
@ 2022-09-26 14:42   ` Ayan Kumar Halder
  2022-09-26 22:54   ` Stefano Stabellini
  2 siblings, 0 replies; 25+ messages in thread
From: Ayan Kumar Halder @ 2022-09-26 14:42 UTC (permalink / raw)
  To: xen-devel


On 26/09/2022 12:04, Michal Orzel wrote:
> Currently, all the arm64 defconfig build jobs, regardless of the
> container used, end up building Xen with the extra config options
> specified in the main build script (e.g. CONFIG_EXPERT,
> CONFIG_STATIC_MEMORY). Because these options are only needed for
> specific test jobs, the current behavior of the CI is incorrect
> as we add the extra options to all the defconfig builds. This means
> that on arm64 there is not a single job performing proper defconfig build.
>
> To fix this issue, add custom build jobs each time there is a need for
> building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
> variable to be used by these jobs to store the required options. This
> variable will be then read by the main build script to modify the .config
> file. This will also help users to understand what is needed to run specific
> test.
>
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
Reviewed-by: Ayan Kumar halder <ayankuma@amd.com>
> ---
> This patch is supposed to be merged for 4.17. The release manager agreed
> on that.
>
> Changes in v2:
> - replace [ with [[ to be consistent with other ifs in build script
> ---
>   automation/gitlab-ci/build.yaml | 15 +++++++++++++++
>   automation/gitlab-ci/test.yaml  |  4 ++--
>   automation/scripts/build        |  8 ++------
>   3 files changed, 19 insertions(+), 8 deletions(-)
>
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 720ce6e07ba0..a39ed72aac6d 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -566,6 +566,21 @@ alpine-3.12-gcc-debug-arm64:
>     variables:
>       CONTAINER: alpine:3.12-arm64v8
>   
> +alpine-3.12-gcc-arm64-staticmem:
> +  extends: .gcc-arm64-build
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_EXPERT=y
> +      CONFIG_UNSUPPORTED=y
> +      CONFIG_STATIC_MEMORY=y
> +
> +alpine-3.12-gcc-arm64-boot-cpupools:
> +  extends: .gcc-arm64-build
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_BOOT_TIME_CPUPOOLS=y
>   
>   ## Test artifacts common
>   
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index d899b3bdbf7a..4f96e6e322de 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -88,7 +88,7 @@ qemu-smoke-arm64-gcc-staticmem:
>     script:
>       - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
>     needs:
> -    - alpine-3.12-gcc-arm64
> +    - alpine-3.12-gcc-arm64-staticmem
>       - alpine-3.12-arm64-rootfs-export
>       - kernel-5.19-arm64-export
>       - qemu-system-aarch64-6.0.0-arm64-export
> @@ -107,7 +107,7 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>     script:
>       - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
>     needs:
> -    - alpine-3.12-gcc-arm64
> +    - alpine-3.12-gcc-arm64-boot-cpupools
>       - alpine-3.12-arm64-rootfs-export
>       - kernel-5.19-arm64-export
>       - qemu-system-aarch64-6.0.0-arm64-export
> diff --git a/automation/scripts/build b/automation/scripts/build
> index 2f15ab3198e6..bcfa6838f0bb 100755
> --- a/automation/scripts/build
> +++ b/automation/scripts/build
> @@ -15,12 +15,8 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
>       make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
>       hypervisor_only="y"
>   else
> -    if [[ "${XEN_TARGET_ARCH}" = "arm64" ]]; then
> -        echo "
> -CONFIG_EXPERT=y
> -CONFIG_UNSUPPORTED=y
> -CONFIG_STATIC_MEMORY=y
> -CONFIG_BOOT_TIME_CPUPOOLS=y" > xen/.config
> +    if [[ -n "${EXTRA_XEN_CONFIG}" ]]; then
> +        echo "${EXTRA_XEN_CONFIG}" > xen/.config
>           make -j$(nproc) -C xen olddefconfig
>       else
>           make -j$(nproc) -C xen defconfig


^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder
  2022-09-26 11:04 ` [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder Michal Orzel
  2022-09-26 12:03   ` Luca Fancellu
@ 2022-09-26 14:46   ` Stewart Hildebrand
  2022-09-26 14:59     ` Michal Orzel
  1 sibling, 1 reply; 25+ messages in thread
From: Stewart Hildebrand @ 2022-09-26 14:46 UTC (permalink / raw)
  To: Michal Orzel, xen-devel; +Cc: Doug Goldstein, Stefano Stabellini

On 9/26/22 07:04, Michal Orzel wrote:
> Take an example from arm64 qemu test scripts and use ImageBuilder
> to generate u-boot script automatically. Calculating the addresses
> manually is quite error prone and also we will be able to benefit
> from using ImageBuilder when adding domUs to this test in the future.
> 
> Install and use u-boot from the debian package.
> Modify the script so that binaries are loaded from u-boot via tftp.
> Remove dtuart=/pl011@9000000 because stdout-path in QEMU dtb is always
> set to pl011 node path.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> ---
> Changes in v2:
> - add explanation in commit msg why we remove dtuart path
> ---
>   automation/scripts/qemu-smoke-arm32.sh | 57 ++++++++++++--------------
>   1 file changed, 27 insertions(+), 30 deletions(-)
> 
> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
> index 530f3892fdd3..765facbe4d66 100755
> --- a/automation/scripts/qemu-smoke-arm32.sh
> +++ b/automation/scripts/qemu-smoke-arm32.sh
> @@ -4,7 +4,9 @@ set -ex
>   
>   export DEBIAN_FRONTENT=noninteractive

While you are here, there's a typo. s/DEBIAN_FRONTENT/DEBIAN_FRONTEND/


^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder
  2022-09-26 14:46   ` Stewart Hildebrand
@ 2022-09-26 14:59     ` Michal Orzel
  2022-09-26 22:52       ` Stefano Stabellini
  0 siblings, 1 reply; 25+ messages in thread
From: Michal Orzel @ 2022-09-26 14:59 UTC (permalink / raw)
  To: Stewart Hildebrand, xen-devel; +Cc: Doug Goldstein, Stefano Stabellini

Hi Stewart,

On 26/09/2022 16:46, Stewart Hildebrand wrote:
> 
> 
> On 9/26/22 07:04, Michal Orzel wrote:
>> Take an example from arm64 qemu test scripts and use ImageBuilder
>> to generate u-boot script automatically. Calculating the addresses
>> manually is quite error prone and also we will be able to benefit
>> from using ImageBuilder when adding domUs to this test in the future.
>>
>> Install and use u-boot from the debian package.
>> Modify the script so that binaries are loaded from u-boot via tftp.
>> Remove dtuart=/pl011@9000000 because stdout-path in QEMU dtb is always
>> set to pl011 node path.
>>
>> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
>> ---
>> Changes in v2:
>> - add explanation in commit msg why we remove dtuart path
>> ---
>>   automation/scripts/qemu-smoke-arm32.sh | 57 ++++++++++++--------------
>>   1 file changed, 27 insertions(+), 30 deletions(-)
>>
>> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
>> index 530f3892fdd3..765facbe4d66 100755
>> --- a/automation/scripts/qemu-smoke-arm32.sh
>> +++ b/automation/scripts/qemu-smoke-arm32.sh
>> @@ -4,7 +4,9 @@ set -ex
>>
>>   export DEBIAN_FRONTENT=noninteractive
> 
> While you are here, there's a typo. s/DEBIAN_FRONTENT/DEBIAN_FRONTEND/

This is a good catch. Thanks!
It looks like all the Arm scripts use FRONTENT instead of FRONTEND. This is clearly incorrect.
I will need to fix all the occurrences as part of this cleanup series.

~Michal


^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 08/10] automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh
  2022-09-26 11:04 ` [PATCH v2 08/10] automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh Michal Orzel
@ 2022-09-26 15:12   ` Andrew Cooper
  0 siblings, 0 replies; 25+ messages in thread
From: Andrew Cooper @ 2022-09-26 15:12 UTC (permalink / raw)
  To: Michal Orzel, xen-devel; +Cc: Doug Goldstein, Stefano Stabellini

On 26/09/2022 12:04, Michal Orzel wrote:
> Testing arm64 is done using the qemu-alpine-arm64.sh and
> qemu-smoke-arm64.sh scripts. These scripts are executed with exactly
> the same artifacts (container, rootfs, kernel, qemu) and the only
> difference is that the former is used to perform dom0 based testing
> and the latter - dom0less based testing.
>
> Because the current naming is quite umbiguous, rename qemu-smoke-arm64.sh

"ambiguous"

~Andrew

^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 02/10] automation: Add randconfig build jobs for arm64 alpine container
  2022-09-26 11:04 ` [PATCH v2 02/10] automation: Add randconfig build jobs for arm64 alpine container Michal Orzel
  2022-09-26 12:00   ` Luca Fancellu
@ 2022-09-26 22:50   ` Stefano Stabellini
  1 sibling, 0 replies; 25+ messages in thread
From: Stefano Stabellini @ 2022-09-26 22:50 UTC (permalink / raw)
  To: Michal Orzel; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini

On Mon, 26 Sep 2022, Michal Orzel wrote:
> For arm64 we perform builds using debian and alpine containers.
> We are missing the randconfig build jobs for the latter, so add them.
> This way for each container we have 4 fundamental build jobs:
> - defconfig non-debug/debug
> - randconfig non-debug/debug
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
> Changes in v2:
> - none
> ---
>  automation/gitlab-ci/build.yaml | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index a39ed72aac6d..d931441b910a 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -566,6 +566,18 @@ alpine-3.12-gcc-debug-arm64:
>    variables:
>      CONTAINER: alpine:3.12-arm64v8
>  
> +alpine-3.12-gcc-arm64-randconfig:
> +  extends: .gcc-arm64-build
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    RANDCONFIG: y
> +
> +alpine-3.12-gcc-debug-arm64-randconfig:
> +  extends: .gcc-arm64-build-debug
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    RANDCONFIG: y
> +
>  alpine-3.12-gcc-arm64-staticmem:
>    extends: .gcc-arm64-build
>    variables:
> -- 
> 2.25.1
> 


^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 03/10] automation: qemu-smoke-arm64.sh: Increase RAM size
  2022-09-26 11:04 ` [PATCH v2 03/10] automation: qemu-smoke-arm64.sh: Increase RAM size Michal Orzel
  2022-09-26 12:02   ` Luca Fancellu
@ 2022-09-26 22:50   ` Stefano Stabellini
  1 sibling, 0 replies; 25+ messages in thread
From: Stefano Stabellini @ 2022-09-26 22:50 UTC (permalink / raw)
  To: Michal Orzel; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini

On Mon, 26 Sep 2022, Michal Orzel wrote:
> In the follow-up patch we will add new jobs using debug Xen builds.
> Because the debug builds take more space and we might end up in
> a situation when there is not enough free space (especially during
> a static memory test that reserves some region in the middle), increase
> RAM size for QEMU from 1GB to 2GB.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
> Changes in v2:
> - new patch carved out from previous v1 [3/9] patch
> ---
>  automation/scripts/qemu-smoke-arm64.sh | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-arm64.sh
> index f803835779f4..dea26c6c0a8f 100755
> --- a/automation/scripts/qemu-smoke-arm64.sh
> +++ b/automation/scripts/qemu-smoke-arm64.sh
> @@ -52,7 +52,7 @@ curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
>  ./binaries/qemu-system-aarch64 \
>     -machine virtualization=true \
>     -cpu cortex-a57 -machine type=virt \
> -   -m 1024 -smp 2 -display none \
> +   -m 2048 -smp 2 -display none \
>     -machine dumpdtb=binaries/virt-gicv2.dtb
>  
>  # XXX disable pl061 to avoid Linux crash
> @@ -117,7 +117,7 @@ cd ..
>  
>  # ImageBuilder
>  echo 'MEMORY_START="0x40000000"
> -MEMORY_END="0x80000000"
> +MEMORY_END="0xC0000000"
>  
>  DEVICE_TREE="virt-gicv2.dtb"
>  XEN="xen"
> @@ -158,7 +158,7 @@ timeout -k 1 240 \
>  ./binaries/qemu-system-aarch64 \
>      -machine virtualization=true \
>      -cpu cortex-a57 -machine type=virt \
> -    -m 1024 -monitor none -serial stdio \
> +    -m 2048 -monitor none -serial stdio \
>      -smp 2 \
>      -no-reboot \
>      -device virtio-net-pci,netdev=n0 \
> -- 
> 2.25.1
> 


^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 04/10] automation: Add debug versions of Arm tests
  2022-09-26 11:04 ` [PATCH v2 04/10] automation: Add debug versions of Arm tests Michal Orzel
  2022-09-26 12:06   ` Luca Fancellu
@ 2022-09-26 22:50   ` Stefano Stabellini
  1 sibling, 0 replies; 25+ messages in thread
From: Stefano Stabellini @ 2022-09-26 22:50 UTC (permalink / raw)
  To: Michal Orzel; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini

On Mon, 26 Sep 2022, Michal Orzel wrote:
> At the moment, all the tests are executed on non-debug Xen builds.
> To improve the coverage (e.g. we might catch some asserts), add new
> test jobs using debug Xen builds.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>



> ---
> Changes in v2:
> - carve out the RAM size change to a separate patch [v2 03/10]
> ---
>  automation/gitlab-ci/build.yaml | 16 ++++++
>  automation/gitlab-ci/test.yaml  | 93 +++++++++++++++++++++++++++++++++
>  2 files changed, 109 insertions(+)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index d931441b910a..6ce92531f61f 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -587,6 +587,15 @@ alpine-3.12-gcc-arm64-staticmem:
>        CONFIG_UNSUPPORTED=y
>        CONFIG_STATIC_MEMORY=y
>  
> +alpine-3.12-gcc-debug-arm64-staticmem:
> +  extends: .gcc-arm64-build-debug
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_EXPERT=y
> +      CONFIG_UNSUPPORTED=y
> +      CONFIG_STATIC_MEMORY=y
> +
>  alpine-3.12-gcc-arm64-boot-cpupools:
>    extends: .gcc-arm64-build
>    variables:
> @@ -594,6 +603,13 @@ alpine-3.12-gcc-arm64-boot-cpupools:
>      EXTRA_XEN_CONFIG: |
>        CONFIG_BOOT_TIME_CPUPOOLS=y
>  
> +alpine-3.12-gcc-debug-arm64-boot-cpupools:
> +  extends: .gcc-arm64-build-debug
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_BOOT_TIME_CPUPOOLS=y
> +
>  ## Test artifacts common
>  
>  .test-jobs-artifact-common:
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 4f96e6e322de..1b51030c6175 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -44,6 +44,25 @@ qemu-alpine-arm64-gcc:
>    tags:
>      - arm64
>  
> +qemu-alpine-arm64-gcc-debug:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-alpine-x86_64-gcc:
>    extends: .test-jobs-common
>    variables:
> @@ -81,6 +100,25 @@ qemu-smoke-arm64-gcc:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm64-gcc-debug:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-arm64-gcc-staticmem:
>    extends: .test-jobs-common
>    variables:
> @@ -100,6 +138,25 @@ qemu-smoke-arm64-gcc-staticmem:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm64-gcc-debug-staticmem:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64-staticmem
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-arm64-gcc-boot-cpupools:
>    extends: .test-jobs-common
>    variables:
> @@ -119,6 +176,25 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm64-gcc-debug-boot-cpupools:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
> +  needs:
> +    - alpine-3.12-gcc-debug-arm64-boot-cpupools
> +    - alpine-3.12-arm64-rootfs-export
> +    - kernel-5.19-arm64-export
> +    - qemu-system-aarch64-6.0.0-arm64-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-arm32-gcc:
>    extends: .test-jobs-common
>    variables:
> @@ -136,6 +212,23 @@ qemu-smoke-arm32-gcc:
>    tags:
>      - arm64
>  
> +qemu-smoke-arm32-gcc-debug:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: debian:unstable-arm64v8
> +  script:
> +    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
> +  needs:
> +    - debian-unstable-gcc-arm32-debug
> +    - qemu-system-aarch64-6.0.0-arm32-export
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - arm64
> +
>  qemu-smoke-x86-64-gcc:
>    extends: .test-jobs-common
>    variables:
> -- 
> 2.25.1
> 


^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder
  2022-09-26 14:59     ` Michal Orzel
@ 2022-09-26 22:52       ` Stefano Stabellini
  0 siblings, 0 replies; 25+ messages in thread
From: Stefano Stabellini @ 2022-09-26 22:52 UTC (permalink / raw)
  To: Michal Orzel
  Cc: Stewart Hildebrand, xen-devel, Doug Goldstein, Stefano Stabellini

On Mon, 26 Sep 2022, Michal Orzel wrote:
> Hi Stewart,
> 
> On 26/09/2022 16:46, Stewart Hildebrand wrote:
> > 
> > 
> > On 9/26/22 07:04, Michal Orzel wrote:
> >> Take an example from arm64 qemu test scripts and use ImageBuilder
> >> to generate u-boot script automatically. Calculating the addresses
> >> manually is quite error prone and also we will be able to benefit
> >> from using ImageBuilder when adding domUs to this test in the future.
> >>
> >> Install and use u-boot from the debian package.
> >> Modify the script so that binaries are loaded from u-boot via tftp.
> >> Remove dtuart=/pl011@9000000 because stdout-path in QEMU dtb is always
> >> set to pl011 node path.
> >>
> >> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
> >> ---
> >> Changes in v2:
> >> - add explanation in commit msg why we remove dtuart path
> >> ---
> >>   automation/scripts/qemu-smoke-arm32.sh | 57 ++++++++++++--------------
> >>   1 file changed, 27 insertions(+), 30 deletions(-)
> >>
> >> diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
> >> index 530f3892fdd3..765facbe4d66 100755
> >> --- a/automation/scripts/qemu-smoke-arm32.sh
> >> +++ b/automation/scripts/qemu-smoke-arm32.sh
> >> @@ -4,7 +4,9 @@ set -ex
> >>
> >>   export DEBIAN_FRONTENT=noninteractive
> > 
> > While you are here, there's a typo. s/DEBIAN_FRONTENT/DEBIAN_FRONTEND/
> 
> This is a good catch. Thanks!
> It looks like all the Arm scripts use FRONTENT instead of FRONTEND. This is clearly incorrect.
> I will need to fix all the occurrences as part of this cleanup series.

For this specific patch, given that it is not changing DEBIAN_FRONTENT:

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

Please send a separate patch to address the DEBIAN_FRONTENT issue, and I
think that should target 4.17 (so it should be the first patch of the
series next time.)


^ permalink raw reply	[flat|nested] 25+ messages in thread
* Re: [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed
  2022-09-26 11:04 ` [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed Michal Orzel
  2022-09-26 12:03   ` Luca Fancellu
  2022-09-26 14:42   ` Ayan Kumar Halder
@ 2022-09-26 22:54   ` Stefano Stabellini
  2 siblings, 0 replies; 25+ messages in thread
From: Stefano Stabellini @ 2022-09-26 22:54 UTC (permalink / raw)
  To: Michal Orzel; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini

On Mon, 26 Sep 2022, Michal Orzel wrote:
> Currently, all the arm64 defconfig build jobs, regardless of the
> container used, end up building Xen with the extra config options
> specified in the main build script (e.g. CONFIG_EXPERT,
> CONFIG_STATIC_MEMORY). Because these options are only needed for
> specific test jobs, the current behavior of the CI is incorrect
> as we add the extra options to all the defconfig builds. This means
> that on arm64 there is not a single job performing proper defconfig build.
> 
> To fix this issue, add custom build jobs each time there is a need for
> building Xen with additional config options. Introduce EXTRA_XEN_CONFIG
> variable to be used by these jobs to store the required options. This
> variable will be then read by the main build script to modify the .config
> file. This will also help users to understand what is needed to run specific
> test.
> 
> Signed-off-by: Michal Orzel <michal.orzel@amd.com>


Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

and committed to staging given Henry's previous ack.


> ---
> This patch is supposed to be merged for 4.17. The release manager agreed
> on that.
> 
> Changes in v2:
> - replace [ with [[ to be consistent with other ifs in build script
> ---
>  automation/gitlab-ci/build.yaml | 15 +++++++++++++++
>  automation/gitlab-ci/test.yaml  |  4 ++--
>  automation/scripts/build        |  8 ++------
>  3 files changed, 19 insertions(+), 8 deletions(-)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 720ce6e07ba0..a39ed72aac6d 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -566,6 +566,21 @@ alpine-3.12-gcc-debug-arm64:
>    variables:
>      CONTAINER: alpine:3.12-arm64v8
>  
> +alpine-3.12-gcc-arm64-staticmem:
> +  extends: .gcc-arm64-build
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_EXPERT=y
> +      CONFIG_UNSUPPORTED=y
> +      CONFIG_STATIC_MEMORY=y
> +
> +alpine-3.12-gcc-arm64-boot-cpupools:
> +  extends: .gcc-arm64-build
> +  variables:
> +    CONTAINER: alpine:3.12-arm64v8
> +    EXTRA_XEN_CONFIG: |
> +      CONFIG_BOOT_TIME_CPUPOOLS=y
>  
>  ## Test artifacts common
>  
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index d899b3bdbf7a..4f96e6e322de 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -88,7 +88,7 @@ qemu-smoke-arm64-gcc-staticmem:
>    script:
>      - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
>    needs:
> -    - alpine-3.12-gcc-arm64
> +    - alpine-3.12-gcc-arm64-staticmem
>      - alpine-3.12-arm64-rootfs-export
>      - kernel-5.19-arm64-export
>      - qemu-system-aarch64-6.0.0-arm64-export
> @@ -107,7 +107,7 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>    script:
>      - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
>    needs:
> -    - alpine-3.12-gcc-arm64
> +    - alpine-3.12-gcc-arm64-boot-cpupools
>      - alpine-3.12-arm64-rootfs-export
>      - kernel-5.19-arm64-export
>      - qemu-system-aarch64-6.0.0-arm64-export
> diff --git a/automation/scripts/build b/automation/scripts/build
> index 2f15ab3198e6..bcfa6838f0bb 100755
> --- a/automation/scripts/build
> +++ b/automation/scripts/build
> @@ -15,12 +15,8 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
>      make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
>      hypervisor_only="y"
>  else
> -    if [[ "${XEN_TARGET_ARCH}" = "arm64" ]]; then
> -        echo "
> -CONFIG_EXPERT=y
> -CONFIG_UNSUPPORTED=y
> -CONFIG_STATIC_MEMORY=y
> -CONFIG_BOOT_TIME_CPUPOOLS=y" > xen/.config
> +    if [[ -n "${EXTRA_XEN_CONFIG}" ]]; then
> +        echo "${EXTRA_XEN_CONFIG}" > xen/.config
>          make -j$(nproc) -C xen olddefconfig
>      else
>          make -j$(nproc) -C xen defconfig
> -- 
> 2.25.1
> 


^ permalink raw reply	[flat|nested] 25+ messages in thread
end of thread, other threads:[~2022-09-26 22:55 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-09-26 11:04 [PATCH v2 00/10] GitLab CI cleanup & improvements for Arm Michal Orzel
2022-09-26 11:04 ` [PATCH v2 01/10] automation: Use custom build jobs when extra config options are needed Michal Orzel
2022-09-26 12:03   ` Luca Fancellu
2022-09-26 14:42   ` Ayan Kumar Halder
2022-09-26 22:54   ` Stefano Stabellini
2022-09-26 11:04 ` [PATCH v2 02/10] automation: Add randconfig build jobs for arm64 alpine container Michal Orzel
2022-09-26 12:00   ` Luca Fancellu
2022-09-26 22:50   ` Stefano Stabellini
2022-09-26 11:04 ` [PATCH v2 03/10] automation: qemu-smoke-arm64.sh: Increase RAM size Michal Orzel
2022-09-26 12:02   ` Luca Fancellu
2022-09-26 22:50   ` Stefano Stabellini
2022-09-26 11:04 ` [PATCH v2 04/10] automation: Add debug versions of Arm tests Michal Orzel
2022-09-26 12:06   ` Luca Fancellu
2022-09-26 22:50   ` Stefano Stabellini
2022-09-26 11:04 ` [PATCH v2 05/10] automation: Add Arm containers to containerize script Michal Orzel
2022-09-26 11:04 ` [PATCH v2 06/10] automation: qemu-smoke-arm32.sh: Modify script to use ImageBuilder Michal Orzel
2022-09-26 12:03   ` Luca Fancellu
2022-09-26 14:46   ` Stewart Hildebrand
2022-09-26 14:59     ` Michal Orzel
2022-09-26 22:52       ` Stefano Stabellini
2022-09-26 11:04 ` [PATCH v2 07/10] automation: qemu-alpine-arm64: Cleanup and fixes Michal Orzel
2022-09-26 11:04 ` [PATCH v2 08/10] automation: Rename qemu-smoke-arm64.sh to qemu-smoke-dom0less-arm64.sh Michal Orzel
2022-09-26 15:12   ` Andrew Cooper
2022-09-26 11:04 ` [PATCH v2 09/10] automation: Rename qemu-alpine-arm64.sh to qemu-smoke-dom0-arm64.sh Michal Orzel
2022-09-26 11:04 ` [PATCH v2 10/10] automation: Rename qemu-smoke-arm32.sh to qemu-smoke-dom0-arm32.sh Michal Orzel

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.