Re: [PATCH 3/9] automation: Add debug versions of Arm tests

[Michal Orzel <michal.orzel@amd.com>]

Hi Stefano,

On 22/09/2022 23:53, Stefano Stabellini wrote:
> 
> 
> On Thu, 22 Sep 2022, Michal Orzel wrote:
>> At the moment, all the tests are executed on non-debug Xen builds.
>> To improve the coverage (e.g. we might catch some asserts), add new
>> test jobs using debug Xen builds.
> 
> This is interesting. On one hand, I agree it is good to test DEBUG and
> non-DEBUG builds in runtime tests too, on the other hand I am worried
> about our test capacity.

We still have plenty of capacity. We can add many more Arm tests because
Arm machine is very powerful. I reckon that adding more x86 tests would
have a performance impact but not the other way around. See below.

> 
> I take you have tried a few pipelines -- are the additional tests
> causing the pipeline to take longer? Do we still have "room" because x86
> is slower? What's your thinking?
> 

With the additional tests introduced by this patch one cannot observe any
impact for the performance. I ran several tests and the pipeline finishes
in the same time or faster.

Performance comparison:
current master: 92 jobs (82 minutes, 19 seconds)
master + my series: 103 jobs (79 minutes, 40 seconds)

That said, let's improve the coverage and add these non-debug tests for Arm.

> One idea would be to only do:
> - qemu-alpine-arm64-gcc-debug
> - qemu-smoke-arm64-gcc-debug
> 
> and not the others
> 
> 
>> Take the opportunity to increase RAM size for QEMU from 1GB to 2GB
>> in qemu-smoke-arm64.sh as the debug builds take more space and we might
>> end up in a situation when there is not enough free space (especially
>> during a static memory test that reserves some region in the middle).
>>
>> Signed-off-by: Michal Orzel <michal.orzel@amd.com>
>> ---
>>  automation/gitlab-ci/build.yaml        | 16 +++++
>>  automation/gitlab-ci/test.yaml         | 93 ++++++++++++++++++++++++++
>>  automation/scripts/qemu-smoke-arm64.sh |  6 +-
>>  3 files changed, 112 insertions(+), 3 deletions(-)
>>
>> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
>> index d931441b910a..6ce92531f61f 100644
>> --- a/automation/gitlab-ci/build.yaml
>> +++ b/automation/gitlab-ci/build.yaml
>> @@ -587,6 +587,15 @@ alpine-3.12-gcc-arm64-staticmem:
>>        CONFIG_UNSUPPORTED=y
>>        CONFIG_STATIC_MEMORY=y
>>
>> +alpine-3.12-gcc-debug-arm64-staticmem:
>> +  extends: .gcc-arm64-build-debug
>> +  variables:
>> +    CONTAINER: alpine:3.12-arm64v8
>> +    EXTRA_XEN_CONFIG: |
>> +      CONFIG_EXPERT=y
>> +      CONFIG_UNSUPPORTED=y
>> +      CONFIG_STATIC_MEMORY=y
>> +
>>  alpine-3.12-gcc-arm64-boot-cpupools:
>>    extends: .gcc-arm64-build
>>    variables:
>> @@ -594,6 +603,13 @@ alpine-3.12-gcc-arm64-boot-cpupools:
>>      EXTRA_XEN_CONFIG: |
>>        CONFIG_BOOT_TIME_CPUPOOLS=y
>>
>> +alpine-3.12-gcc-debug-arm64-boot-cpupools:
>> +  extends: .gcc-arm64-build-debug
>> +  variables:
>> +    CONTAINER: alpine:3.12-arm64v8
>> +    EXTRA_XEN_CONFIG: |
>> +      CONFIG_BOOT_TIME_CPUPOOLS=y
>> +
>>  ## Test artifacts common
>>
>>  .test-jobs-artifact-common:
>> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
>> index 4f96e6e322de..1b51030c6175 100644
>> --- a/automation/gitlab-ci/test.yaml
>> +++ b/automation/gitlab-ci/test.yaml
>> @@ -44,6 +44,25 @@ qemu-alpine-arm64-gcc:
>>    tags:
>>      - arm64
>>
>> +qemu-alpine-arm64-gcc-debug:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
>> +  needs:
>> +    - alpine-3.12-gcc-debug-arm64
>> +    - alpine-3.12-arm64-rootfs-export
>> +    - kernel-5.19-arm64-export
>> +    - qemu-system-aarch64-6.0.0-arm64-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-alpine-x86_64-gcc:
>>    extends: .test-jobs-common
>>    variables:
>> @@ -81,6 +100,25 @@ qemu-smoke-arm64-gcc:
>>    tags:
>>      - arm64
>>
>> +qemu-smoke-arm64-gcc-debug:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee qemu-smoke-arm64.log
>> +  needs:
>> +    - alpine-3.12-gcc-debug-arm64
>> +    - alpine-3.12-arm64-rootfs-export
>> +    - kernel-5.19-arm64-export
>> +    - qemu-system-aarch64-6.0.0-arm64-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-smoke-arm64-gcc-staticmem:
>>    extends: .test-jobs-common
>>    variables:
>> @@ -100,6 +138,25 @@ qemu-smoke-arm64-gcc-staticmem:
>>    tags:
>>      - arm64
>>
>> +qemu-smoke-arm64-gcc-debug-staticmem:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee qemu-smoke-arm64.log
>> +  needs:
>> +    - alpine-3.12-gcc-debug-arm64-staticmem
>> +    - alpine-3.12-arm64-rootfs-export
>> +    - kernel-5.19-arm64-export
>> +    - qemu-system-aarch64-6.0.0-arm64-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-smoke-arm64-gcc-boot-cpupools:
>>    extends: .test-jobs-common
>>    variables:
>> @@ -119,6 +176,25 @@ qemu-smoke-arm64-gcc-boot-cpupools:
>>    tags:
>>      - arm64
>>
>> +qemu-smoke-arm64-gcc-debug-boot-cpupools:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee qemu-smoke-arm64.log
>> +  needs:
>> +    - alpine-3.12-gcc-debug-arm64-boot-cpupools
>> +    - alpine-3.12-arm64-rootfs-export
>> +    - kernel-5.19-arm64-export
>> +    - qemu-system-aarch64-6.0.0-arm64-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-smoke-arm32-gcc:
>>    extends: .test-jobs-common
>>    variables:
>> @@ -136,6 +212,23 @@ qemu-smoke-arm32-gcc:
>>    tags:
>>      - arm64
>>
>> +qemu-smoke-arm32-gcc-debug:
>> +  extends: .test-jobs-common
>> +  variables:
>> +    CONTAINER: debian:unstable-arm64v8
>> +  script:
>> +    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
>> +  needs:
>> +    - debian-unstable-gcc-arm32-debug
>> +    - qemu-system-aarch64-6.0.0-arm32-export
>> +  artifacts:
>> +    paths:
>> +      - smoke.serial
>> +      - '*.log'
>> +    when: always
>> +  tags:
>> +    - arm64
>> +
>>  qemu-smoke-x86-64-gcc:
>>    extends: .test-jobs-common
>>    variables:
>> diff --git a/automation/scripts/qemu-smoke-arm64.sh b/automation/scripts/qemu-smoke-arm64.sh
>> index f803835779f4..dea26c6c0a8f 100755
>> --- a/automation/scripts/qemu-smoke-arm64.sh
>> +++ b/automation/scripts/qemu-smoke-arm64.sh
>> @@ -52,7 +52,7 @@ curl -fsSLO https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fraw%2Fv5.2.0%2Fpc-bios%2Fefi-virtio.rom&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cf6b0b8fabb02478b009c08da9ce4eac9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994804254367719%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=jgGdO93kLoieg6UqajFWYAsufvvv5mZVYHD%2B7fUW7ME%3D&amp;reserved=0
>>  ./binaries/qemu-system-aarch64 \
>>     -machine virtualization=true \
>>     -cpu cortex-a57 -machine type=virt \
>> -   -m 1024 -smp 2 -display none \
>> +   -m 2048 -smp 2 -display none \
>>     -machine dumpdtb=binaries/virt-gicv2.dtb
>>
>>  # XXX disable pl061 to avoid Linux crash
>> @@ -117,7 +117,7 @@ cd ..
>>
>>  # ImageBuilder
>>  echo 'MEMORY_START="0x40000000"
>> -MEMORY_END="0x80000000"
>> +MEMORY_END="0xC0000000"
>>
>>  DEVICE_TREE="virt-gicv2.dtb"
>>  XEN="xen"
>> @@ -158,7 +158,7 @@ timeout -k 1 240 \
>>  ./binaries/qemu-system-aarch64 \
>>      -machine virtualization=true \
>>      -cpu cortex-a57 -machine type=virt \
>> -    -m 1024 -monitor none -serial stdio \
>> +    -m 2048 -monitor none -serial stdio \
>>      -smp 2 \
>>      -no-reboot \
>>      -device virtio-net-pci,netdev=n0 \
> 
> The changes to qemu-smoke-arm64.sh make sense, but could you move them
> to a separate patch in case we need to do any backports?

Ok, I will do a prerequisite patch for that one.

> 
> Also I would change MEMORY_END to 0xC0000000 in qemu-alpine-arm64.sh too
> for uniformity.

I already did that in patch [6/9] automation: qemu-alpine-arm64: Cleanup and fixes.

~Michal