Re: [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml

[Thomas Huth <thuth@redhat.com>]

On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
> Extract the build/test jobs run by default on the mainstream
> CI into a new file (buildtest.yml) and their templates into
> (buildtest-template.yml), to be able to reuse the templates
> without having to run all these mainstream jobs by default.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml |  80 +++
>   .gitlab-ci.d/buildtest.yml          | 744 +++++++++++++++++++++++++
>   .gitlab-ci.yml                      | 824 +---------------------------
>   3 files changed, 825 insertions(+), 823 deletions(-)
>   create mode 100644 .gitlab-ci.d/buildtest-template.yml
>   create mode 100644 .gitlab-ci.d/buildtest.yml

I like the idea, the main .gitlab-ci.yml was way to crowded already anyway.

Not sure if "buildtest.yml" is such a good name (since it's not only about 
building, but also about testing), but I fail to come up with a better 
suggestion...

Just a suggestion for discussion, not sure whether it's such a good idea 
yet: Maybe we could split it up even further into "system.yml" (with the 
various jobs that only test the "softmmu" binaries), "user.yml" (with the 
--disable-system jobs) and "misc.yml" (with mixed jobs) ?

> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> new file mode 100644
> index 00000000000..b24a153d904
> --- /dev/null
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -0,0 +1,80 @@
> +.native_build_job_template:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  before_script:
> +    - JOBS=$(expr $(nproc) + 1)
> +  script:
> +    - mkdir build
> +    - cd build
> +    - if test -n "$TARGETS";
> +      then
> +        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS --target-list="$TARGETS" ;
> +      else
> +        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS ;
> +      fi || { cat config.log meson-logs/meson-log.txt && exit 1; }
> +    - if test -n "$LD_JOBS";
> +      then
> +        meson configure . -Dbackend_max_links="$LD_JOBS" ;
> +      fi || exit 1;
> +    - make -j"$JOBS"
> +    - if test -n "$MAKE_CHECK_ARGS";
> +      then
> +        make -j"$JOBS" $MAKE_CHECK_ARGS ;
> +      fi
> +
> +.native_test_job_template:
> +  stage: test
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  script:
> +    - scripts/git-submodule.sh update
> +        $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
> +    - cd build
> +    - find . -type f -exec touch {} +
> +    # Avoid recompiling by hiding ninja with NINJA=":"
> +    - make NINJA=":" $MAKE_CHECK_ARGS
> +
> +.integration_test_job_template:
> +  cache:
> +    key: "${CI_JOB_NAME}-cache"
> +    paths:
> +      - ${CI_PROJECT_DIR}/avocado-cache
> +    policy: pull-push
> +  artifacts:
> +    name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
> +    when: always
> +    expire_in: 2 days
> +    paths:
> +      - build/tests/results/latest/results.xml
> +      - build/tests/results/latest/test-results
> +    reports:
> +      junit: build/tests/results/latest/results.xml
> +  before_script:
> +    - mkdir -p ~/.config/avocado
> +    - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
> +    - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
> +           >> ~/.config/avocado/avocado.conf
> +    - echo -e '[job.output.testlogs]\nstatuses = ["FAIL", "INTERRUPT"]'
> +           >> ~/.config/avocado/avocado.conf
> +    - if [ -d ${CI_PROJECT_DIR}/avocado-cache ]; then
> +        du -chs ${CI_PROJECT_DIR}/avocado-cache ;
> +      fi
> +    - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
> +  after_script:
> +    - cd build
> +    - du -chs ${CI_PROJECT_DIR}/avocado-cache
> +
> +build-system-alpine:
> +  extends: .native_build_job_template
> +  needs:
> +    - job: amd64-alpine-container
> +  variables:
> +    IMAGE: alpine
> +    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
> +      moxie-softmmu microblazeel-softmmu mips64el-softmmu
> +    MAKE_CHECK_ARGS: check-build
> +    CONFIGURE_ARGS: --enable-docs --enable-trace-backends=log,simple,syslog
> +  artifacts:
> +    expire_in: 2 days
> +    paths:
> +      - .git-submodule-status
> +      - build

build-system-alpine should certainly not go into the template file, but into 
the file below instead.

> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> new file mode 100644
> index 00000000000..c24553eab05
> --- /dev/null
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -0,0 +1,744 @@
[...]
> +check-patch:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> +  needs:
> +    job: amd64-centos8-container
> +  script: .gitlab-ci.d/check-patch.py
> +  except:
> +    variables:
> +      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
> +  variables:
> +    GIT_DEPTH: 1000
> +  allow_failure: true
> +
> +check-dco:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> +  needs:
> +    job: amd64-centos8-container
> +  script: .gitlab-ci.d/check-dco.py
> +  except:
> +    variables:
> +      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
> +  variables:
> +    GIT_DEPTH: 1000

These two should not be here anymore since they have been handled in patch 
08/15 already.

 > +# No targets are built here, just tools, docs, and unit tests. This
 > +# also feeds into the eventual documentation deployment steps later
 > +build-tools-and-docs-debian:
 > +  extends: .native_build_job_template
 > +  needs:
 > +    job: amd64-debian-container
 > +  variables:
 > +    IMAGE: debian-amd64
 > +    MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
 > +    CONFIGURE_ARGS: --disable-system --disable-user --enable-docs 
--enable-tools
 > +  artifacts:
 > +    expire_in: 2 days
 > +    paths:
 > +      - build
 > +
 > +# Prepare for GitLab pages deployment. Anything copied into the
 > +# "public" directory will be deployed to $USER.gitlab.io/$PROJECT
 > +pages:
 > +  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
 > +  stage: test
 > +  needs:
 > +    - job: build-tools-and-docs-debian
 > +  script:
 > +    - mkdir -p public
 > +    # HTML-ised source tree
 > +    - make gtags
 > +    - htags -anT --tree-view=filetree -m qemu_init
 > +        -t "Welcome to the QEMU sourcecode"
 > +    - mv HTML public/src
 > +    # Project documentation
 > +    - make -C build install DESTDIR=$(pwd)/temp-install
 > +    - mv temp-install/usr/local/share/doc/qemu/* public/
 > +  artifacts:
 > +    paths:
 > +      - public

I think the "pages" job should maybe rather go into a separate file instead 
(maybe together with the build-tools-and-docs-debian job)?

  Thomas