[PATCH v1 0/9] testing/next (tags!, more travis/shippable deprecation)

[PATCH v1 0/9] testing/next (tags!, more travis/shippable deprecation)

[Alex Bennée]

Hi,

Not a lot in this tree at the moment but I did grab a few patches that
got inadvertently included in my pre-PR series for semihosting and
gdbstub. It does include Phillipe's deprecation of most of the
shippable builds. I've also moved one more test across from Travis
while trying to beef up the documentation we generate. We really need
a better "official" way to install the docs in an arbitrary directory
though.

The following need review:

 - gitlab: move docs and tools build across from Travis
 - docker: expand debian-amd64 image to include tag tools
 - Makefile: wrap cscope in quiet-command calls
 - Makefile: wrap etags in quiet-command calls
 - Makefile: wrap ctags in quiet-command calls
 - Add newline when generating Dockerfile
 - shippable.yml: Remove jobs duplicated on Gitlab-CI
 - Makefile: add GNU global tags support

Alessandro Di Federico (1):
  Add newline when generating Dockerfile

Alex Bennée (6):
  Makefile: add GNU global tags support
  Makefile: wrap ctags in quiet-command calls
  Makefile: wrap etags in quiet-command calls
  Makefile: wrap cscope in quiet-command calls
  docker: expand debian-amd64 image to include tag tools
  gitlab: move docs and tools build across from Travis

Philippe Mathieu-Daudé (2):
  tests/docker: Remove Debian 9 remnant lines
  shippable.yml: Remove jobs duplicated on Gitlab-CI

 Makefile                                     | 46 ++++++++++++++++----
 .gitignore                                   |  3 ++
 .gitlab-ci.yml                               | 22 +++++++---
 .shippable.yml                               | 14 +-----
 .travis.yml                                  | 16 -------
 tests/docker/Makefile.include                |  1 -
 tests/docker/docker.py                       |  4 +-
 tests/docker/dockerfiles/debian-amd64.docker |  5 ++-
 8 files changed, 64 insertions(+), 47 deletions(-)

-- 
2.20.1

[PATCH v1 1/9] tests/docker: Remove Debian 9 remnant lines

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Debian 9 base container has been removed in commits
e3755276d1f and c9d78b06c06. Remove the last remnants.

Fixes: e3755276d1f ("tests/docker: Remove old Debian 9 containers")
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20210107072933.3828450-1-f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
---
 tests/docker/Makefile.include | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index c254ac38d0..0779dab5b9 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -108,7 +108,6 @@ ifneq ($(HOST_ARCH),x86_64)
 DOCKER_PARTIAL_IMAGES += debian-mips-cross debian-mipsel-cross debian-mips64el-cross
 DOCKER_PARTIAL_IMAGES += debian-ppc64el-cross
 DOCKER_PARTIAL_IMAGES += debian-s390x-cross
-DOCKER_PARTIAL_IMAGES += debian-win32-cross debian-win64-cross
 DOCKER_PARTIAL_IMAGES += fedora travis
 endif
 
-- 
2.20.1

[PATCH v1 2/9] Makefile: add GNU global tags support

[Alex Bennée]

GNU Global is another tags engine which is more like cscope in being
able to support finding both references and definitions. You will be
un-surprised to know it also integrates well with Emacs.

The main benefit of integrating it into find-src-path is it takes less
time to rebuild the database from scratch when you have a lot of build
directories under your source tree.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - run in SRC_PATH
  - wrap in quiet-command
---
 Makefile   | 14 +++++++++++++-
 .gitignore |  3 +++
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index fb9923ff22..0c509a7704 100644
--- a/Makefile
+++ b/Makefile
@@ -253,6 +253,18 @@ ctags:
 	rm -f "$(SRC_PATH)/"tags
 	$(find-src-path) -exec ctags -f "$(SRC_PATH)/"tags --append {} +
 
+.PHONY: gtags
+gtags:
+	$(call quiet-command, 			\
+		rm -f "$(SRC_PATH)/"GTAGS; 	\
+		rm -f "$(SRC_PATH)/"GRTAGS; 	\
+		rm -f "$(SRC_PATH)/"GPATH, 	\
+		"GTAGS", "Remove old $@ files")
+	$(call quiet-command, 				\
+	        (cd $(SRC_PATH) && 			\
+		 $(find-src-path) | gtags -f -), 	\
+		"GTAGS", "Re-index $(SRC_PATH)")
+
 .PHONY: TAGS
 TAGS:
 	rm -f "$(SRC_PATH)/"TAGS
@@ -279,7 +291,7 @@ help:
 	$(call print-help,all,Build all)
 	$(call print-help,dir/file.o,Build specified target only)
 	$(call print-help,install,Install QEMU, documentation and tools)
-	$(call print-help,ctags/TAGS,Generate tags file for editors)
+	$(call print-help,ctags/gtags/TAGS,Generate tags file for editors)
 	$(call print-help,cscope,Generate cscope index)
 	$(call print-help,sparse,Run sparse on the QEMU source)
 	@echo  ''
diff --git a/.gitignore b/.gitignore
index b32bca1315..75a4be0724 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,6 +7,9 @@
 cscope.*
 tags
 TAGS
+GPATH
+GRTAGS
+GTAGS
 *~
 *.ast_raw
 *.depend_raw
-- 
2.20.1

[PATCH v1 3/9] shippable.yml: Remove jobs duplicated on Gitlab-CI

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

The following jobs are duplicated on Gitlab-CI since commit
6bcb5fc0f7a ("gitlab-ci: Add cross-compiling build tests"):

- IMAGE=debian-armel-cross

  TARGET_LIST=arm-softmmu               -> cross-armel-system
  TARGET_LIST=arm-linux-user            -> cross-armel-user
  TARGET_LIST=armeb-linux-user          -> cross-armel-user

- IMAGE=debian-armhf-cross

  TARGET_LIST=arm-softmmu               -> cross-armhf-system
  TARGET_LIST=arm-linux-user            -> cross-armhf-user
  TARGET_LIST=armeb-linux-user          -> cross-armhf-user

- IMAGE=debian-arm64-cross

  TARGET_LIST=aarch64-softmmu           -> cross-arm64-system
  TARGET_LIST=aarch64-linux-user        -> cross-arm64-user

- IMAGE=debian-s390x-cross

  TARGET_LIST=s390x-softmmu             -> cross-s390x-system
  TARGET_LIST=s390x-linux-user          -> cross-s390x-user

- IMAGE=debian-mips-cross

  TARGET_LIST=mipsel-linux-user         -> cross-mips-user

- IMAGE=debian-mips64el-cross

  TARGET_LIST=mips64el-softmmu          -> cross-mips64el-system
  TARGET_LIST=mips64el-linux-user       -> cross-mips64el-user

- IMAGE=debian-ppc64el-cross

  TARGET_LIST=ppc64-softmmu             -> cross-ppc64el-system
  TARGET_LIST=ppc64-linux-user          -> cross-ppc64el-user
  TARGET_LIST=ppc64abi32-linux-user     -> cross-ppc64el-user

Remove them from Shippable CI.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210108145103.269353-1-f4bug@amsat.org>
---
 .shippable.yml | 14 +-------------
 1 file changed, 1 insertion(+), 13 deletions(-)

diff --git a/.shippable.yml b/.shippable.yml
index 14350e6de8..97bfa2a0f3 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -7,20 +7,8 @@ env:
   matrix:
     - IMAGE=debian-amd64
       TARGET_LIST=x86_64-softmmu,x86_64-linux-user
-    - IMAGE=debian-armel-cross
-      TARGET_LIST=arm-softmmu,arm-linux-user,armeb-linux-user
-    - IMAGE=debian-armhf-cross
-      TARGET_LIST=arm-softmmu,arm-linux-user,armeb-linux-user
-    - IMAGE=debian-arm64-cross
-      TARGET_LIST=aarch64-softmmu,aarch64-linux-user
-    - IMAGE=debian-s390x-cross
-      TARGET_LIST=s390x-softmmu,s390x-linux-user
     - IMAGE=debian-mips-cross
-      TARGET_LIST=mips-softmmu,mipsel-linux-user
-    - IMAGE=debian-mips64el-cross
-      TARGET_LIST=mips64el-softmmu,mips64el-linux-user
-    - IMAGE=debian-ppc64el-cross
-      TARGET_LIST=ppc64-softmmu,ppc64-linux-user,ppc64abi32-linux-user
+      TARGET_LIST=mips-softmmu
 build:
   pre_ci_boot:
     image_name: registry.gitlab.com/qemu-project/qemu/qemu/${IMAGE}
-- 
2.20.1

[PATCH v1 4/9] Add newline when generating Dockerfile

[Alex Bennée]

From: Alessandro Di Federico <ale@rev.ng>

Signed-off-by: Alessandro Di Federico <ale@rev.ng>
Message-Id: <1610080146-14968-36-git-send-email-tsimpson@quicinc.com>
---
 tests/docker/docker.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 36b7868406..884dfeb29c 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -332,9 +332,9 @@ class Docker(object):
                          (uname, uid, uname))
 
         tmp_df.write("\n")
-        tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" % (checksum))
+        tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s\n" % (checksum))
         for f, c in extra_files_cksum:
-            tmp_df.write("LABEL com.qemu.%s-checksum=%s" % (f, c))
+            tmp_df.write("LABEL com.qemu.%s-checksum=%s\n" % (f, c))
 
         tmp_df.flush()
 
-- 
2.20.1

[PATCH v1 5/9] Makefile: wrap ctags in quiet-command calls

[Alex Bennée]

For prettier output.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 Makefile | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 0c509a7704..bbab640b31 100644
--- a/Makefile
+++ b/Makefile
@@ -250,8 +250,13 @@ find-src-path = find "$(SRC_PATH)/" -path "$(SRC_PATH)/meson" -prune -o \( -name
 
 .PHONY: ctags
 ctags:
-	rm -f "$(SRC_PATH)/"tags
-	$(find-src-path) -exec ctags -f "$(SRC_PATH)/"tags --append {} +
+	$(call quiet-command, 			\
+		rm -f "$(SRC_PATH)/"tags, 	\
+		"CTAGS", "Remove old tags")
+	$(call quiet-command, \
+		$(find-src-path) -exec ctags 		\
+		-f "$(SRC_PATH)/"tags --append {} +,	\
+		"CTAGS", "Re-index $(SRC_PATH)")
 
 .PHONY: gtags
 gtags:
-- 
2.20.1

[PATCH v1 6/9] Makefile: wrap etags in quiet-command calls

[Alex Bennée]

For prettier output.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 Makefile | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index bbab640b31..f7e9eb9f08 100644
--- a/Makefile
+++ b/Makefile
@@ -272,8 +272,13 @@ gtags:
 
 .PHONY: TAGS
 TAGS:
-	rm -f "$(SRC_PATH)/"TAGS
-	$(find-src-path) -exec etags -f "$(SRC_PATH)/"TAGS --append {} +
+	$(call quiet-command, 			\
+		rm -f "$(SRC_PATH)/"TAGS,	\
+		"TAGS", "Remove old $@")
+	$(call quiet-command, 				\
+		$(find-src-path) -exec etags 		\
+		-f "$(SRC_PATH)/"TAGS --append {} +, 	\
+		"TAGS", "Re-index $(SRC_PATH)")
 
 .PHONY: cscope
 cscope:
-- 
2.20.1

[PATCH v1 7/9] Makefile: wrap cscope in quiet-command calls

[Alex Bennée]

For prettier output.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 Makefile | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index f7e9eb9f08..2a926aaeb0 100644
--- a/Makefile
+++ b/Makefile
@@ -282,9 +282,17 @@ TAGS:
 
 .PHONY: cscope
 cscope:
-	rm -f "$(SRC_PATH)"/cscope.*
-	$(find-src-path) -print | sed -e 's,^\./,,' > "$(SRC_PATH)/cscope.files"
-	cscope -b -i"$(SRC_PATH)/cscope.files" -f"$(SRC_PATH)"/cscope.out
+	$(call quiet-command,			\
+		rm -f "$(SRC_PATH)/"cscope.* ,	\
+		"cscope", "Remove old $@ files")
+	$(call quiet-command, 					\
+		($(find-src-path) -print | sed -e 's,^\./,,'    \
+		> "$(SRC_PATH)/cscope.files"), 			\
+		"cscope", "Create file list")
+	$(call quiet-command, 				\
+		cscope -b -i"$(SRC_PATH)/cscope.files" 	\
+		-f"$(SRC_PATH)"/cscope.out, 		\
+		"cscope", "Re-index $(SRC_PATH)")
 
 # Needed by "meson install"
 export DESTDIR
-- 
2.20.1

[PATCH v1 8/9] docker: expand debian-amd64 image to include tag tools

[Alex Bennée]

This is going to be helpful when we want to both test the tool
integration and in the case of global generate a xref doc build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-amd64.docker | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index 55075d9fce..a98314757d 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -1,7 +1,7 @@
 #
 # Docker x86_64 target
 #
-# This docker target builds on the debian Stretch base image. Further
+# This docker target builds on the Debian Buster base image. Further
 # libraries which are not widely available are installed by hand.
 #
 FROM qemu/debian10
@@ -14,7 +14,10 @@ RUN apt update && \
 RUN apt update && \
     DEBIAN_FRONTEND=noninteractive eatmydata \
     apt install -y --no-install-recommends \
+        cscope \
         genisoimage \
+        exuberant-ctags \
+        global \
         libbz2-dev \
         liblzo2-dev \
         libgcrypt20-dev \
-- 
2.20.1

[PATCH v1 9/9] gitlab: move docs and tools build across from Travis

[Alex Bennée]

While we are at it we might as well check the tag generation. For
bonus points we run GNU globals htags into the public pages directory
for publishing with the auto generated pages.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.yml | 22 ++++++++++++++++------
 .travis.yml    | 16 ----------------
 2 files changed, 16 insertions(+), 22 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 4532f1718a..c07064a4f7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -79,7 +79,6 @@ build-system-ubuntu:
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
       moxie-softmmu microblazeel-softmmu mips64el-softmmu
     MAKE_CHECK_ARGS: check-build
-    CONFIGURE_ARGS: --enable-docs
   artifacts:
     expire_in: 2 days
     paths:
@@ -111,7 +110,6 @@ build-system-debian:
     TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
       riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
     MAKE_CHECK_ARGS: check-build
-    CONFIGURE_ARGS: --enable-docs
   artifacts:
     expire_in: 2 days
     paths:
@@ -126,6 +124,17 @@ check-system-debian:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check
 
+build-tools-and-docs-debian:
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: debian-amd64
+    MAKE_CHECK_ARGS: ctags gtags TAGS cscope
+    CONFIGURE_ARGS: --enable-docs --enable-tools
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
 acceptance-system-debian:
   <<: *native_test_job_definition
   needs:
@@ -597,13 +606,14 @@ build-libvhost-user:
     - ninja
 
 pages:
-  image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest
+  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
   stage: test
   needs:
-    - job: build-system-ubuntu
-      artifacts: true
+    - job: build-tools-and-docs-debian
   script:
-    - mkdir public
+    - mkdir -p public/src
+    - htags --suggest --tree-view=filetree -m qemu_init
+        -t "Welcome to the QEMU source code" public/src
     - mv build/docs/index.html public/
     - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done
   artifacts:
diff --git a/.travis.yml b/.travis.yml
index f2a101936c..3b574a5968 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -148,22 +148,6 @@ jobs:
         - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
-    # Check we can build docs and tools (out of tree)
-    - name: "tools and docs (bionic)"
-      dist: bionic
-      env:
-        - BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
-        - BASE_CONFIG="--enable-tools --enable-docs"
-        - CONFIG="--target-list=x86_64-softmmu,aarch64-linux-user"
-        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
-      addons:
-        apt:
-          packages:
-            - ninja-build
-            - python3-sphinx
-            - perl
-
-
     # Test with Clang for compile portability (Travis uses clang-5.0)
     - name: "Clang (user)"
       env:
-- 
2.20.1

Re: [PATCH v1 9/9] gitlab: move docs and tools build across from Travis

[Thomas Huth]

On 13/01/2021 16.14, Alex Bennée wrote:
> While we are at it we might as well check the tag generation. For
> bonus points we run GNU globals htags into the public pages directory
> for publishing with the auto generated pages.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.yml | 22 ++++++++++++++++------
>   .travis.yml    | 16 ----------------
>   2 files changed, 16 insertions(+), 22 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 4532f1718a..c07064a4f7 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -79,7 +79,6 @@ build-system-ubuntu:
>       TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
>         moxie-softmmu microblazeel-softmmu mips64el-softmmu
>       MAKE_CHECK_ARGS: check-build
> -    CONFIGURE_ARGS: --enable-docs
>     artifacts:
>       expire_in: 2 days
>       paths:
> @@ -111,7 +110,6 @@ build-system-debian:
>       TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
>         riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
>       MAKE_CHECK_ARGS: check-build
> -    CONFIGURE_ARGS: --enable-docs
>     artifacts:
>       expire_in: 2 days
>       paths:
> @@ -126,6 +124,17 @@ check-system-debian:
>       IMAGE: debian-amd64
>       MAKE_CHECK_ARGS: check
>   
> +build-tools-and-docs-debian:
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: debian-amd64
> +    MAKE_CHECK_ARGS: ctags gtags TAGS cscope
> +    CONFIGURE_ARGS: --enable-docs --enable-tools
> +  artifacts:
> +    expire_in: 2 days
> +    paths:
> +      - build
> +
>   acceptance-system-debian:
>     <<: *native_test_job_definition
>     needs:
> @@ -597,13 +606,14 @@ build-libvhost-user:
>       - ninja
>   
>   pages:
> -  image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest
> +  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
>     stage: test
>     needs:
> -    - job: build-system-ubuntu
> -      artifacts: true
> +    - job: build-tools-and-docs-debian
>     script:
> -    - mkdir public
> +    - mkdir -p public/src
> +    - htags --suggest --tree-view=filetree -m qemu_init
> +        -t "Welcome to the QEMU source code" public/src
>       - mv build/docs/index.html public/
>       - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done
>     artifacts:
> diff --git a/.travis.yml b/.travis.yml
> index f2a101936c..3b574a5968 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -148,22 +148,6 @@ jobs:
>           - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>   
>   
> -    # Check we can build docs and tools (out of tree)
> -    - name: "tools and docs (bionic)"
> -      dist: bionic
> -      env:
> -        - BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
> -        - BASE_CONFIG="--enable-tools --enable-docs"
> -        - CONFIG="--target-list=x86_64-softmmu,aarch64-linux-user"
> -        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
> -      addons:
> -        apt:
> -          packages:
> -            - ninja-build
> -            - python3-sphinx
> -            - perl
> -
> -
>       # Test with Clang for compile portability (Travis uses clang-5.0)
>       - name: "Clang (user)"
>         env:
> 

Acked-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v1 9/9] gitlab: move docs and tools build across from Travis

[Daniel P. Berrangé]

On Wed, Jan 13, 2021 at 03:14:08PM +0000, Alex Bennée wrote:
> While we are at it we might as well check the tag generation. For
> bonus points we run GNU globals htags into the public pages directory
> for publishing with the auto generated pages.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  .gitlab-ci.yml | 22 ++++++++++++++++------
>  .travis.yml    | 16 ----------------
>  2 files changed, 16 insertions(+), 22 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 4532f1718a..c07064a4f7 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -79,7 +79,6 @@ build-system-ubuntu:
>      TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
>        moxie-softmmu microblazeel-softmmu mips64el-softmmu
>      MAKE_CHECK_ARGS: check-build
> -    CONFIGURE_ARGS: --enable-docs
>    artifacts:
>      expire_in: 2 days
>      paths:
> @@ -111,7 +110,6 @@ build-system-debian:
>      TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
>        riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
>      MAKE_CHECK_ARGS: check-build
> -    CONFIGURE_ARGS: --enable-docs
>    artifacts:
>      expire_in: 2 days
>      paths:
> @@ -126,6 +124,17 @@ check-system-debian:
>      IMAGE: debian-amd64
>      MAKE_CHECK_ARGS: check
>  
> +build-tools-and-docs-debian:
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: debian-amd64
> +    MAKE_CHECK_ARGS: ctags gtags TAGS cscope
> +    CONFIGURE_ARGS: --enable-docs --enable-tools
> +  artifacts:
> +    expire_in: 2 days
> +    paths:
> +      - build

Do we actually need this job at all ?

Assuming the depenedancies are in the dockerfile, then all the
build jobs will be testing docs and tools already, as meson will
auto enable it.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 9/9] gitlab: move docs and tools build across from Travis

[Peter Maydell]

On Wed, 13 Jan 2021 at 15:40, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Wed, Jan 13, 2021 at 03:14:08PM +0000, Alex Bennée wrote:
> > While we are at it we might as well check the tag generation. For
> > bonus points we run GNU globals htags into the public pages directory
> > for publishing with the auto generated pages.

> > +build-tools-and-docs-debian:
> > +  <<: *native_build_job_definition
> > +  variables:
> > +    IMAGE: debian-amd64
> > +    MAKE_CHECK_ARGS: ctags gtags TAGS cscope
> > +    CONFIGURE_ARGS: --enable-docs --enable-tools
> > +  artifacts:
> > +    expire_in: 2 days
> > +    paths:
> > +      - build
>
> Do we actually need this job at all ?
>
> Assuming the depenedancies are in the dockerfile, then all the
> build jobs will be testing docs and tools already, as meson will
> auto enable it.

What I would like to see is a set of "just build the docs"
CI jobs which test all our supported Sphinx versions
(eg 1.6, 2.0, 2.4, 3.0, 3.2), since we've found that Sphinx
is not great about consistency of accepted syntax from
version to version. (Somebody sent a patch for this ages
ago, but it was for a Travis CI job IIRC, which we're trying
to move away from.)

thanks
-- PMM

Re: [PATCH v1 9/9] gitlab: move docs and tools build across from Travis

[Daniel P. Berrangé]

On Wed, Jan 13, 2021 at 03:46:17PM +0000, Peter Maydell wrote:
> On Wed, 13 Jan 2021 at 15:40, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Wed, Jan 13, 2021 at 03:14:08PM +0000, Alex Bennée wrote:
> > > While we are at it we might as well check the tag generation. For
> > > bonus points we run GNU globals htags into the public pages directory
> > > for publishing with the auto generated pages.
> 
> > > +build-tools-and-docs-debian:
> > > +  <<: *native_build_job_definition
> > > +  variables:
> > > +    IMAGE: debian-amd64
> > > +    MAKE_CHECK_ARGS: ctags gtags TAGS cscope
> > > +    CONFIGURE_ARGS: --enable-docs --enable-tools
> > > +  artifacts:
> > > +    expire_in: 2 days
> > > +    paths:
> > > +      - build
> >
> > Do we actually need this job at all ?
> >
> > Assuming the depenedancies are in the dockerfile, then all the
> > build jobs will be testing docs and tools already, as meson will
> > auto enable it.
> 
> What I would like to see is a set of "just build the docs"
> CI jobs which test all our supported Sphinx versions
> (eg 1.6, 2.0, 2.4, 3.0, 3.2), since we've found that Sphinx
> is not great about consistency of accepted syntax from
> version to version. (Somebody sent a patch for this ages
> ago, but it was for a Travis CI job IIRC, which we're trying
> to move away from.)

We only care about the Sphinx versions that exist in the OS distros that
we're targetting as build platforms. So as long as we have build jobs for
all the distros we have dockerfiles for, then we're covering all the
sphinx versions that matter.  IOW, I think our gitlab jobs are already
covering a good variety of sphinx versions. There are possibly gaps because
our dockerfiles have a horribly inconsistent set of RPMs listed. This
inconsistency is addressed in this patch series of mine to autogenerate
the dockerfiles:

 https://lists.gnu.org/archive/html/qemu-devel/2020-12/msg00189.html

which will ensure all our dockerfiles have the deps needed to satisfy
all features in QEMU.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 9/9] gitlab: move docs and tools build across from Travis

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Jan 13, 2021 at 03:14:08PM +0000, Alex Bennée wrote:
>> While we are at it we might as well check the tag generation. For
>> bonus points we run GNU globals htags into the public pages directory
>> for publishing with the auto generated pages.
>> 
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>  .gitlab-ci.yml | 22 ++++++++++++++++------
>>  .travis.yml    | 16 ----------------
>>  2 files changed, 16 insertions(+), 22 deletions(-)
>> 
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 4532f1718a..c07064a4f7 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -79,7 +79,6 @@ build-system-ubuntu:
>>      TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
>>        moxie-softmmu microblazeel-softmmu mips64el-softmmu
>>      MAKE_CHECK_ARGS: check-build
>> -    CONFIGURE_ARGS: --enable-docs
>>    artifacts:
>>      expire_in: 2 days
>>      paths:
>> @@ -111,7 +110,6 @@ build-system-debian:
>>      TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
>>        riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
>>      MAKE_CHECK_ARGS: check-build
>> -    CONFIGURE_ARGS: --enable-docs
>>    artifacts:
>>      expire_in: 2 days
>>      paths:
>> @@ -126,6 +124,17 @@ check-system-debian:
>>      IMAGE: debian-amd64
>>      MAKE_CHECK_ARGS: check
>>  
>> +build-tools-and-docs-debian:
>> +  <<: *native_build_job_definition
>> +  variables:
>> +    IMAGE: debian-amd64
>> +    MAKE_CHECK_ARGS: ctags gtags TAGS cscope
>> +    CONFIGURE_ARGS: --enable-docs --enable-tools
>> +  artifacts:
>> +    expire_in: 2 days
>> +    paths:
>> +      - build
>
> Do we actually need this job at all ?
>
> Assuming the depenedancies are in the dockerfile, then all the
> build jobs will be testing docs and tools already, as meson will
> auto enable it.

I pondered making an explicit --disable-docs to save some run time but
really this is providing a golden build of the docs which can tehn be
deployed by pages.

It also ensure the tag generation works for the various tag outputs -
the GNU global output being used to also publish a HTML navigable
version of the source tree.

>
> Regards,
> Daniel


-- 
Alex Bennée

Re: [PATCH v1 9/9] gitlab: move docs and tools build across from Travis

[Daniel P. Berrangé]

On Wed, Jan 13, 2021 at 04:27:54PM +0000, Alex Bennée wrote:
> 
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Wed, Jan 13, 2021 at 03:14:08PM +0000, Alex Bennée wrote:
> >> While we are at it we might as well check the tag generation. For
> >> bonus points we run GNU globals htags into the public pages directory
> >> for publishing with the auto generated pages.
> >> 
> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >> ---
> >>  .gitlab-ci.yml | 22 ++++++++++++++++------
> >>  .travis.yml    | 16 ----------------
> >>  2 files changed, 16 insertions(+), 22 deletions(-)
> >> 
> >> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> >> index 4532f1718a..c07064a4f7 100644
> >> --- a/.gitlab-ci.yml
> >> +++ b/.gitlab-ci.yml
> >> @@ -79,7 +79,6 @@ build-system-ubuntu:
> >>      TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
> >>        moxie-softmmu microblazeel-softmmu mips64el-softmmu
> >>      MAKE_CHECK_ARGS: check-build
> >> -    CONFIGURE_ARGS: --enable-docs
> >>    artifacts:
> >>      expire_in: 2 days
> >>      paths:
> >> @@ -111,7 +110,6 @@ build-system-debian:
> >>      TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
> >>        riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
> >>      MAKE_CHECK_ARGS: check-build
> >> -    CONFIGURE_ARGS: --enable-docs
> >>    artifacts:
> >>      expire_in: 2 days
> >>      paths:
> >> @@ -126,6 +124,17 @@ check-system-debian:
> >>      IMAGE: debian-amd64
> >>      MAKE_CHECK_ARGS: check
> >>  
> >> +build-tools-and-docs-debian:
> >> +  <<: *native_build_job_definition
> >> +  variables:
> >> +    IMAGE: debian-amd64
> >> +    MAKE_CHECK_ARGS: ctags gtags TAGS cscope
> >> +    CONFIGURE_ARGS: --enable-docs --enable-tools
> >> +  artifacts:
> >> +    expire_in: 2 days
> >> +    paths:
> >> +      - build
> >
> > Do we actually need this job at all ?
> >
> > Assuming the depenedancies are in the dockerfile, then all the
> > build jobs will be testing docs and tools already, as meson will
> > auto enable it.
> 
> I pondered making an explicit --disable-docs to save some run time but
> really this is providing a golden build of the docs which can tehn be
> deployed bypages.

Oh right I missed that subtle detail. I saw the separate pages job
but didn't realize this was a pre-req of it.

So with that in mind no objection.

> It also ensure the tag generation works for the various tag outputs -
> the GNU global output being used to also publish a HTML navigable
> version of the source tree.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 9/9] gitlab: move docs and tools build across from Travis

[Wainer dos Santos Moschetta]

On 1/13/21 12:14 PM, Alex Bennée wrote:
> While we are at it we might as well check the tag generation. For
> bonus points we run GNU globals htags into the public pages directory
> for publishing with the auto generated pages.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.yml | 22 ++++++++++++++++------
>   .travis.yml    | 16 ----------------
>   2 files changed, 16 insertions(+), 22 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 4532f1718a..c07064a4f7 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -79,7 +79,6 @@ build-system-ubuntu:
>       TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
>         moxie-softmmu microblazeel-softmmu mips64el-softmmu
>       MAKE_CHECK_ARGS: check-build
> -    CONFIGURE_ARGS: --enable-docs
>     artifacts:
>       expire_in: 2 days
>       paths:
> @@ -111,7 +110,6 @@ build-system-debian:
>       TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
>         riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
>       MAKE_CHECK_ARGS: check-build
> -    CONFIGURE_ARGS: --enable-docs
>     artifacts:
>       expire_in: 2 days
>       paths:
> @@ -126,6 +124,17 @@ check-system-debian:
>       IMAGE: debian-amd64
>       MAKE_CHECK_ARGS: check
>   
> +build-tools-and-docs-debian:
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: debian-amd64
> +    MAKE_CHECK_ARGS: ctags gtags TAGS cscope
> +    CONFIGURE_ARGS: --enable-docs --enable-tools
> +  artifacts:
> +    expire_in: 2 days
> +    paths:
> +      - build
> +
>   acceptance-system-debian:
>     <<: *native_test_job_definition
>     needs:
> @@ -597,13 +606,14 @@ build-libvhost-user:
>       - ninja
>   
>   pages:
> -  image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest
> +  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
>     stage: test
>     needs:
> -    - job: build-system-ubuntu
> -      artifacts: true
> +    - job: build-tools-and-docs-debian
>     script:


Just for the records: artifacts is true by default.


> -    - mkdir public
> +    - mkdir -p public/src
> +    - htags --suggest --tree-view=filetree -m qemu_init
> +        -t "Welcome to the QEMU source code" public/src
>       - mv build/docs/index.html public/
>       - for i in devel interop specs system tools user ; do mv build/docs/$i public/ ; done
>     artifacts:
> diff --git a/.travis.yml b/.travis.yml
> index f2a101936c..3b574a5968 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -148,22 +148,6 @@ jobs:
>           - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>   
>   
> -    # Check we can build docs and tools (out of tree)
> -    - name: "tools and docs (bionic)"
> -      dist: bionic
> -      env:
> -        - BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
> -        - BASE_CONFIG="--enable-tools --enable-docs"
> -        - CONFIG="--target-list=x86_64-softmmu,aarch64-linux-user"
> -        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
> -      addons:
> -        apt:
> -          packages:
> -            - ninja-build
> -            - python3-sphinx
> -            - perl
> -
> -
>       # Test with Clang for compile portability (Travis uses clang-5.0)
>       - name: "Clang (user)"
>         env:


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

Re: [PATCH v1 9/9] gitlab: move docs and tools build across from Travis

[Peter Maydell]

On Wed, 13 Jan 2021 at 16:35, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Wed, Jan 13, 2021 at 04:27:54PM +0000, Alex Bennée wrote:
> >
> > Daniel P. Berrangé <berrange@redhat.com> writes:
> > > Do we actually need this job at all ?
> > >
> > > Assuming the depenedancies are in the dockerfile, then all the
> > > build jobs will be testing docs and tools already, as meson will
> > > auto enable it.
> >
> > I pondered making an explicit --disable-docs to save some run time but
> > really this is providing a golden build of the docs which can tehn be
> > deployed bypages.
>
> Oh right I missed that subtle detail. I saw the separate pages job
> but didn't realize this was a pre-req of it.

This suggests that a brief comment for each job explaining why
we have it would be useful. (compare my inability the other day to
identify what the 'pages' job was doing -- a comment saying
"This job publishes the HTML docs previously built by the
$whatever job as gitlab pages which can then be viewed at $url"
would have helped a lot.)

-- PMM