* [PATCH v6 1/4] cirrus: Fixing and speedup the msys2/mingw CI
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
@ 2020-10-09 11:07 ` Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 2/4] cirrus: msys2/mingw speed is up, add excluded target back Yonggang Luo
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Yonggang Luo @ 2020-10-09 11:07 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, QEMU Trivial,
Philippe Mathieu-Daudé,
Yonggang Luo, Paolo Bonzini
Use cache of cirrus caching msys2
The install of msys2 are refer to https://github.com/msys2/setup-msys2
The first time install msys2 would be time consuming, so increase timeout_in to 90m
according to https://cirrus-ci.org/faq/#instance-timed-out
Apply patch of https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00072.html
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
.cirrus.yml | 111 +++++++++++++++++++++++++++++++---------------------
1 file changed, 66 insertions(+), 45 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index d58782ce67..ed3db10e45 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -46,64 +46,85 @@ macos_xcode_task:
- gmake check
windows_msys2_task:
+ timeout_in: 90m
windows_container:
- image: cirrusci/windowsservercore:cmake
+ image: cirrusci/windowsservercore:2019
os_version: 2019
cpu: 8
memory: 8G
env:
+ CIRRUS_SHELL: powershell
MSYS: winsymlinks:nativestrict
MSYSTEM: MINGW64
CHERE_INVOKING: 1
- printenv_script:
- - C:\tools\msys64\usr\bin\bash.exe -lc 'printenv'
+ setup_script:
+ - choco install -y --no-progress 7zip
+ - Write-Output $env:PATH
+ msys2_cache:
+ folder: C:\tools\archive
+ reupload_on_changes: false
+ fingerprint_script: cat .cirrus.yml
+ populate_script:
+ - |
+ md C:\tools
+ md C:\tools\archive
+ $start_time = Get-Date
+ cd C:\tools
+ bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND https://github.com/msys2/msys2-installer/releases/download/2020-09-03/msys2-base-x86_64-20200903.sfx.exe C:\tools\base.exe
+ Write-Output "Download time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
+ C:\tools\base.exe -y
+ ((Get-Content -path C:\tools\msys64\etc\\post-install\\07-pacman-key.post -Raw) -replace '--refresh-keys', '--version') | Set-Content -Path C:\tools\msys64\etc\\post-install\\07-pacman-key.post
+ C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
+ C:\tools\msys64\usr\bin\bash.exe -lc "export"
+ C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Sy
+ echo Y | C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Suu --overwrite=*
+ taskkill /F /FI "MODULES eq msys-2.0.dll"
+ tasklist
+ C:\tools\msys64\usr\bin\bash.exe -lc "mv -f /etc/pacman.conf.pacnew /etc/pacman.conf || true"
+ C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Suu --overwrite=*"
+ C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed \
+ diffutils git grep make pkg-config sed \
+ mingw-w64-x86_64-python \
+ mingw-w64-x86_64-python-setuptools \
+ mingw-w64-x86_64-toolchain \
+ mingw-w64-x86_64-SDL2 \
+ mingw-w64-x86_64-SDL2_image \
+ mingw-w64-x86_64-gtk3 \
+ mingw-w64-x86_64-glib2 \
+ mingw-w64-x86_64-ninja \
+ mingw-w64-x86_64-jemalloc \
+ mingw-w64-x86_64-lzo2 \
+ mingw-w64-x86_64-zstd \
+ mingw-w64-x86_64-libjpeg-turbo \
+ mingw-w64-x86_64-pixman \
+ mingw-w64-x86_64-libgcrypt \
+ mingw-w64-x86_64-libpng \
+ mingw-w64-x86_64-libssh \
+ mingw-w64-x86_64-libxml2 \
+ mingw-w64-x86_64-snappy \
+ mingw-w64-x86_64-libusb \
+ mingw-w64-x86_64-usbredir \
+ mingw-w64-x86_64-libtasn1 \
+ mingw-w64-x86_64-nettle \
+ mingw-w64-x86_64-cyrus-sasl \
+ mingw-w64-x86_64-curl \
+ mingw-w64-x86_64-gnutls \
+ "
+ C:\tools\msys64\usr\bin\bash.exe -lc "rm -rf /var/cache/pacman/pkg/*"
+ cd C:\tools\msys64
+ echo "Start archive"
+ cmd /C "7z a -ttar . -so | 7z a -txz -simsys2-x86_64.tar C:\tools\archive\msys2-x86_64.tar.xz"
install_script:
- - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
- curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
- - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
- curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz.sig"
- - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
- pacman -U --noconfirm msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
- - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Sy"
- - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
- bash pacman pacman-mirrors msys2-runtime"
- - taskkill /F /IM gpg-agent.exe
- - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Su"
- - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
- base-devel
- git
- mingw-w64-x86_64-python
- mingw-w64-x86_64-python-setuptools
- mingw-w64-x86_64-toolchain
- mingw-w64-x86_64-capstone
- mingw-w64-x86_64-SDL2
- mingw-w64-x86_64-SDL2_image
- mingw-w64-x86_64-gtk3
- mingw-w64-x86_64-glib2
- mingw-w64-x86_64-ninja
- mingw-w64-x86_64-make
- mingw-w64-x86_64-lzo2
- mingw-w64-x86_64-zstd
- mingw-w64-x86_64-libjpeg-turbo
- mingw-w64-x86_64-pixman
- mingw-w64-x86_64-libgcrypt
- mingw-w64-x86_64-libpng
- mingw-w64-x86_64-libssh
- mingw-w64-x86_64-libxml2
- mingw-w64-x86_64-snappy
- mingw-w64-x86_64-libusb
- mingw-w64-x86_64-usbredir
- mingw-w64-x86_64-libtasn1
- mingw-w64-x86_64-nettle
- mingw-w64-x86_64-cyrus-sasl
- mingw-w64-x86_64-curl
- mingw-w64-x86_64-gnutls
- mingw-w64-x86_64-zstd"
+ - |
+ cd C:\tools
+ cmd /C "7z x C:\tools\archive\msys2-x86_64.tar.xz -so | 7z x -aoa -simsys2-x86_64.tar -ttar -omsys64"
+ C:\tools\msys64\usr\bin\bash.exe -lc "export"
+
script:
- C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure
--python=python3 --ninja=ninja
--target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu"
- - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j$NUMBER_OF_PROCESSORS"
+ - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j8"
test_script:
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make V=1 check"
--
2.28.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v6 2/4] cirrus: msys2/mingw speed is up, add excluded target back
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 1/4] cirrus: Fixing and speedup the msys2/mingw CI Yonggang Luo
@ 2020-10-09 11:07 ` Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 3/4] docs: Fixes build docs on msys2/mingw Yonggang Luo
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Yonggang Luo @ 2020-10-09 11:07 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, QEMU Trivial,
Philippe Mathieu-Daudé,
Yonggang Luo, Paolo Bonzini
The following target are add back:
i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
.cirrus.yml | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index ed3db10e45..d1fe63b343 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -123,8 +123,7 @@ windows_msys2_task:
script:
- C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure
- --python=python3 --ninja=ninja
- --target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu"
+ --python=python3 --ninja=ninja"
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j8"
test_script:
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make V=1 check"
--
2.28.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v6 3/4] docs: Fixes build docs on msys2/mingw
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 1/4] cirrus: Fixing and speedup the msys2/mingw CI Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 2/4] cirrus: msys2/mingw speed is up, add excluded target back Yonggang Luo
@ 2020-10-09 11:07 ` Yonggang Luo
2020-10-09 11:08 ` [PATCH v6 4/4] cirrus: Enable doc build " Yonggang Luo
2020-10-09 11:10 ` [PATCH v6 0/4] Improve cirrus msys2 罗勇刚(Yonggang Luo)
4 siblings, 0 replies; 6+ messages in thread
From: Yonggang Luo @ 2020-10-09 11:07 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, QEMU Trivial,
Philippe Mathieu-Daudé,
Yonggang Luo, Paolo Bonzini
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
docs/conf.py | 2 +-
docs/sphinx/kerneldoc.py | 2 +-
scripts/rst-sanitize.py | 21 +++++++++++++++++++++
tests/qapi-schema/meson.build | 5 +++--
4 files changed, 26 insertions(+), 4 deletions(-)
create mode 100644 scripts/rst-sanitize.py
diff --git a/docs/conf.py b/docs/conf.py
index 00e1b750e2..e584f68393 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -241,7 +241,7 @@ texinfo_documents = [
# We use paths starting from qemu_docdir here so that you can run
# sphinx-build from anywhere and the kerneldoc extension can still
# find everything.
-kerneldoc_bin = os.path.join(qemu_docdir, '../scripts/kernel-doc')
+kerneldoc_bin = ['perl', os.path.join(qemu_docdir, '../scripts/kernel-doc')]
kerneldoc_srctree = os.path.join(qemu_docdir, '..')
hxtool_srctree = os.path.join(qemu_docdir, '..')
qapidoc_srctree = os.path.join(qemu_docdir, '..')
diff --git a/docs/sphinx/kerneldoc.py b/docs/sphinx/kerneldoc.py
index 3e87940206..3ac277d162 100644
--- a/docs/sphinx/kerneldoc.py
+++ b/docs/sphinx/kerneldoc.py
@@ -67,7 +67,7 @@ class KernelDocDirective(Directive):
def run(self):
env = self.state.document.settings.env
- cmd = [env.config.kerneldoc_bin, '-rst', '-enable-lineno']
+ cmd = env.config.kerneldoc_bin + ['-rst', '-enable-lineno']
filename = env.config.kerneldoc_srctree + '/' + self.arguments[0]
export_file_patterns = []
diff --git a/scripts/rst-sanitize.py b/scripts/rst-sanitize.py
new file mode 100644
index 0000000000..26060f1208
--- /dev/null
+++ b/scripts/rst-sanitize.py
@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+
+#
+# Script for remove cr line ending in file
+#
+# Authors:
+# Yonggang Luo <luoyonggang@gmail.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or, at your option, any later version. See the COPYING file in
+# the top-level directory.
+
+import sys
+
+def main(_program, file, *unused):
+ with open(file, 'rb') as content_file:
+ content = content_file.read()
+ sys.stdout.buffer.write(content.replace(b'\r', b''))
+
+if __name__ == "__main__":
+ main(*sys.argv)
diff --git a/tests/qapi-schema/meson.build b/tests/qapi-schema/meson.build
index f08c902911..a6832634b3 100644
--- a/tests/qapi-schema/meson.build
+++ b/tests/qapi-schema/meson.build
@@ -250,18 +250,19 @@ qapi_doc_out = custom_target('QAPI rST doc',
# using an explicit '\' character in the command arguments to
# a custom_target(), as Meson will unhelpfully replace it with a '/'
# (https://github.com/mesonbuild/meson/issues/1564)
+rst_sanitize_cmd = [find_program('../../scripts/rst-sanitize.py'), '@INPUT@']
qapi_doc_out_nocr = custom_target('QAPI rST doc newline-sanitized',
output: ['doc-good.txt.nocr'],
input: qapi_doc_out[0],
build_by_default: build_docs,
- command: ['perl', '-pe', '$x = chr 13; s/$x$//', '@INPUT@'],
+ command: rst_sanitize_cmd,
capture: true)
qapi_doc_ref_nocr = custom_target('QAPI rST doc reference newline-sanitized',
output: ['doc-good.ref.nocr'],
input: files('doc-good.txt'),
build_by_default: build_docs,
- command: ['perl', '-pe', '$x = chr 13; s/$x$//', '@INPUT@'],
+ command: rst_sanitize_cmd,
capture: true)
if build_docs
--
2.28.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v6 4/4] cirrus: Enable doc build on msys2/mingw
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
` (2 preceding siblings ...)
2020-10-09 11:07 ` [PATCH v6 3/4] docs: Fixes build docs on msys2/mingw Yonggang Luo
@ 2020-10-09 11:08 ` Yonggang Luo
2020-10-09 11:10 ` [PATCH v6 0/4] Improve cirrus msys2 罗勇刚(Yonggang Luo)
4 siblings, 0 replies; 6+ messages in thread
From: Yonggang Luo @ 2020-10-09 11:08 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, QEMU Trivial,
Philippe Mathieu-Daudé,
Yonggang Luo, Paolo Bonzini
Currently rST depends on old version sphinx-2.x.
Install it by downloading it.
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
.cirrus.yml | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/.cirrus.yml b/.cirrus.yml
index d1fe63b343..599fd299e3 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -110,6 +110,11 @@ windows_msys2_task:
mingw-w64-x86_64-curl \
mingw-w64-x86_64-gnutls \
"
+ bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND `
+ https://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz `
+ C:\tools\mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz
+ C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -U /c/tools/mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz"
+ del C:\tools\mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz
C:\tools\msys64\usr\bin\bash.exe -lc "rm -rf /var/cache/pacman/pkg/*"
cd C:\tools\msys64
echo "Start archive"
--
2.28.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH v6 0/4] Improve cirrus msys2
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
` (3 preceding siblings ...)
2020-10-09 11:08 ` [PATCH v6 4/4] cirrus: Enable doc build " Yonggang Luo
@ 2020-10-09 11:10 ` 罗勇刚(Yonggang Luo)
4 siblings, 0 replies; 6+ messages in thread
From: 罗勇刚(Yonggang Luo) @ 2020-10-09 11:10 UTC (permalink / raw)
To: qemu-level, Alex Bennée
Cc: QEMU Trivial, Peter Maydell, Thomas Huth,
Philippe Mathieu-Daudé,
Paolo Bonzini
[-- Attachment #1: Type: text/plain, Size: 1577 bytes --]
Hi, alex, the msys2 main repo are back, so I removed the university mirror
On Fri, Oct 9, 2020 at 7:08 PM Yonggang Luo <luoyonggang@gmail.com> wrote:
>
> Fixes the broken msys2/mingw ci and speed it up.
>
> V5-V6
> Remove the university mirror, the main repo are back.
> rename to setup_script.
> Add fixes for msys2/mingw rST document fixes
>
> V4-V5
> Now the cache are usefull by using 7zip to archive msys64
> Saved about 18min, compare
> https://cirrus-ci.com/task/5093551157542912
> https://cirrus-ci.com/task/6177196538593280
>
> V3-V4
> Using cirrus cache to speed up msys2 ci instead of downloading archive
file
>
> V2-V3
> Add one more patch:
> cirrus: msys2/mingw speed is up, add excluded target back
> Do not build sphinx on windows, that's failing
> set the number of parallel count to fixed number 8
>
> V1-V2
> Resolve the cirrus conflict
>
> Yonggang Luo (4):
> cirrus: Fixing and speedup the msys2/mingw CI
> cirrus: msys2/mingw speed is up, add excluded target back
> docs: Fixes build docs on msys2/mingw
> cirrus: Enable doc build on msys2/mingw
>
> .cirrus.yml | 119 ++++++++++++++++++++--------------
> docs/conf.py | 2 +-
> docs/sphinx/kerneldoc.py | 2 +-
> scripts/rst-sanitize.py | 21 ++++++
> tests/qapi-schema/meson.build | 5 +-
> 5 files changed, 98 insertions(+), 51 deletions(-)
> create mode 100644 scripts/rst-sanitize.py
>
> --
> 2.28.0.windows.1
>
--
此致
礼
罗勇刚
Yours
sincerely,
Yonggang Luo
[-- Attachment #2: Type: text/html, Size: 2049 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread