* [PATCH v6 0/4] Improve cirrus msys2
@ 2020-10-09 11:07 Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 1/4] cirrus: Fixing and speedup the msys2/mingw CI Yonggang Luo
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Yonggang Luo @ 2020-10-09 11:07 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, QEMU Trivial,
Philippe Mathieu-Daudé,
Yonggang Luo, Paolo Bonzini
Fixes the broken msys2/mingw ci and speed it up.
V5-V6
Remove the university mirror, the main repo are back.
rename to setup_script.
Add fixes for msys2/mingw rST document fixes
V4-V5
Now the cache are usefull by using 7zip to archive msys64
Saved about 18min, compare
https://cirrus-ci.com/task/5093551157542912
https://cirrus-ci.com/task/6177196538593280
V3-V4
Using cirrus cache to speed up msys2 ci instead of downloading archive file
V2-V3
Add one more patch:
cirrus: msys2/mingw speed is up, add excluded target back
Do not build sphinx on windows, that's failing
set the number of parallel count to fixed number 8
V1-V2
Resolve the cirrus conflict
Yonggang Luo (4):
cirrus: Fixing and speedup the msys2/mingw CI
cirrus: msys2/mingw speed is up, add excluded target back
docs: Fixes build docs on msys2/mingw
cirrus: Enable doc build on msys2/mingw
.cirrus.yml | 119 ++++++++++++++++++++--------------
docs/conf.py | 2 +-
docs/sphinx/kerneldoc.py | 2 +-
scripts/rst-sanitize.py | 21 ++++++
tests/qapi-schema/meson.build | 5 +-
5 files changed, 98 insertions(+), 51 deletions(-)
create mode 100644 scripts/rst-sanitize.py
--
2.28.0.windows.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v6 1/4] cirrus: Fixing and speedup the msys2/mingw CI
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
@ 2020-10-09 11:07 ` Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 2/4] cirrus: msys2/mingw speed is up, add excluded target back Yonggang Luo
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Yonggang Luo @ 2020-10-09 11:07 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, QEMU Trivial,
Philippe Mathieu-Daudé,
Yonggang Luo, Paolo Bonzini
Use cache of cirrus caching msys2
The install of msys2 are refer to https://github.com/msys2/setup-msys2
The first time install msys2 would be time consuming, so increase timeout_in to 90m
according to https://cirrus-ci.org/faq/#instance-timed-out
Apply patch of https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00072.html
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
.cirrus.yml | 111 +++++++++++++++++++++++++++++++---------------------
1 file changed, 66 insertions(+), 45 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index d58782ce67..ed3db10e45 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -46,64 +46,85 @@ macos_xcode_task:
- gmake check
windows_msys2_task:
+ timeout_in: 90m
windows_container:
- image: cirrusci/windowsservercore:cmake
+ image: cirrusci/windowsservercore:2019
os_version: 2019
cpu: 8
memory: 8G
env:
+ CIRRUS_SHELL: powershell
MSYS: winsymlinks:nativestrict
MSYSTEM: MINGW64
CHERE_INVOKING: 1
- printenv_script:
- - C:\tools\msys64\usr\bin\bash.exe -lc 'printenv'
+ setup_script:
+ - choco install -y --no-progress 7zip
+ - Write-Output $env:PATH
+ msys2_cache:
+ folder: C:\tools\archive
+ reupload_on_changes: false
+ fingerprint_script: cat .cirrus.yml
+ populate_script:
+ - |
+ md C:\tools
+ md C:\tools\archive
+ $start_time = Get-Date
+ cd C:\tools
+ bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND https://github.com/msys2/msys2-installer/releases/download/2020-09-03/msys2-base-x86_64-20200903.sfx.exe C:\tools\base.exe
+ Write-Output "Download time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
+ C:\tools\base.exe -y
+ ((Get-Content -path C:\tools\msys64\etc\\post-install\\07-pacman-key.post -Raw) -replace '--refresh-keys', '--version') | Set-Content -Path C:\tools\msys64\etc\\post-install\\07-pacman-key.post
+ C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
+ C:\tools\msys64\usr\bin\bash.exe -lc "export"
+ C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Sy
+ echo Y | C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Suu --overwrite=*
+ taskkill /F /FI "MODULES eq msys-2.0.dll"
+ tasklist
+ C:\tools\msys64\usr\bin\bash.exe -lc "mv -f /etc/pacman.conf.pacnew /etc/pacman.conf || true"
+ C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Suu --overwrite=*"
+ C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed \
+ diffutils git grep make pkg-config sed \
+ mingw-w64-x86_64-python \
+ mingw-w64-x86_64-python-setuptools \
+ mingw-w64-x86_64-toolchain \
+ mingw-w64-x86_64-SDL2 \
+ mingw-w64-x86_64-SDL2_image \
+ mingw-w64-x86_64-gtk3 \
+ mingw-w64-x86_64-glib2 \
+ mingw-w64-x86_64-ninja \
+ mingw-w64-x86_64-jemalloc \
+ mingw-w64-x86_64-lzo2 \
+ mingw-w64-x86_64-zstd \
+ mingw-w64-x86_64-libjpeg-turbo \
+ mingw-w64-x86_64-pixman \
+ mingw-w64-x86_64-libgcrypt \
+ mingw-w64-x86_64-libpng \
+ mingw-w64-x86_64-libssh \
+ mingw-w64-x86_64-libxml2 \
+ mingw-w64-x86_64-snappy \
+ mingw-w64-x86_64-libusb \
+ mingw-w64-x86_64-usbredir \
+ mingw-w64-x86_64-libtasn1 \
+ mingw-w64-x86_64-nettle \
+ mingw-w64-x86_64-cyrus-sasl \
+ mingw-w64-x86_64-curl \
+ mingw-w64-x86_64-gnutls \
+ "
+ C:\tools\msys64\usr\bin\bash.exe -lc "rm -rf /var/cache/pacman/pkg/*"
+ cd C:\tools\msys64
+ echo "Start archive"
+ cmd /C "7z a -ttar . -so | 7z a -txz -simsys2-x86_64.tar C:\tools\archive\msys2-x86_64.tar.xz"
install_script:
- - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
- curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
- - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
- curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz.sig"
- - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
- pacman -U --noconfirm msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
- - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Sy"
- - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
- bash pacman pacman-mirrors msys2-runtime"
- - taskkill /F /IM gpg-agent.exe
- - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Su"
- - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
- base-devel
- git
- mingw-w64-x86_64-python
- mingw-w64-x86_64-python-setuptools
- mingw-w64-x86_64-toolchain
- mingw-w64-x86_64-capstone
- mingw-w64-x86_64-SDL2
- mingw-w64-x86_64-SDL2_image
- mingw-w64-x86_64-gtk3
- mingw-w64-x86_64-glib2
- mingw-w64-x86_64-ninja
- mingw-w64-x86_64-make
- mingw-w64-x86_64-lzo2
- mingw-w64-x86_64-zstd
- mingw-w64-x86_64-libjpeg-turbo
- mingw-w64-x86_64-pixman
- mingw-w64-x86_64-libgcrypt
- mingw-w64-x86_64-libpng
- mingw-w64-x86_64-libssh
- mingw-w64-x86_64-libxml2
- mingw-w64-x86_64-snappy
- mingw-w64-x86_64-libusb
- mingw-w64-x86_64-usbredir
- mingw-w64-x86_64-libtasn1
- mingw-w64-x86_64-nettle
- mingw-w64-x86_64-cyrus-sasl
- mingw-w64-x86_64-curl
- mingw-w64-x86_64-gnutls
- mingw-w64-x86_64-zstd"
+ - |
+ cd C:\tools
+ cmd /C "7z x C:\tools\archive\msys2-x86_64.tar.xz -so | 7z x -aoa -simsys2-x86_64.tar -ttar -omsys64"
+ C:\tools\msys64\usr\bin\bash.exe -lc "export"
+
script:
- C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure
--python=python3 --ninja=ninja
--target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu"
- - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j$NUMBER_OF_PROCESSORS"
+ - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j8"
test_script:
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make V=1 check"
--
2.28.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v6 2/4] cirrus: msys2/mingw speed is up, add excluded target back
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 1/4] cirrus: Fixing and speedup the msys2/mingw CI Yonggang Luo
@ 2020-10-09 11:07 ` Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 3/4] docs: Fixes build docs on msys2/mingw Yonggang Luo
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Yonggang Luo @ 2020-10-09 11:07 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, QEMU Trivial,
Philippe Mathieu-Daudé,
Yonggang Luo, Paolo Bonzini
The following target are add back:
i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
.cirrus.yml | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index ed3db10e45..d1fe63b343 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -123,8 +123,7 @@ windows_msys2_task:
script:
- C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure
- --python=python3 --ninja=ninja
- --target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu"
+ --python=python3 --ninja=ninja"
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j8"
test_script:
- C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make V=1 check"
--
2.28.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v6 3/4] docs: Fixes build docs on msys2/mingw
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 1/4] cirrus: Fixing and speedup the msys2/mingw CI Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 2/4] cirrus: msys2/mingw speed is up, add excluded target back Yonggang Luo
@ 2020-10-09 11:07 ` Yonggang Luo
2020-10-09 11:08 ` [PATCH v6 4/4] cirrus: Enable doc build " Yonggang Luo
2020-10-09 11:10 ` [PATCH v6 0/4] Improve cirrus msys2 罗勇刚(Yonggang Luo)
4 siblings, 0 replies; 6+ messages in thread
From: Yonggang Luo @ 2020-10-09 11:07 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, QEMU Trivial,
Philippe Mathieu-Daudé,
Yonggang Luo, Paolo Bonzini
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
docs/conf.py | 2 +-
docs/sphinx/kerneldoc.py | 2 +-
scripts/rst-sanitize.py | 21 +++++++++++++++++++++
tests/qapi-schema/meson.build | 5 +++--
4 files changed, 26 insertions(+), 4 deletions(-)
create mode 100644 scripts/rst-sanitize.py
diff --git a/docs/conf.py b/docs/conf.py
index 00e1b750e2..e584f68393 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -241,7 +241,7 @@ texinfo_documents = [
# We use paths starting from qemu_docdir here so that you can run
# sphinx-build from anywhere and the kerneldoc extension can still
# find everything.
-kerneldoc_bin = os.path.join(qemu_docdir, '../scripts/kernel-doc')
+kerneldoc_bin = ['perl', os.path.join(qemu_docdir, '../scripts/kernel-doc')]
kerneldoc_srctree = os.path.join(qemu_docdir, '..')
hxtool_srctree = os.path.join(qemu_docdir, '..')
qapidoc_srctree = os.path.join(qemu_docdir, '..')
diff --git a/docs/sphinx/kerneldoc.py b/docs/sphinx/kerneldoc.py
index 3e87940206..3ac277d162 100644
--- a/docs/sphinx/kerneldoc.py
+++ b/docs/sphinx/kerneldoc.py
@@ -67,7 +67,7 @@ class KernelDocDirective(Directive):
def run(self):
env = self.state.document.settings.env
- cmd = [env.config.kerneldoc_bin, '-rst', '-enable-lineno']
+ cmd = env.config.kerneldoc_bin + ['-rst', '-enable-lineno']
filename = env.config.kerneldoc_srctree + '/' + self.arguments[0]
export_file_patterns = []
diff --git a/scripts/rst-sanitize.py b/scripts/rst-sanitize.py
new file mode 100644
index 0000000000..26060f1208
--- /dev/null
+++ b/scripts/rst-sanitize.py
@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+
+#
+# Script for remove cr line ending in file
+#
+# Authors:
+# Yonggang Luo <luoyonggang@gmail.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or, at your option, any later version. See the COPYING file in
+# the top-level directory.
+
+import sys
+
+def main(_program, file, *unused):
+ with open(file, 'rb') as content_file:
+ content = content_file.read()
+ sys.stdout.buffer.write(content.replace(b'\r', b''))
+
+if __name__ == "__main__":
+ main(*sys.argv)
diff --git a/tests/qapi-schema/meson.build b/tests/qapi-schema/meson.build
index f08c902911..a6832634b3 100644
--- a/tests/qapi-schema/meson.build
+++ b/tests/qapi-schema/meson.build
@@ -250,18 +250,19 @@ qapi_doc_out = custom_target('QAPI rST doc',
# using an explicit '\' character in the command arguments to
# a custom_target(), as Meson will unhelpfully replace it with a '/'
# (https://github.com/mesonbuild/meson/issues/1564)
+rst_sanitize_cmd = [find_program('../../scripts/rst-sanitize.py'), '@INPUT@']
qapi_doc_out_nocr = custom_target('QAPI rST doc newline-sanitized',
output: ['doc-good.txt.nocr'],
input: qapi_doc_out[0],
build_by_default: build_docs,
- command: ['perl', '-pe', '$x = chr 13; s/$x$//', '@INPUT@'],
+ command: rst_sanitize_cmd,
capture: true)
qapi_doc_ref_nocr = custom_target('QAPI rST doc reference newline-sanitized',
output: ['doc-good.ref.nocr'],
input: files('doc-good.txt'),
build_by_default: build_docs,
- command: ['perl', '-pe', '$x = chr 13; s/$x$//', '@INPUT@'],
+ command: rst_sanitize_cmd,
capture: true)
if build_docs
--
2.28.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v6 4/4] cirrus: Enable doc build on msys2/mingw
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
` (2 preceding siblings ...)
2020-10-09 11:07 ` [PATCH v6 3/4] docs: Fixes build docs on msys2/mingw Yonggang Luo
@ 2020-10-09 11:08 ` Yonggang Luo
2020-10-09 11:10 ` [PATCH v6 0/4] Improve cirrus msys2 罗勇刚(Yonggang Luo)
4 siblings, 0 replies; 6+ messages in thread
From: Yonggang Luo @ 2020-10-09 11:08 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Thomas Huth, QEMU Trivial,
Philippe Mathieu-Daudé,
Yonggang Luo, Paolo Bonzini
Currently rST depends on old version sphinx-2.x.
Install it by downloading it.
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
.cirrus.yml | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/.cirrus.yml b/.cirrus.yml
index d1fe63b343..599fd299e3 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -110,6 +110,11 @@ windows_msys2_task:
mingw-w64-x86_64-curl \
mingw-w64-x86_64-gnutls \
"
+ bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND `
+ https://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz `
+ C:\tools\mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz
+ C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -U /c/tools/mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz"
+ del C:\tools\mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz
C:\tools\msys64\usr\bin\bash.exe -lc "rm -rf /var/cache/pacman/pkg/*"
cd C:\tools\msys64
echo "Start archive"
--
2.28.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v6 0/4] Improve cirrus msys2
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
` (3 preceding siblings ...)
2020-10-09 11:08 ` [PATCH v6 4/4] cirrus: Enable doc build " Yonggang Luo
@ 2020-10-09 11:10 ` 罗勇刚(Yonggang Luo)
4 siblings, 0 replies; 6+ messages in thread
From: 罗勇刚(Yonggang Luo) @ 2020-10-09 11:10 UTC (permalink / raw)
To: qemu-level, Alex Bennée
Cc: QEMU Trivial, Peter Maydell, Thomas Huth,
Philippe Mathieu-Daudé,
Paolo Bonzini
[-- Attachment #1: Type: text/plain, Size: 1577 bytes --]
Hi, alex, the msys2 main repo are back, so I removed the university mirror
On Fri, Oct 9, 2020 at 7:08 PM Yonggang Luo <luoyonggang@gmail.com> wrote:
>
> Fixes the broken msys2/mingw ci and speed it up.
>
> V5-V6
> Remove the university mirror, the main repo are back.
> rename to setup_script.
> Add fixes for msys2/mingw rST document fixes
>
> V4-V5
> Now the cache are usefull by using 7zip to archive msys64
> Saved about 18min, compare
> https://cirrus-ci.com/task/5093551157542912
> https://cirrus-ci.com/task/6177196538593280
>
> V3-V4
> Using cirrus cache to speed up msys2 ci instead of downloading archive
file
>
> V2-V3
> Add one more patch:
> cirrus: msys2/mingw speed is up, add excluded target back
> Do not build sphinx on windows, that's failing
> set the number of parallel count to fixed number 8
>
> V1-V2
> Resolve the cirrus conflict
>
> Yonggang Luo (4):
> cirrus: Fixing and speedup the msys2/mingw CI
> cirrus: msys2/mingw speed is up, add excluded target back
> docs: Fixes build docs on msys2/mingw
> cirrus: Enable doc build on msys2/mingw
>
> .cirrus.yml | 119 ++++++++++++++++++++--------------
> docs/conf.py | 2 +-
> docs/sphinx/kerneldoc.py | 2 +-
> scripts/rst-sanitize.py | 21 ++++++
> tests/qapi-schema/meson.build | 5 +-
> 5 files changed, 98 insertions(+), 51 deletions(-)
> create mode 100644 scripts/rst-sanitize.py
>
> --
> 2.28.0.windows.1
>
--
此致
礼
罗勇刚
Yours
sincerely,
Yonggang Luo
[-- Attachment #2: Type: text/html, Size: 2049 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-10-09 11:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-09 11:07 [PATCH v6 0/4] Improve cirrus msys2 Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 1/4] cirrus: Fixing and speedup the msys2/mingw CI Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 2/4] cirrus: msys2/mingw speed is up, add excluded target back Yonggang Luo
2020-10-09 11:07 ` [PATCH v6 3/4] docs: Fixes build docs on msys2/mingw Yonggang Luo
2020-10-09 11:08 ` [PATCH v6 4/4] cirrus: Enable doc build " Yonggang Luo
2020-10-09 11:10 ` [PATCH v6 0/4] Improve cirrus msys2 罗勇刚(Yonggang Luo)
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.