[PATCH v7 0/4] Improve cirrus msys2

[PATCH v7 0/4] Improve cirrus msys2

[Yonggang Luo]

Fixes the broken msys2/mingw ci and speed it up.

V6-V7
* cirrus: Enable doc build on msys2/mingw
* docs: Fixes build docs on msys2/mingw
* gitignore: ignore a bit more
* scripts: Convert qemu-version.sh to qemu-version.py

V5-V6
Remove the university mirror, the main repo are back.
rename to setup_script.
Add fixes for msys2/mingw rST document fixes

V4-V5
Now the cache are usefull by using 7zip to archive msys64
Saved about 18min, compare
https://cirrus-ci.com/task/5093551157542912
https://cirrus-ci.com/task/6177196538593280

V3-V4
Using cirrus cache to speed up msys2 ci instead of downloading archive file

V2-V3
Add one more patch:
cirrus: msys2/mingw speed is up, add excluded target back
Do not build sphinx on windows, that's failing
set the number of parallel count to fixed number 8

V1-V2
Resolve the cirrus conflict

Yonggang Luo (4):
  scripts: Convert qemu-version.sh to qemu-version.py
  gitignore: ignore a bit more
  docs: Fixes build docs on msys2/mingw
  cirrus: Enable doc build on msys2/mingw

 .cirrus.yml                   |  6 +++++-
 .gitignore                    |  4 +++-
 docs/conf.py                  |  2 +-
 docs/sphinx/kerneldoc.py      |  2 +-
 meson.build                   |  2 +-
 scripts/qemu-version.py       | 37 +++++++++++++++++++++++++++++++++++
 scripts/qemu-version.sh       | 25 -----------------------
 scripts/rst-sanitize.py       | 21 ++++++++++++++++++++
 tests/qapi-schema/meson.build |  5 +++--
 9 files changed, 72 insertions(+), 32 deletions(-)
 create mode 100644 scripts/qemu-version.py
 delete mode 100755 scripts/qemu-version.sh
 create mode 100644 scripts/rst-sanitize.py

-- 
2.28.0.windows.1

[PATCH v7 1/4] scripts: Convert qemu-version.sh to qemu-version.py

[Yonggang Luo]

The sh script are harder to maintain for compatible different
xsh environment so convert it to python script
Also incorporate the fixes in
https://patchew.org/QEMU/20200929143654.518157-1-marcandre.lureau@redhat.com/

According to https://github.com/msys2/MSYS2-packages/issues/2176
We need use CYGWIN=noglob and MSYS=noglob in the environment variable
for disable wildcard expanding in msys or cygwin git, and setting the shell=False

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Message-Id: <20201006112139.700-1-luoyonggang@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 meson.build             |  2 +-
 scripts/qemu-version.py | 37 +++++++++++++++++++++++++++++++++++++
 scripts/qemu-version.sh | 25 -------------------------
 3 files changed, 38 insertions(+), 26 deletions(-)
 create mode 100644 scripts/qemu-version.py
 delete mode 100755 scripts/qemu-version.sh

diff --git a/meson.build b/meson.build
index 17c89c87c6..c23167c61b 100644
--- a/meson.build
+++ b/meson.build
@@ -1132,7 +1132,7 @@ tracetool = [
    '--backend=' + config_host['TRACE_BACKENDS']
 ]
 
-qemu_version_cmd = [find_program('scripts/qemu-version.sh'),
+qemu_version_cmd = [find_program('scripts/qemu-version.py'),
                     meson.current_source_dir(),
                     config_host['PKGVERSION'], meson.project_version()]
 qemu_version = custom_target('qemu-version.h',
diff --git a/scripts/qemu-version.py b/scripts/qemu-version.py
new file mode 100644
index 0000000000..cf97b2bbb5
--- /dev/null
+++ b/scripts/qemu-version.py
@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+
+#
+# Script for retrieve qemu git version information
+#
+# Authors:
+#  Yonggang Luo <luoyonggang@gmail.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or, at your option, any later version.  See the COPYING file in
+# the top-level directory.
+
+import sys
+import subprocess
+import os
+import os.path
+
+def main(_program, dir, pkgversion, version, *unused):
+    os.chdir(dir)
+    if not pkgversion and os.path.exists('.git'):
+        pc = subprocess.run(
+            ['git', 'describe', '--match', 'v*', '--dirty', '--always'],
+            env=dict(os.environ, CYGWIN="noglob", MSYS='noglob'),
+            stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
+            encoding='utf8', shell=False)
+        if pc.returncode == 0:
+            pkgversion = pc.stdout.strip()
+
+    fullversion = version
+    if pkgversion:
+        fullversion = "{} ({})".format(version, pkgversion)
+
+    print('#define QEMU_PKGVERSION "%s"' % pkgversion)
+    print('#define QEMU_FULL_VERSION "%s"' % fullversion)
+
+if __name__ == "__main__":
+    main(*sys.argv)
diff --git a/scripts/qemu-version.sh b/scripts/qemu-version.sh
deleted file mode 100755
index 03128c56a2..0000000000
--- a/scripts/qemu-version.sh
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/bin/sh
-
-set -eu
-
-dir="$1"
-pkgversion="$2"
-version="$3"
-
-if [ -z "$pkgversion" ]; then
-    cd "$dir"
-    if [ -e .git ]; then
-        pkgversion=$(git describe --match 'v*' --dirty | echo "")
-    fi
-fi
-
-if [ -n "$pkgversion" ]; then
-    fullversion="$version ($pkgversion)"
-else
-    fullversion="$version"
-fi
-
-cat <<EOF
-#define QEMU_PKGVERSION "$pkgversion"
-#define QEMU_FULL_VERSION "$fullversion"
-EOF
-- 
2.28.0.windows.1

[PATCH v7 2/4] gitignore: ignore a bit more

[Yonggang Luo]

Enable the creating multiple build directory at the source root.
Ignore /meson/ and /roms/ for better search experience.

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
 .gitignore | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/.gitignore b/.gitignore
index b32bca1315..f78ee9f297 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,5 @@
 /GNUmakefile
-/build/
+/build*/
 *.pyc
 .sdk
 .stgit-*
@@ -10,3 +10,5 @@ TAGS
 *~
 *.ast_raw
 *.depend_raw
+/meson/
+/roms/**/*
-- 
2.28.0.windows.1

[PATCH v7 3/4] docs: Fixes build docs on msys2/mingw

[Yonggang Luo]

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
 docs/conf.py                  |  2 +-
 docs/sphinx/kerneldoc.py      |  2 +-
 scripts/rst-sanitize.py       | 21 +++++++++++++++++++++
 tests/qapi-schema/meson.build |  5 +++--
 4 files changed, 26 insertions(+), 4 deletions(-)
 create mode 100644 scripts/rst-sanitize.py

diff --git a/docs/conf.py b/docs/conf.py
index 00e1b750e2..e584f68393 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -241,7 +241,7 @@ texinfo_documents = [
 # We use paths starting from qemu_docdir here so that you can run
 # sphinx-build from anywhere and the kerneldoc extension can still
 # find everything.
-kerneldoc_bin = os.path.join(qemu_docdir, '../scripts/kernel-doc')
+kerneldoc_bin = ['perl', os.path.join(qemu_docdir, '../scripts/kernel-doc')]
 kerneldoc_srctree = os.path.join(qemu_docdir, '..')
 hxtool_srctree = os.path.join(qemu_docdir, '..')
 qapidoc_srctree = os.path.join(qemu_docdir, '..')
diff --git a/docs/sphinx/kerneldoc.py b/docs/sphinx/kerneldoc.py
index 3e87940206..3ac277d162 100644
--- a/docs/sphinx/kerneldoc.py
+++ b/docs/sphinx/kerneldoc.py
@@ -67,7 +67,7 @@ class KernelDocDirective(Directive):
 
     def run(self):
         env = self.state.document.settings.env
-        cmd = [env.config.kerneldoc_bin, '-rst', '-enable-lineno']
+        cmd = env.config.kerneldoc_bin + ['-rst', '-enable-lineno']
 
         filename = env.config.kerneldoc_srctree + '/' + self.arguments[0]
         export_file_patterns = []
diff --git a/scripts/rst-sanitize.py b/scripts/rst-sanitize.py
new file mode 100644
index 0000000000..26060f1208
--- /dev/null
+++ b/scripts/rst-sanitize.py
@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+
+#
+# Script for remove cr line ending in file
+#
+# Authors:
+#  Yonggang Luo <luoyonggang@gmail.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or, at your option, any later version.  See the COPYING file in
+# the top-level directory.
+
+import sys
+
+def main(_program, file, *unused):
+    with open(file, 'rb') as content_file:
+        content = content_file.read()
+        sys.stdout.buffer.write(content.replace(b'\r', b''))
+
+if __name__ == "__main__":
+    main(*sys.argv)
diff --git a/tests/qapi-schema/meson.build b/tests/qapi-schema/meson.build
index 1f222a7a13..20a7641af8 100644
--- a/tests/qapi-schema/meson.build
+++ b/tests/qapi-schema/meson.build
@@ -251,18 +251,19 @@ qapi_doc_out = custom_target('QAPI rST doc',
 # using an explicit '\' character in the command arguments to
 # a custom_target(), as Meson will unhelpfully replace it with a '/'
 # (https://github.com/mesonbuild/meson/issues/1564)
+rst_sanitize_cmd = [find_program('../../scripts/rst-sanitize.py'), '@INPUT@']
 qapi_doc_out_nocr = custom_target('QAPI rST doc newline-sanitized',
                                   output: ['doc-good.txt.nocr'],
                                   input: qapi_doc_out[0],
                                   build_by_default: build_docs,
-                                  command: ['perl', '-pe', '$x = chr 13; s/$x$//', '@INPUT@'],
+                                  command: rst_sanitize_cmd,
                                   capture: true)
 
 qapi_doc_ref_nocr = custom_target('QAPI rST doc reference newline-sanitized',
                                   output: ['doc-good.ref.nocr'],
                                   input: files('doc-good.txt'),
                                   build_by_default: build_docs,
-                                  command: ['perl', '-pe', '$x = chr 13; s/$x$//', '@INPUT@'],
+                                  command: rst_sanitize_cmd,
                                   capture: true)
 
 if build_docs
-- 
2.28.0.windows.1

[PATCH v7 4/4] cirrus: Enable doc build on msys2/mingw

[Yonggang Luo]

Currently rST depends on old version sphinx-2.x.
Install it by downloading it.
Remove the need of university mirror, the main repo are recovered.

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
 .cirrus.yml | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 99d118239c..f42ccb956a 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -76,7 +76,6 @@ windows_msys2_task:
         ((Get-Content -path C:\tools\msys64\etc\\post-install\\07-pacman-key.post -Raw) -replace '--refresh-keys', '--version') | Set-Content -Path C:\tools\msys64\etc\\post-install\\07-pacman-key.post
         C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
         C:\tools\msys64\usr\bin\bash.exe -lc "export"
-        C:\tools\msys64\usr\bin\bash.exe -lc "grep -rl 'repo.msys2.org/' /etc/pacman.d/mirrorlist.* | xargs sed -i 's/repo.msys2.org\//mirrors.tuna.tsinghua.edu.cn\/msys2\//g'"
         C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Sy
         echo Y | C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Suu --overwrite=*
         taskkill /F /FI "MODULES eq msys-2.0.dll"
@@ -111,6 +110,11 @@ windows_msys2_task:
           mingw-w64-x86_64-curl \
           mingw-w64-x86_64-gnutls \
           "
+        bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND `
+          https://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz `
+          C:\tools\mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz
+        C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -U /c/tools/mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz"
+        del C:\tools\mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz
         C:\tools\msys64\usr\bin\bash.exe -lc "rm -rf /var/cache/pacman/pkg/*"
         cd C:\tools\msys64
         echo "Start archive"
-- 
2.28.0.windows.1

Re: [PATCH v7 2/4] gitignore: ignore a bit more

[Philippe Mathieu-Daudé]

On 10/13/20 1:37 AM, Yonggang Luo wrote:
> Enable the creating multiple build directory at the source root.
> Ignore /meson/ and /roms/ for better search experience.
> 
> Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
> ---
>   .gitignore | 4 +++-
>   1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitignore b/.gitignore
> index b32bca1315..f78ee9f297 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -1,5 +1,5 @@
>   /GNUmakefile
> -/build/
> +/build*/
>   *.pyc
>   .sdk
>   .stgit-*
> @@ -10,3 +10,5 @@ TAGS
>   *~
>   *.ast_raw
>   *.depend_raw
> +/meson/
> +/roms/**/*

Why?

Re: [PATCH v7 2/4] gitignore: ignore a bit more

[罗勇刚(Yonggang Luo)]

[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

On Tue, Oct 13, 2020 at 4:38 PM Philippe Mathieu-Daudé <philmd@redhat.com>
wrote:
>
> On 10/13/20 1:37 AM, Yonggang Luo wrote:
> > Enable the creating multiple build directory at the source root.
> > Ignore /meson/ and /roms/ for better search experience.
> >
> > Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
> > ---
> >   .gitignore | 4 +++-
> >   1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/.gitignore b/.gitignore
> > index b32bca1315..f78ee9f297 100644
> > --- a/.gitignore
> > +++ b/.gitignore
> > @@ -1,5 +1,5 @@
> >   /GNUmakefile
> > -/build/
> > +/build*/
> >   *.pyc
> >   .sdk
> >   .stgit-*
> > @@ -10,3 +10,5 @@ TAGS
> >   *~
> >   *.ast_raw
> >   *.depend_raw
> > +/meson/
> > +/roms/**/*
>
> Why?
As I said, help for searching tool ignore it, meson and roms are thirdparty
packages.
>


--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo

[-- Attachment #2: Type: text/html, Size: 1330 bytes --]

Re: [PATCH v7 2/4] gitignore: ignore a bit more

[Thomas Huth]

On 13/10/2020 10.53, 罗勇刚(Yonggang Luo) wrote:
> 
> 
> On Tue, Oct 13, 2020 at 4:38 PM Philippe Mathieu-Daudé <philmd@redhat.com
> <mailto:philmd@redhat.com>> wrote:
>>
>> On 10/13/20 1:37 AM, Yonggang Luo wrote:
>> > Enable the creating multiple build directory at the source root.
>> > Ignore /meson/ and /roms/ for better search experience.
>> >
>> > Signed-off-by: Yonggang Luo <luoyonggang@gmail.com
> <mailto:luoyonggang@gmail.com>>
>> > ---
>> >   .gitignore | 4 +++-
>> >   1 file changed, 3 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/.gitignore b/.gitignore
>> > index b32bca1315..f78ee9f297 100644
>> > --- a/.gitignore
>> > +++ b/.gitignore
>> > @@ -1,5 +1,5 @@
>> >   /GNUmakefile
>> > -/build/
>> > +/build*/

The naming of your private build directories is IMHO something that should
not be part of the public .gitignore file. I think you can use your
~/.gitignore file for that.

>> >   *.pyc
>> >   .sdk
>> >   .stgit-*
>> > @@ -10,3 +10,5 @@ TAGS
>> >   *~
>> >   *.ast_raw
>> >   *.depend_raw
>> > +/meson/
>> > +/roms/**/*
>>
>> Why?
> As I said, help for searching tool ignore it, meson and roms are thirdparty
> packages.

Hmm, but "git status" should still show whether there are modifications in
the subdirectories, so IMHO this should not be ignored?

 Thomas

Re: [PATCH v7 2/4] gitignore: ignore a bit more

[罗勇刚(Yonggang Luo)]

[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]

On Tue, Oct 13, 2020 at 5:02 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 13/10/2020 10.53, 罗勇刚(Yonggang Luo) wrote:
> >
> >
> > On Tue, Oct 13, 2020 at 4:38 PM Philippe Mathieu-Daudé <
philmd@redhat.com
> > <mailto:philmd@redhat.com>> wrote:
> >>
> >> On 10/13/20 1:37 AM, Yonggang Luo wrote:
> >> > Enable the creating multiple build directory at the source root.
> >> > Ignore /meson/ and /roms/ for better search experience.
> >> >
> >> > Signed-off-by: Yonggang Luo <luoyonggang@gmail.com
> > <mailto:luoyonggang@gmail.com>>
> >> > ---
> >> >   .gitignore | 4 +++-
> >> >   1 file changed, 3 insertions(+), 1 deletion(-)
> >> >
> >> > diff --git a/.gitignore b/.gitignore
> >> > index b32bca1315..f78ee9f297 100644
> >> > --- a/.gitignore
> >> > +++ b/.gitignore
> >> > @@ -1,5 +1,5 @@
> >> >   /GNUmakefile
> >> > -/build/
> >> > +/build*/
>
> The naming of your private build directories is IMHO something that should
> not be part of the public .gitignore file. I think you can use your
> ~/.gitignore file for that.
>
> >> >   *.pyc
> >> >   .sdk
> >> >   .stgit-*
> >> > @@ -10,3 +10,5 @@ TAGS
> >> >   *~
> >> >   *.ast_raw
> >> >   *.depend_raw
> >> > +/meson/
> >> > +/roms/**/*
> >>
> >> Why?
> > As I said, help for searching tool ignore it, meson and roms are
thirdparty
> > packages.
>
> Hmm, but "git status" should still show whether there are modifications in
> the subdirectories, so IMHO this should not be ignored?
Yeap, it's doens't affect git, just affect search tools. If this is
improper, skip it, I can use it locally
>
>  Thomas
>


--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo

[-- Attachment #2: Type: text/html, Size: 2536 bytes --]