[PATCH 0/2] Improve cirrus msys2

[PATCH 0/2] Improve cirrus msys2

[Yonggang Luo]

Fixes the broken msys2/mingw ci and speed it up.

Yonggang Luo (2):
  docker: Add win32/msys2/mingw64 docker
  cirrus: Fixing and speedup the msys2/mingw CI

 .cirrus.yml                           | 60 +++++++--------------------
 tests/docker/dockerfiles/msys2.docker | 57 +++++++++++++++++++++++++
 2 files changed, 73 insertions(+), 44 deletions(-)
 create mode 100644 tests/docker/dockerfiles/msys2.docker

-- 
2.28.0.windows.1

[PATCH 1/2] docker: Add win32/msys2/mingw64 docker

[Yonggang Luo]

This docker is used to preparing a msys2/mingw with basic software installed.

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
 tests/docker/dockerfiles/msys2.docker | 57 +++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 tests/docker/dockerfiles/msys2.docker

diff --git a/tests/docker/dockerfiles/msys2.docker b/tests/docker/dockerfiles/msys2.docker
new file mode 100644
index 0000000000..ce1404cbb8
--- /dev/null
+++ b/tests/docker/dockerfiles/msys2.docker
@@ -0,0 +1,57 @@
+FROM mcr.microsoft.com/windows/servercore:2004
+RUN cmd /S /C powershell -NoLogo -NoProfile -Command netsh interface ipv4 show interfaces ; netsh interface ipv4 set subinterface 18 mtu=1460 store=persistent ; netsh interface ipv4 show interfaces ; Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1')) ;
+RUN choco install -y --no-progress git 7zip
+RUN cmd /S /C powershell -NoLogo -NoProfile -Command Remove-Item C:\ProgramData\chocolatey\logs -Force -Recurse ; Remove-Item C:\Users\ContainerAdministrator\AppData\Local\Temp -Force -Recurse
+RUN mkdir C:\tools
+RUN bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND https://github.com/msys2/msys2-installer/releases/download/2020-09-03/msys2-base-x86_64-20200903.sfx.exe C:\tools\base.exe
+RUN cd /d C:\tools && base.exe -y
+RUN cmd /S /C powershell -NoLogo -NoProfile -Command "((Get-Content -path C:\tools\msys64\etc\\post-install\\07-pacman-key.post -Raw) -replace '--refresh-keys', '--version') | Set-Content -Path C:\tools\msys64\etc\\post-install\\07-pacman-key.post"
+RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
+RUN C:\tools\msys64\usr\bin\bash.exe -lc "echo $MSYSTEM" >nul 2>&1
+# RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/Server = http:\/\/repo.msys2.org\/msys\/.arch\///g' /etc/pacman.d/mirrorlist.msys"
+# RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/Server = http:\/\/repo.msys2.org\/mingw\/i686\///g' /etc/pacman.d/mirrorlist.mingw32"
+# RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/Server = http:\/\/repo.msys2.org\/mingw\/x86_64\///g' /etc/pacman.d/mirrorlist.mingw64"
+RUN C:\tools\msys64\usr\bin\bash.exe -lc "grep -rl 'repo.msys2.org/' /etc/pacman.d/mirrorlist.* | xargs sed -i 's/repo.msys2.org\//mirrors.tuna.tsinghua.edu.cn\/msys2\//g'"
+RUN C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Sy
+RUN echo Y | C:\tools\msys64\usr\bin\pacman.exe -Suu --noconfirm --needed
+RUN taskkill /F /FI "MODULES eq msys-2.0.dll"
+RUN tasklist
+RUN C:\tools\msys64\usr\bin\bash.exe -lc "pacman.exe --noconfirm -S --needed \
+base-devel \
+git \
+mingw-w64-x86_64-python \
+mingw-w64-x86_64-python-setuptools \
+mingw-w64-x86_64-toolchain \
+mingw-w64-x86_64-SDL2 \
+mingw-w64-x86_64-SDL2_image \
+mingw-w64-x86_64-gtk3 \
+mingw-w64-x86_64-glib2 \
+mingw-w64-x86_64-ninja \
+mingw-w64-x86_64-make \
+mingw-w64-x86_64-jemalloc \
+mingw-w64-x86_64-lzo2 \
+mingw-w64-x86_64-zstd \
+mingw-w64-x86_64-libjpeg-turbo \
+mingw-w64-x86_64-pixman \
+mingw-w64-x86_64-libgcrypt \
+mingw-w64-x86_64-libpng \
+mingw-w64-x86_64-libssh \
+mingw-w64-x86_64-libxml2 \
+mingw-w64-x86_64-snappy \
+mingw-w64-x86_64-libusb \
+mingw-w64-x86_64-usbredir \
+mingw-w64-x86_64-libtasn1 \
+mingw-w64-x86_64-nettle \
+mingw-w64-x86_64-cyrus-sasl \
+mingw-w64-x86_64-curl \
+mingw-w64-x86_64-gnutls \
+mingw-w64-x86_64-zstd"
+
+RUN C:\tools\msys64\usr\bin\bash.exe -lc "rm -rf /var/cache/pacman/pkg/*"
+RUN del C:\tools\base.exe
+RUN cd C:\tools\msys64 && cmd /C "7z a -ttar . -so | 7z a -txz -simsys2-x86_64.tar C:\tools\msys2-x86_64.tar.xz"
+# docker build --tag lygstate/windowsservercore:msys2 -f "msys2.docker" .
+# docker run -it --rm -v c:/work:c:/work lygstate/windowsservercore:msys2 cmd
+# docker push lygstate/windowsservercore:msys2
+# no cache
+# docker build --no-cache --tag lygstate/windowsservercore:msys2 -f "msys2.docker" .
-- 
2.28.0.windows.1

[PATCH 2/2] cirrus: Fixing and speedup the msys2/mingw CI

[Yonggang Luo]

Using a prepared https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz
for speed up the msys2/mingw prepare speed on the cirrus CI.
This file are generated by the msys2.docker

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
 .cirrus.yml | 60 ++++++++++++++---------------------------------------
 1 file changed, 16 insertions(+), 44 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 5363142373..1c2a05abd4 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -47,59 +47,31 @@ macos_xcode_task:
 
 windows_msys2_task:
   windows_container:
-    image: cirrusci/windowsservercore:cmake
+    image: cirrusci/windowsservercore:2019
     os_version: 2019
     cpu: 8
     memory: 8G
   env:
+    CIRRUS_SHELL: powershell
     MSYS: winsymlinks:nativestrict
     MSYSTEM: MINGW64
     CHERE_INVOKING: 1
-  printenv_script:
-    - C:\tools\msys64\usr\bin\bash.exe -lc 'printenv'
   install_script:
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
-        curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
-        curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz.sig"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
-        pacman -U --noconfirm msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Sy"
+    - |
+      Write-Output $env:PATH
+      md C:\tools
+      $start_time = Get-Date
+      bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz C:\tools\msys2-x86_64.tar.xz
+      Write-Output "Download time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
+      $start_time = Get-Date
+      cd C:\tools
+      choco install -y --no-progress 7zip
+      cmd /C "7z x msys2-x86_64.tar.xz -so | 7z x -aoa -simsys2-x86_64.tar -ttar -omsys64"
+      Write-Output "Extract time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
     - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
-        bash pacman pacman-mirrors msys2-runtime"
-    - taskkill /F /IM gpg-agent.exe
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Su"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
-        base-devel
-        git
-        mingw-w64-x86_64-python
-        mingw-w64-x86_64-python-setuptools
-        mingw-w64-x86_64-toolchain
-        mingw-w64-x86_64-capstone
-        mingw-w64-x86_64-SDL2
-        mingw-w64-x86_64-SDL2_image
-        mingw-w64-x86_64-gtk3
-        mingw-w64-x86_64-glib2
-        mingw-w64-x86_64-ninja
-        mingw-w64-x86_64-make
-        mingw-w64-x86_64-lzo2
-        mingw-w64-x86_64-zstd
-        mingw-w64-x86_64-libjpeg-turbo
-        mingw-w64-x86_64-pixman
-        mingw-w64-x86_64-libgcrypt
-        mingw-w64-x86_64-libpng
-        mingw-w64-x86_64-libssh
-        mingw-w64-x86_64-libxml2
-        mingw-w64-x86_64-snappy
-        mingw-w64-x86_64-libusb
-        mingw-w64-x86_64-usbredir
-        mingw-w64-x86_64-libtasn1
-        mingw-w64-x86_64-libnfs
-        mingw-w64-x86_64-nettle
-        mingw-w64-x86_64-cyrus-sasl
-        mingw-w64-x86_64-curl
-        mingw-w64-x86_64-gnutls
-        mingw-w64-x86_64-zstd"
+      mingw-w64-x86_64-python-sphinx
+      mingw-w64-x86_64-libnfs"
+
   script:
     - C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure
-- 
2.28.0.windows.1

Re: [PATCH 1/2] docker: Add win32/msys2/mingw64 docker

[Thomas Huth]

On 07/10/2020 04.41, Yonggang Luo wrote:
> This docker is used to preparing a msys2/mingw with basic software installed.
> 
> Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
> ---
>  tests/docker/dockerfiles/msys2.docker | 57 +++++++++++++++++++++++++++
>  1 file changed, 57 insertions(+)
>  create mode 100644 tests/docker/dockerfiles/msys2.docker
> 
> diff --git a/tests/docker/dockerfiles/msys2.docker b/tests/docker/dockerfiles/msys2.docker
> new file mode 100644
> index 0000000000..ce1404cbb8
> --- /dev/null
> +++ b/tests/docker/dockerfiles/msys2.docker
> @@ -0,0 +1,57 @@
> +FROM mcr.microsoft.com/windows/servercore:2004
> +RUN cmd /S /C powershell -NoLogo -NoProfile -Command netsh interface ipv4 show interfaces ; netsh interface ipv4 set subinterface 18 mtu=1460 store=persistent ; netsh interface ipv4 show interfaces ; Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1')) ;
> +RUN choco install -y --no-progress git 7zip
> +RUN cmd /S /C powershell -NoLogo -NoProfile -Command Remove-Item C:\ProgramData\chocolatey\logs -Force -Recurse ; Remove-Item C:\Users\ContainerAdministrator\AppData\Local\Temp -Force -Recurse
> +RUN mkdir C:\tools
> +RUN bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND https://github.com/msys2/msys2-installer/releases/download/2020-09-03/msys2-base-x86_64-20200903.sfx.exe C:\tools\base.exe
> +RUN cd /d C:\tools && base.exe -y
> +RUN cmd /S /C powershell -NoLogo -NoProfile -Command "((Get-Content -path C:\tools\msys64\etc\\post-install\\07-pacman-key.post -Raw) -replace '--refresh-keys', '--version') | Set-Content -Path C:\tools\msys64\etc\\post-install\\07-pacman-key.post"
> +RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
> +RUN C:\tools\msys64\usr\bin\bash.exe -lc "echo $MSYSTEM" >nul 2>&1
> +# RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/Server = http:\/\/repo.msys2.org\/msys\/.arch\///g' /etc/pacman.d/mirrorlist.msys"
> +# RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/Server = http:\/\/repo.msys2.org\/mingw\/i686\///g' /etc/pacman.d/mirrorlist.mingw32"
> +# RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/Server = http:\/\/repo.msys2.org\/mingw\/x86_64\///g' /etc/pacman.d/mirrorlist.mingw64"
> +RUN C:\tools\msys64\usr\bin\bash.exe -lc "grep -rl 'repo.msys2.org/' /etc/pacman.d/mirrorlist.* | xargs sed -i 's/repo.msys2.org\//mirrors.tuna.tsinghua.edu.cn\/msys2\//g'"
> +RUN C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Sy
> +RUN echo Y | C:\tools\msys64\usr\bin\pacman.exe -Suu --noconfirm --needed
> +RUN taskkill /F /FI "MODULES eq msys-2.0.dll"
> +RUN tasklist
> +RUN C:\tools\msys64\usr\bin\bash.exe -lc "pacman.exe --noconfirm -S --needed \
> +base-devel \

base-devel pulls in a lot of stuff that we do not really need (automake...),
can you please replace it with the packages that are really required? See:

https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00072.html

 Thanks,
  Thomas

Re: [PATCH 1/2] docker: Add win32/msys2/mingw64 docker

[罗勇刚(Yonggang Luo)]

[-- Attachment #1: Type: text/plain, Size: 3362 bytes --]

On Wed, Oct 7, 2020 at 2:43 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 07/10/2020 04.41, Yonggang Luo wrote:
> > This docker is used to preparing a msys2/mingw with basic software
installed.
> >
> > Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
> > ---
> >  tests/docker/dockerfiles/msys2.docker | 57 +++++++++++++++++++++++++++
> >  1 file changed, 57 insertions(+)
> >  create mode 100644 tests/docker/dockerfiles/msys2.docker
> >
> > diff --git a/tests/docker/dockerfiles/msys2.docker
b/tests/docker/dockerfiles/msys2.docker
> > new file mode 100644
> > index 0000000000..ce1404cbb8
> > --- /dev/null
> > +++ b/tests/docker/dockerfiles/msys2.docker
> > @@ -0,0 +1,57 @@
> > +FROM mcr.microsoft.com/windows/servercore:2004
> > +RUN cmd /S /C powershell -NoLogo -NoProfile -Command netsh interface
ipv4 show interfaces ; netsh interface ipv4 set subinterface 18 mtu=1460
store=persistent ; netsh interface ipv4 show interfaces ;
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object
System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
;
> > +RUN choco install -y --no-progress git 7zip
> > +RUN cmd /S /C powershell -NoLogo -NoProfile -Command Remove-Item
C:\ProgramData\chocolatey\logs -Force -Recurse ; Remove-Item
C:\Users\ContainerAdministrator\AppData\Local\Temp -Force -Recurse
> > +RUN mkdir C:\tools
> > +RUN bitsadmin /transfer msys_download /dynamic /download /priority
FOREGROUND
https://github.com/msys2/msys2-installer/releases/download/2020-09-03/msys2-base-x86_64-20200903.sfx.exe
C:\tools\base.exe
> > +RUN cd /d C:\tools && base.exe -y
> > +RUN cmd /S /C powershell -NoLogo -NoProfile -Command "((Get-Content
-path C:\tools\msys64\etc\\post-install\\07-pacman-key.post -Raw) -replace
'--refresh-keys', '--version') | Set-Content -Path
C:\tools\msys64\etc\\post-install\\07-pacman-key.post"
> > +RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i
's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
> > +RUN C:\tools\msys64\usr\bin\bash.exe -lc "echo $MSYSTEM" >nul 2>&1
> > +# RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/Server =
http:\/\/repo.msys2.org\/msys\/.arch\///g' /etc/pacman.d/mirrorlist.msys"
> > +# RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/Server =
http:\/\/repo.msys2.org\/mingw\/i686\///g' /etc/pacman.d/mirrorlist.mingw32"
> > +# RUN C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/Server =
http:\/\/repo.msys2.org\/mingw\/x86_64\///g'
/etc/pacman.d/mirrorlist.mingw64"
> > +RUN C:\tools\msys64\usr\bin\bash.exe -lc "grep -rl 'repo.msys2.org/'
/etc/pacman.d/mirrorlist.* | xargs sed -i 's/repo.msys2.org\//
mirrors.tuna.tsinghua.edu.cn\/msys2\//g'"
> > +RUN C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Sy
> > +RUN echo Y | C:\tools\msys64\usr\bin\pacman.exe -Suu --noconfirm
--needed
> > +RUN taskkill /F /FI "MODULES eq msys-2.0.dll"
> > +RUN tasklist
> > +RUN C:\tools\msys64\usr\bin\bash.exe -lc "pacman.exe --noconfirm -S
--needed \
> > +base-devel \
>
> base-devel pulls in a lot of stuff that we do not really need
(automake...),
> can you please replace it with the packages that are really required? See:
OK
>
> https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00072.html
>
>  Thanks,
>   Thomas
>


--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo

[-- Attachment #2: Type: text/html, Size: 4628 bytes --]

Re: [PATCH 2/2] cirrus: Fixing and speedup the msys2/mingw CI

[Daniel P. Berrangé]

These kind of changes are not appropriate for QEMU trivial.

On Wed, Oct 07, 2020 at 10:41:35AM +0800, Yonggang Luo wrote:
> Using a prepared https://github.com/lygstate/qemu/releases/download/v5.1.0/msys2-x86_64.tar.xz
> for speed up the msys2/mingw prepare speed on the cirrus CI.

I appreciate the desire to speed up CI, but relying on the contents of
a arbitrary blob provided by a user as the build environment is not
satisfatory for QEMU's CI.  The project has no visibility, control
or tracking over what is in that blob.

Any caches need to be managed by the CI system, and populated by
the defined CI job logic.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|