All of lore.kernel.org
 help / color / mirror / Atom feed
From: Yonggang Luo <luoyonggang@gmail.com>
To: qemu-devel@nongnu.org
Cc: Paolo Bonzini <pbonzini@redhat.com>,
	Sunil Muthuswamy <sunilmut@microsoft.com>,
	Ed Maste <emaste@freebsd.org>,
	Yonggang Luo <luoyonggang@gmail.com>
Subject: [PATCH 2/4] cirrus/msys2: Cache msys2 mingw in a better way.
Date: Wed,  6 Jan 2021 21:38:23 -0800	[thread overview]
Message-ID: <20210107053825.2010-3-luoyonggang@gmail.com> (raw)
In-Reply-To: <20210107053825.2010-1-luoyonggang@gmail.com>

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
 .cirrus.yml | 117 ++++++++++++++++++++++++++++++----------------------
 1 file changed, 68 insertions(+), 49 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index ff6adabd0d..6f2a958472 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -58,23 +58,61 @@ windows_msys2_task:
     CIRRUS_SHELL: powershell
     MSYS: winsymlinks:nativestrict
     MSYSTEM: MINGW64
+    MSYS2_URL: https://github.com/msys2/msys2-installer/releases/download/2021-01-05/msys2-base-x86_64-20210105.sfx.exe
+    MSYS2_FINGERPRINT: 0
+    MSYS2_PACKAGES: "
+      diffutils git grep make pkg-config sed
+      mingw-w64-x86_64-python
+      mingw-w64-x86_64-python-sphinx
+      mingw-w64-x86_64-toolchain
+      mingw-w64-x86_64-SDL2
+      mingw-w64-x86_64-SDL2_image
+      mingw-w64-x86_64-gtk3
+      mingw-w64-x86_64-glib2
+      mingw-w64-x86_64-ninja
+      mingw-w64-x86_64-jemalloc
+      mingw-w64-x86_64-lzo2
+      mingw-w64-x86_64-zstd
+      mingw-w64-x86_64-libjpeg-turbo
+      mingw-w64-x86_64-pixman
+      mingw-w64-x86_64-libgcrypt
+      mingw-w64-x86_64-libpng
+      mingw-w64-x86_64-libssh
+      mingw-w64-x86_64-libxml2
+      mingw-w64-x86_64-snappy
+      mingw-w64-x86_64-libusb
+      mingw-w64-x86_64-usbredir
+      mingw-w64-x86_64-libtasn1
+      mingw-w64-x86_64-nettle
+      mingw-w64-x86_64-cyrus-sasl
+      mingw-w64-x86_64-curl
+      mingw-w64-x86_64-gnutls
+      mingw-w64-x86_64-libnfs
+    "
     CHERE_INVOKING: 1
-  setup_script:
-    - choco install -y --no-progress 7zip
-    - Write-Output $env:PATH
   msys2_cache:
     folder: C:\tools\archive
     reupload_on_changes: false
-    fingerprint_script: cat .cirrus.yml
+    # These env variables are used to generate fingerprint to trigger the cache procedure
+    # If wanna to force re-populate msys2, increase MSYS2_FINGERPRINT
+    fingerprint_script:
+      - |
+        echo $env:CIRRUS_TASK_NAME
+        echo $env:MSYS2_URL
+        echo $env:MSYS2_FINGERPRINT
+        echo $env:MSYS2_PACKAGES
     populate_script:
       - |
-        md C:\tools
-        md C:\tools\archive
+        md -Force C:\tools\archive\pkg
         $start_time = Get-Date
+        bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND $env:MSYS2_URL C:\tools\archive\base.exe
+        Write-Output "Download time taken: $((Get-Date).Subtract($start_time))"
         cd C:\tools
-        bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND https://github.com/msys2/msys2-installer/releases/download/2020-09-03/msys2-base-x86_64-20200903.sfx.exe C:\tools\base.exe
-        Write-Output "Download time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
-        C:\tools\base.exe -y
+        C:\tools\archive\base.exe -y
+        del -Force C:\tools\archive\base.exe
+        Write-Output "Base install time taken: $((Get-Date).Subtract($start_time))"
+        $start_time = Get-Date
+
         ((Get-Content -path C:\tools\msys64\etc\\post-install\\07-pacman-key.post -Raw) -replace '--refresh-keys', '--version') | Set-Content -Path C:\tools\msys64\etc\\post-install\\07-pacman-key.post
         C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
         C:\tools\msys64\usr\bin\bash.exe -lc "export"
@@ -84,49 +122,30 @@ windows_msys2_task:
         tasklist
         C:\tools\msys64\usr\bin\bash.exe -lc "mv -f /etc/pacman.conf.pacnew /etc/pacman.conf || true"
         C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Suu --overwrite=*"
-        C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed \
-          diffutils git grep make pkg-config sed \
-          mingw-w64-x86_64-python \
-          mingw-w64-x86_64-toolchain \
-          mingw-w64-x86_64-SDL2 \
-          mingw-w64-x86_64-SDL2_image \
-          mingw-w64-x86_64-gtk3 \
-          mingw-w64-x86_64-glib2 \
-          mingw-w64-x86_64-ninja \
-          mingw-w64-x86_64-jemalloc \
-          mingw-w64-x86_64-lzo2 \
-          mingw-w64-x86_64-zstd \
-          mingw-w64-x86_64-libjpeg-turbo \
-          mingw-w64-x86_64-pixman \
-          mingw-w64-x86_64-libgcrypt \
-          mingw-w64-x86_64-libpng \
-          mingw-w64-x86_64-libssh \
-          mingw-w64-x86_64-libxml2 \
-          mingw-w64-x86_64-snappy \
-          mingw-w64-x86_64-libusb \
-          mingw-w64-x86_64-usbredir \
-          mingw-w64-x86_64-libtasn1 \
-          mingw-w64-x86_64-nettle \
-          mingw-w64-x86_64-cyrus-sasl \
-          mingw-w64-x86_64-curl \
-          mingw-w64-x86_64-gnutls \
-          mingw-w64-x86_64-libnfs \
-          "
-        bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND `
-          https://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz `
-          C:\tools\mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz
-        C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -U /c/tools/mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz"
-        del C:\tools\mingw-w64-x86_64-python-sphinx-2.3.1-1-any.pkg.tar.xz
-        C:\tools\msys64\usr\bin\bash.exe -lc "rm -rf /var/cache/pacman/pkg/*"
-        cd C:\tools\msys64
-        echo "Start archive"
-        cmd /C "7z a -ttar . -so | 7z a -txz -simsys2-x86_64.tar C:\tools\archive\msys2-x86_64.tar.xz"
+        Write-Output "Core install time taken: $((Get-Date).Subtract($start_time))"
+        $start_time = Get-Date
+
+        C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed $env:MSYS2_PACKAGES"
+        Write-Output "Package install time taken: $((Get-Date).Subtract($start_time))"
+        $start_time = Get-Date
+
+        del -Force -ErrorAction SilentlyContinue C:\tools\msys64\etc\mtab
+        del -Force -ErrorAction SilentlyContinue C:\tools\msys64\dev\fd
+        del -Force -ErrorAction SilentlyContinue C:\tools\msys64\dev\stderr
+        del -Force -ErrorAction SilentlyContinue C:\tools\msys64\dev\stdin
+        del -Force -ErrorAction SilentlyContinue C:\tools\msys64\dev\stdout
+        del -Force -Recurse -ErrorAction SilentlyContinue C:\tools\msys64\var\cache\pacman\pkg
+        tar cf C:\tools\archive\msys64.tar -C C:\tools\ msys64
+
+        Write-Output "Package archive time taken: $((Get-Date).Subtract($start_time))"
+        del -Force -Recurse -ErrorAction SilentlyContinue c:\tools\msys64 
   install_script:
     - |
+      $start_time = Get-Date
       cd C:\tools
-      cmd /C "7z x C:\tools\archive\msys2-x86_64.tar.xz -so | 7z x -aoa -simsys2-x86_64.tar -ttar -omsys64"
-      C:\tools\msys64\usr\bin\bash.exe -lc "export"
-
+      ls C:\tools\archive\msys64.tar
+      tar xf C:\tools\archive\msys64.tar
+      Write-Output "Extract msys2 time taken: $((Get-Date).Subtract($start_time))"
   script:
     - C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure --python=python3"
-- 
2.29.2.windows.3



  parent reply	other threads:[~2021-01-07  5:41 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-07  5:38 [PATCH 0/4] Fixes whpx build and improve Windows host CI Yonggang Luo
2021-01-07  5:38 ` [PATCH 1/4] cirrus/msys2: Exit powershell with $LastExitCode Yonggang Luo
2021-01-07  6:58   ` Philippe Mathieu-Daudé
2021-01-07  5:38 ` Yonggang Luo [this message]
2021-01-07  5:38 ` [PATCH 3/4] whpx: Fixes include of whp-dispatch.h in whpx.h Yonggang Luo
2021-01-07  7:22   ` Philippe Mathieu-Daudé
2021-01-07  5:38 ` [PATCH 4/4] maintainers: Add me as Windows Hosted Continuous Integration maintainer Yonggang Luo
2021-01-07  7:20   ` Philippe Mathieu-Daudé

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210107053825.2010-3-luoyonggang@gmail.com \
    --to=luoyonggang@gmail.com \
    --cc=emaste@freebsd.org \
    --cc=pbonzini@redhat.com \
    --cc=qemu-devel@nongnu.org \
    --cc=sunilmut@microsoft.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.