From: "Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: Johannes Schindelin <johannes.schindelin@gmx.de>,
Johannes Schindelin <johannes.schindelin@gmx.de>
Subject: [PATCH v2] ci: avoid pounding on the poor ci-artifacts container
Date: Fri, 15 May 2020 07:55:18 +0000 [thread overview]
Message-ID: <pull.632.v2.git.1589529318969.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.632.git.1589316430595.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
When this developer tested how the git-sdk-64-minimal artifact could be
served to all the GitHub workflow runs that need it, Azure Blobs looked
like a pretty good choice: it is reliable, fast and we already use it in
Git for Windows to serve components like OpenSSL, cURL, etc
It came as an unpleasant surprise just _how many_ times this artifact
was downloaded. It exploded the bandwidth to a point where the free tier
would no longer be enough, threatening to block other, essential Git for
Windows services.
Let's switch back to using the Build Artifacts of our trusty Azure
Pipeline for the time being.
To avoid unnecessary hammering of the Azure Pipeline artifacts, we use
the GitHub Action `actions/upload-artifact` in the `windows-build` job
and the GitHub Action `actions/download-artifact` in the `windows-test`
and `vs-test` jobs (the latter now depends on `windows-build` for that
reason, too).
Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
ci: avoid pounding on the poor ci-artifacts container
I did not think through how I designed the Windows job initializing the
minimal subset of Git for Windows' SDK (containing e.g. make, gcc etc):
it currently accesses Azure Blobs and uses way too much bandwidth,
blowing right through my quota.
So for now, all the Windows/Visual Studio builds will be failing to
download that SDK subset.
With this patch, we use the (slower) method of downloading the
git-sdk-64-minimal Build Artifact of our Azure Pipeline again, which
fixes that particular problem.
To be sure, it is a shame that we now spend around a whole minute
downloading that SDK subset instead of those sweet, sweet seven seconds.
But what does not work does not work, and that's that.
Change since v1:
* Simplifying the scripted code by using jq (thanks, Danh!)
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-632%2Fdscho%2Favoid-ci-artifacts-for-now-git.git-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-632/dscho/avoid-ci-artifacts-for-now-git.git-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/632
Range-diff vs v1:
1: 274ae816be6 ! 1: 14606cc2ed7 ci: avoid pounding on the poor ci-artifacts container
@@ Commit message
and `vs-test` jobs (the latter now depends on `windows-build` for that
reason, too).
+ Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
## .github/workflows/main.yml ##
@@ .github/workflows/main.yml: jobs:
shell: bash
- run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+ run: |
-+ ## Add `json_pp` to the search path
-+ PATH=$PATH:/usr/bin/core_perl
-+
+ ## Get artifact
+ urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+ id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-+ json_pp |
-+ sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p')
++ jq -r ".value[] | .id")
+ download_url="$(curl "$urlbase/$id/artifacts" |
-+ json_pp |
-+ sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"
++ jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
+ curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+ -o artifacts.zip "$download_url"
+
@@ .github/workflows/main.yml: jobs:
shell: bash
- run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+ run: |
-+ ## Add `json_pp` to the search path
-+ PATH=$PATH:/usr/bin/core_perl
-+
+ ## Get artifact
+ urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+ id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-+ json_pp |
-+ sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p')
++ jq -r ".value[] | .id")
+ download_url="$(curl "$urlbase/$id/artifacts" |
-+ json_pp |
-+ sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')"
++ jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
+ curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+ -o artifacts.zip "$download_url"
+
.github/workflows/main.yml | 55 ++++++++++++++++++++++++++++++--------
1 file changed, 44 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index fd4df939b50..482df466510 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -12,7 +12,19 @@ jobs:
- uses: actions/checkout@v1
- name: download git-sdk-64-minimal
shell: bash
- run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+ run: |
+ ## Get artifact
+ urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+ id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
+ jq -r ".value[] | .id")
+ download_url="$(curl "$urlbase/$id/artifacts" |
+ jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
+ curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+ -o artifacts.zip "$download_url"
+
+ ## Unzip and remove the artifact
+ unzip artifacts.zip
+ rm artifacts.zip
- name: build
shell: powershell
env:
@@ -30,6 +42,11 @@ jobs:
with:
name: windows-artifacts
path: artifacts
+ - name: upload git-sdk-64-minimal
+ uses: actions/upload-artifact@v1
+ with:
+ name: git-sdk-64-minimal
+ path: git-sdk-64-minimal
windows-test:
runs-on: windows-latest
needs: [windows-build]
@@ -38,9 +55,6 @@ jobs:
nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
steps:
- uses: actions/checkout@v1
- - name: download git-sdk-64-minimal
- shell: bash
- run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
- name: download build artifacts
uses: actions/download-artifact@v1
with:
@@ -49,6 +63,11 @@ jobs:
- name: extract build artifacts
shell: bash
run: tar xf artifacts.tar.gz
+ - name: download git-sdk-64-minimal
+ uses: actions/download-artifact@v1
+ with:
+ name: git-sdk-64-minimal
+ path: ${{github.workspace}}/git-sdk-64-minimal/
- name: test
shell: powershell
run: |
@@ -79,7 +98,19 @@ jobs:
- uses: actions/checkout@v1
- name: download git-sdk-64-minimal
shell: bash
- run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+ run: |
+ ## Get artifact
+ urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
+ id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
+ jq -r ".value[] | .id")
+ download_url="$(curl "$urlbase/$id/artifacts" |
+ jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
+ curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
+ -o artifacts.zip "$download_url"
+
+ ## Unzip and remove the artifact
+ unzip artifacts.zip
+ rm artifacts.zip
- name: generate Visual Studio solution
shell: powershell
run: |
@@ -119,15 +150,17 @@ jobs:
path: artifacts
vs-test:
runs-on: windows-latest
- needs: [vs-build]
+ needs: [vs-build, windows-build]
strategy:
matrix:
nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
steps:
- uses: actions/checkout@v1
- - name: download git-64-portable
- shell: bash
- run: a=git-64-portable && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -
+ - name: download git-sdk-64-minimal
+ uses: actions/download-artifact@v1
+ with:
+ name: git-sdk-64-minimal
+ path: ${{github.workspace}}/git-sdk-64-minimal/
- name: download build artifacts
uses: actions/download-artifact@v1
with:
@@ -143,9 +176,9 @@ jobs:
NO_SVN_TESTS: 1
GIT_TEST_SKIP_REBASE_P: 1
run: |
- & git-64-portable\git-cmd.exe --command=usr\bin\bash.exe -lc @"
+ & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @"
# Let Git ignore the SDK and the test-cache
- printf '%s\n' /git-64-portable/ /test-cache/ >>.git/info/exclude
+ printf '%s\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude
cd t &&
PATH=\"`$PWD/helper:`$PATH\" &&
base-commit: f72f328bc57e1b0db380ef76e0c1e94a9ed0ac7c
--
gitgitgadget
next prev parent reply other threads:[~2020-05-15 7:55 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-12 20:47 [PATCH] ci: avoid pounding on the poor ci-artifacts container Johannes Schindelin via GitGitGadget
2020-05-12 23:39 ` Junio C Hamano
2020-05-15 13:59 ` Johannes Schindelin
2020-05-13 12:44 ` Đoàn Trần Công Danh
2020-05-15 14:01 ` Johannes Schindelin
2020-05-15 7:55 ` Johannes Schindelin via GitGitGadget [this message]
2020-05-15 15:09 ` [PATCH v2] " Junio C Hamano
2020-05-15 21:35 ` Johannes Schindelin
2020-05-15 21:38 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=pull.632.v2.git.1589529318969.gitgitgadget@gmail.com \
--to=gitgitgadget@gmail.com \
--cc=git@vger.kernel.org \
--cc=johannes.schindelin@gmx.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).