From: Johannes Schindelin <johannes.schindelin@gmx.de> When this developer tested how the git-sdk-64-minimal artifact could be served to all the GitHub workflow runs that need it, Azure Blobs looked like a pretty good choice: it is reliable, fast and we already use it in Git for Windows to serve components like OpenSSL, cURL, etc It came as an unpleasant surprise just _how many_ times this artifact was downloaded. It exploded the bandwidth to a point where the free tier would no longer be enough, threatening to block other, essential Git for Windows services. Let's switch back to using the Build Artifacts of our trusty Azure Pipeline for the time being. To avoid unnecessary hammering of the Azure Pipeline artifacts, we use the GitHub Action `actions/upload-artifact` in the `windows-build` job and the GitHub Action `actions/download-artifact` in the `windows-test` and `vs-test` jobs (the latter now depends on `windows-build` for that reason, too). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> --- ci: avoid pounding on the poor ci-artifacts container I did not think through how I designed the Windows job initializing the minimal subset of Git for Windows' SDK (containing e.g. make, gcc etc): it currently accesses Azure Blobs and uses way too much bandwidth, blowing right through my quota. So for now, all the Windows/Visual Studio builds will be failing to download that SDK subset. With this patch, we use the (slower) method of downloading the git-sdk-64-minimal Build Artifact of our Azure Pipeline again, which fixes that particular problem. To be sure, it is a shame that we now spend around a whole minute downloading that SDK subset instead of those sweet, sweet seven seconds. But what does not work does not work, and that's that. Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-632%2Fdscho%2Favoid-ci-artifacts-for-now-git.git-v1 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-632/dscho/avoid-ci-artifacts-for-now-git.git-v1 Pull-Request: https://github.com/gitgitgadget/git/pull/632 .github/workflows/main.yml | 65 +++++++++++++++++++++++++++++++------- 1 file changed, 54 insertions(+), 11 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index fd4df939b50..e2e1611aa22 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -12,7 +12,24 @@ jobs: - uses: actions/checkout@v1 - name: download git-sdk-64-minimal shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + run: | + ## Add `json_pp` to the search path + PATH=$PATH:/usr/bin/core_perl + + ## Get artifact + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | + json_pp | + sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') + download_url="$(curl "$urlbase/$id/artifacts" | + json_pp | + sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" + curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ + -o artifacts.zip "$download_url" + + ## Unzip and remove the artifact + unzip artifacts.zip + rm artifacts.zip - name: build shell: powershell env: @@ -30,6 +47,11 @@ jobs: with: name: windows-artifacts path: artifacts + - name: upload git-sdk-64-minimal + uses: actions/upload-artifact@v1 + with: + name: git-sdk-64-minimal + path: git-sdk-64-minimal windows-test: runs-on: windows-latest needs: [windows-build] @@ -38,9 +60,6 @@ jobs: nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] steps: - uses: actions/checkout@v1 - - name: download git-sdk-64-minimal - shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - - name: download build artifacts uses: actions/download-artifact@v1 with: @@ -49,6 +68,11 @@ jobs: - name: extract build artifacts shell: bash run: tar xf artifacts.tar.gz + - name: download git-sdk-64-minimal + uses: actions/download-artifact@v1 + with: + name: git-sdk-64-minimal + path: ${{github.workspace}}/git-sdk-64-minimal/ - name: test shell: powershell run: | @@ -79,7 +103,24 @@ jobs: - uses: actions/checkout@v1 - name: download git-sdk-64-minimal shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + run: | + ## Add `json_pp` to the search path + PATH=$PATH:/usr/bin/core_perl + + ## Get artifact + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | + json_pp | + sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') + download_url="$(curl "$urlbase/$id/artifacts" | + json_pp | + sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" + curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ + -o artifacts.zip "$download_url" + + ## Unzip and remove the artifact + unzip artifacts.zip + rm artifacts.zip - name: generate Visual Studio solution shell: powershell run: | @@ -119,15 +160,17 @@ jobs: path: artifacts vs-test: runs-on: windows-latest - needs: [vs-build] + needs: [vs-build, windows-build] strategy: matrix: nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] steps: - uses: actions/checkout@v1 - - name: download git-64-portable - shell: bash - run: a=git-64-portable && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + - name: download git-sdk-64-minimal + uses: actions/download-artifact@v1 + with: + name: git-sdk-64-minimal + path: ${{github.workspace}}/git-sdk-64-minimal/ - name: download build artifacts uses: actions/download-artifact@v1 with: @@ -143,9 +186,9 @@ jobs: NO_SVN_TESTS: 1 GIT_TEST_SKIP_REBASE_P: 1 run: | - & git-64-portable\git-cmd.exe --command=usr\bin\bash.exe -lc @" + & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @" # Let Git ignore the SDK and the test-cache - printf '%s\n' /git-64-portable/ /test-cache/ >>.git/info/exclude + printf '%s\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude cd t && PATH=\"`$PWD/helper:`$PATH\" && base-commit: f72f328bc57e1b0db380ef76e0c1e94a9ed0ac7c -- gitgitgadget
"Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>
writes:
> Let's switch back to using the Build Artifacts of our trusty Azure
> Pipeline for the time being.
>
> To avoid unnecessary hammering of the Azure Pipeline artifacts, we use
> the GitHub Action `actions/upload-artifact` in the `windows-build` job
> and the GitHub Action `actions/download-artifact` in the `windows-test`
> and `vs-test` jobs (the latter now depends on `windows-build` for that
> reason, too).
I guess this answers a question I sent earlier to the list (our
mails almost crossed, I guess, as two of us were looking at the same
problem at around the same time?).
Hopefully when cmake-for-windows-build topic lands, this can go away
altogether, but that is probably at least 8 weeks away (3 weeks
remaining before the next cycle opens, plus a half of 10 week per
cycle for a typical major release).
Today's final integration (these days I'm pushing out twice or three
times a day) contains this one, and it seems to have passed ;-)
Thanks.
On 2020-05-12 20:47:10+0000, Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com> wrote: > - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - > + run: | > + ## Add `json_pp` to the search path > + PATH=$PATH:/usr/bin/core_perl > + > + ## Get artifact > + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds > + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | > + json_pp | > + sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') > + download_url="$(curl "$urlbase/$id/artifacts" | > + json_pp | > + sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" Hi Dscho, I wonder if it's acceptable to introduce jq (already installed in GitHub Actions and Travis) into our codebase (only in GitHub Actions). If yes, I think this will be easier to follow than depending on static number of space and sed branching. ---------------------8<----------------- diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index e2e1611aa2..482df46651 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -13,17 +13,12 @@ jobs: - name: download git-sdk-64-minimal shell: bash run: | - ## Add `json_pp` to the search path - PATH=$PATH:/usr/bin/core_perl - ## Get artifact urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | - json_pp | - sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') + jq -r ".value[] | .id") download_url="$(curl "$urlbase/$id/artifacts" | - json_pp | - sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ -o artifacts.zip "$download_url" @@ -104,17 +99,12 @@ jobs: - name: download git-sdk-64-minimal shell: bash run: | - ## Add `json_pp` to the search path - PATH=$PATH:/usr/bin/core_perl - ## Get artifact urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | - json_pp | - sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') + jq -r ".value[] | .id") download_url="$(curl "$urlbase/$id/artifacts" | - json_pp | - sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ -o artifacts.zip "$download_url" ----------->8------------ -- Danh
From: Johannes Schindelin <johannes.schindelin@gmx.de> When this developer tested how the git-sdk-64-minimal artifact could be served to all the GitHub workflow runs that need it, Azure Blobs looked like a pretty good choice: it is reliable, fast and we already use it in Git for Windows to serve components like OpenSSL, cURL, etc It came as an unpleasant surprise just _how many_ times this artifact was downloaded. It exploded the bandwidth to a point where the free tier would no longer be enough, threatening to block other, essential Git for Windows services. Let's switch back to using the Build Artifacts of our trusty Azure Pipeline for the time being. To avoid unnecessary hammering of the Azure Pipeline artifacts, we use the GitHub Action `actions/upload-artifact` in the `windows-build` job and the GitHub Action `actions/download-artifact` in the `windows-test` and `vs-test` jobs (the latter now depends on `windows-build` for that reason, too). Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> --- ci: avoid pounding on the poor ci-artifacts container I did not think through how I designed the Windows job initializing the minimal subset of Git for Windows' SDK (containing e.g. make, gcc etc): it currently accesses Azure Blobs and uses way too much bandwidth, blowing right through my quota. So for now, all the Windows/Visual Studio builds will be failing to download that SDK subset. With this patch, we use the (slower) method of downloading the git-sdk-64-minimal Build Artifact of our Azure Pipeline again, which fixes that particular problem. To be sure, it is a shame that we now spend around a whole minute downloading that SDK subset instead of those sweet, sweet seven seconds. But what does not work does not work, and that's that. Change since v1: * Simplifying the scripted code by using jq (thanks, Danh!) Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-632%2Fdscho%2Favoid-ci-artifacts-for-now-git.git-v2 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-632/dscho/avoid-ci-artifacts-for-now-git.git-v2 Pull-Request: https://github.com/gitgitgadget/git/pull/632 Range-diff vs v1: 1: 274ae816be6 ! 1: 14606cc2ed7 ci: avoid pounding on the poor ci-artifacts container @@ Commit message and `vs-test` jobs (the latter now depends on `windows-build` for that reason, too). + Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> ## .github/workflows/main.yml ## @@ .github/workflows/main.yml: jobs: shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + run: | -+ ## Add `json_pp` to the search path -+ PATH=$PATH:/usr/bin/core_perl -+ + ## Get artifact + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | -+ json_pp | -+ sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') ++ jq -r ".value[] | .id") + download_url="$(curl "$urlbase/$id/artifacts" | -+ json_pp | -+ sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" ++ jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" + curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ + -o artifacts.zip "$download_url" + @@ .github/workflows/main.yml: jobs: shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + run: | -+ ## Add `json_pp` to the search path -+ PATH=$PATH:/usr/bin/core_perl -+ + ## Get artifact + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | -+ json_pp | -+ sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') ++ jq -r ".value[] | .id") + download_url="$(curl "$urlbase/$id/artifacts" | -+ json_pp | -+ sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" ++ jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" + curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ + -o artifacts.zip "$download_url" + .github/workflows/main.yml | 55 ++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index fd4df939b50..482df466510 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -12,7 +12,19 @@ jobs: - uses: actions/checkout@v1 - name: download git-sdk-64-minimal shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + run: | + ## Get artifact + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | + jq -r ".value[] | .id") + download_url="$(curl "$urlbase/$id/artifacts" | + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" + curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ + -o artifacts.zip "$download_url" + + ## Unzip and remove the artifact + unzip artifacts.zip + rm artifacts.zip - name: build shell: powershell env: @@ -30,6 +42,11 @@ jobs: with: name: windows-artifacts path: artifacts + - name: upload git-sdk-64-minimal + uses: actions/upload-artifact@v1 + with: + name: git-sdk-64-minimal + path: git-sdk-64-minimal windows-test: runs-on: windows-latest needs: [windows-build] @@ -38,9 +55,6 @@ jobs: nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] steps: - uses: actions/checkout@v1 - - name: download git-sdk-64-minimal - shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - - name: download build artifacts uses: actions/download-artifact@v1 with: @@ -49,6 +63,11 @@ jobs: - name: extract build artifacts shell: bash run: tar xf artifacts.tar.gz + - name: download git-sdk-64-minimal + uses: actions/download-artifact@v1 + with: + name: git-sdk-64-minimal + path: ${{github.workspace}}/git-sdk-64-minimal/ - name: test shell: powershell run: | @@ -79,7 +98,19 @@ jobs: - uses: actions/checkout@v1 - name: download git-sdk-64-minimal shell: bash - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + run: | + ## Get artifact + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | + jq -r ".value[] | .id") + download_url="$(curl "$urlbase/$id/artifacts" | + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" + curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ + -o artifacts.zip "$download_url" + + ## Unzip and remove the artifact + unzip artifacts.zip + rm artifacts.zip - name: generate Visual Studio solution shell: powershell run: | @@ -119,15 +150,17 @@ jobs: path: artifacts vs-test: runs-on: windows-latest - needs: [vs-build] + needs: [vs-build, windows-build] strategy: matrix: nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] steps: - uses: actions/checkout@v1 - - name: download git-64-portable - shell: bash - run: a=git-64-portable && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - + - name: download git-sdk-64-minimal + uses: actions/download-artifact@v1 + with: + name: git-sdk-64-minimal + path: ${{github.workspace}}/git-sdk-64-minimal/ - name: download build artifacts uses: actions/download-artifact@v1 with: @@ -143,9 +176,9 @@ jobs: NO_SVN_TESTS: 1 GIT_TEST_SKIP_REBASE_P: 1 run: | - & git-64-portable\git-cmd.exe --command=usr\bin\bash.exe -lc @" + & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @" # Let Git ignore the SDK and the test-cache - printf '%s\n' /git-64-portable/ /test-cache/ >>.git/info/exclude + printf '%s\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude cd t && PATH=\"`$PWD/helper:`$PATH\" && base-commit: f72f328bc57e1b0db380ef76e0c1e94a9ed0ac7c -- gitgitgadget
Hi Junio, On Tue, 12 May 2020, Junio C Hamano wrote: > "Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com> > writes: > > > Let's switch back to using the Build Artifacts of our trusty Azure > > Pipeline for the time being. > > > > To avoid unnecessary hammering of the Azure Pipeline artifacts, we use > > the GitHub Action `actions/upload-artifact` in the `windows-build` job > > and the GitHub Action `actions/download-artifact` in the `windows-test` > > and `vs-test` jobs (the latter now depends on `windows-build` for that > > reason, too). > > I guess this answers a question I sent earlier to the list (our > mails almost crossed, I guess, as two of us were looking at the same > problem at around the same time?). I am terribly sorry, but I did not get to read the Git mailing list at all this week (or for that matter, my private mail). So I would not even have seen your message... :-( > Hopefully when cmake-for-windows-build topic lands, this can go away > altogether, but that is probably at least 8 weeks away (3 weeks > remaining before the next cycle opens, plus a half of 10 week per > cycle for a typical major release). The `cmake-for-windows-build` would address only the build part for Visual Studio. The regular Windows build, as well as the parallelized tests _still_ need the `git-sdk-64-minimal` artifact. With or without CMake. That's because neither CMake nor Visual Studio can accommodate the fact that our test suite is implemented in shell script _and_ requires a working Perl interpreter. > Today's final integration (these days I'm pushing out twice or three > times a day) contains this one, and it seems to have passed ;-) Excellent! Thanks, Dscho
[-- Attachment #1: Type: text/plain, Size: 3828 bytes --] Hi Danh, On Wed, 13 May 2020, Đoàn Trần Công Danh wrote: > On 2020-05-12 20:47:10+0000, Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com> wrote: > > - run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf - > > + run: | > > + ## Add `json_pp` to the search path > > + PATH=$PATH:/usr/bin/core_perl > > + > > + ## Get artifact > > + urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds > > + id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | > > + json_pp | > > + sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') > > + download_url="$(curl "$urlbase/$id/artifacts" | > > + json_pp | > > + sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" > > Hi Dscho, > > I wonder if it's acceptable to introduce jq (already installed in > GitHub Actions and Travis) into our codebase (only in GitHub Actions). > > If yes, I think this will be easier to follow than depending on static > number of space and sed branching. > ---------------------8<----------------- > diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml > index e2e1611aa2..482df46651 100644 > --- a/.github/workflows/main.yml > +++ b/.github/workflows/main.yml > @@ -13,17 +13,12 @@ jobs: > - name: download git-sdk-64-minimal > shell: bash > run: | > - ## Add `json_pp` to the search path > - PATH=$PATH:/usr/bin/core_perl > - > ## Get artifact > urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds > id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | > - json_pp | > - sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') > + jq -r ".value[] | .id") > download_url="$(curl "$urlbase/$id/artifacts" | > - json_pp | > - sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" > + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" > curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ > -o artifacts.zip "$download_url" > > @@ -104,17 +99,12 @@ jobs: > - name: download git-sdk-64-minimal > shell: bash > run: | > - ## Add `json_pp` to the search path > - PATH=$PATH:/usr/bin/core_perl > - > ## Get artifact > urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds > id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" | > - json_pp | > - sed -n 's/^ "id" : \([1-9][0-9]*\).*/\1/p') > + jq -r ".value[] | .id") > download_url="$(curl "$urlbase/$id/artifacts" | > - json_pp | > - sed -n '/^ {/{:1;N;/\n }/b2;b1;:2;/"name" : "git-sdk-64-minimal"/{s/.*"downloadUrl" : "\([^\"]*\).*/\1/p}}')" > + jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')" > curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \ > -o artifacts.zip "$download_url" > > ----------->8------------ Thank you for that. Indeed, I should have checked whether `jq` is available on the Windows build agents (and then I should have learned how to hold that tool right). I sent out a v2 with this change, and I already merged that version into Git for Windows' `master` branch in preparation for v2.27.0-rc0. Thanks, Dscho
"Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>
writes:
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> When this developer tested how the git-sdk-64-minimal artifact could be
> served to all the GitHub workflow runs that need it, Azure Blobs looked
> like a pretty good choice: it is reliable, fast and we already use it in
> Git for Windows to serve components like OpenSSL, cURL, etc
>
> It came as an unpleasant surprise just _how many_ times this artifact
> was downloaded. It exploded the bandwidth to a point where the free tier
> would no longer be enough, threatening to block other, essential Git for
> Windows services.
>
> Let's switch back to using the Build Artifacts of our trusty Azure
> Pipeline for the time being.
>
> To avoid unnecessary hammering of the Azure Pipeline artifacts, we use
> the GitHub Action `actions/upload-artifact` in the `windows-build` job
> and the GitHub Action `actions/download-artifact` in the `windows-test`
> and `vs-test` jobs (the latter now depends on `windows-build` for that
> reason, too).
>
> Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>
> Change since v1:
>
> * Simplifying the scripted code by using jq (thanks, Danh!)
Thanks, both. Will replace and let's merge it down before -rc1.
[-- Attachment #1: Type: text/plain, Size: 1546 bytes --] Hi Junio, On Fri, 15 May 2020, Junio C Hamano wrote: > "Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com> > writes: > > > From: Johannes Schindelin <johannes.schindelin@gmx.de> > > > > When this developer tested how the git-sdk-64-minimal artifact could be > > served to all the GitHub workflow runs that need it, Azure Blobs looked > > like a pretty good choice: it is reliable, fast and we already use it in > > Git for Windows to serve components like OpenSSL, cURL, etc > > > > It came as an unpleasant surprise just _how many_ times this artifact > > was downloaded. It exploded the bandwidth to a point where the free tier > > would no longer be enough, threatening to block other, essential Git for > > Windows services. > > > > Let's switch back to using the Build Artifacts of our trusty Azure > > Pipeline for the time being. > > > > To avoid unnecessary hammering of the Azure Pipeline artifacts, we use > > the GitHub Action `actions/upload-artifact` in the `windows-build` job > > and the GitHub Action `actions/download-artifact` in the `windows-test` > > and `vs-test` jobs (the latter now depends on `windows-build` for that > > reason, too). > > > > Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> > > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> > > --- > > > > Change since v1: > > > > * Simplifying the scripted code by using jq (thanks, Danh!) > > Thanks, both. Will replace and let's merge it down before -rc1. Thank you! Dscho
Johannes Schindelin <Johannes.Schindelin@gmx.de> writes: > Hi Junio, > > On Fri, 15 May 2020, Junio C Hamano wrote: > >> Thanks, both. Will replace and let's merge it down before -rc1. 'next' passes with this https://github.com/git/git/runs/679473788 https://github.com/git/git/runs/679473866 etc. Thanks.