Reducing redundant build at Travis?

Reducing redundant build at Travis?

[Junio C Hamano]

I usually try to stay as late as possible to finish all the
integration branches in order before pushing out the result; it is
more efficient to be able to batch things (for humans).  

I however noticed that This often means we would have multiple build
jobs at Travis for branches and builds on Windows often fails
waiting for its response.  Since I tagged the tip of 'maint', and I
wanted to give all the build a fair chance to succeed without other
build jobs starving it of resources, I pushed out 'maint' and the
tag before others, even though I already have all the other
integration branches ready.

Unfortunately, https://travis-ci.org/git/git/builds/ shows that it
does not care if it spawned a job to build the tip of 'maint' and
another for 'v2.13.3' that point at the same thing.

I do not mind this behaviour terribly for Linux builds that never
seem to time out, but it is wasteful.  Here is what I came up with
to skip the build and test on a branch tip that is tagged, but I
suspect this would not actually work (when told to build a tag, I
suspect that it would try to find an exact-match and ends up finding
itself, refusing to do the work).

I also do not quite like the way that I had to add a check like this
for every script: thing in the .travis.yml file (the attached does
not even cover MacOS whose build tends to take a lot longer).  I
suspect that you or others who are better versed with the system may
have better idea to have a single centralized check that avoids a
tip of a branch that is pointed at by a release tag.

Thanks.

 ci/run-linux32-build.sh | 6 ++++++
 ci/run-windows-build.sh | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh
index e30fb2cddc..ffb5c8fdf1 100755
--- a/ci/run-linux32-build.sh
+++ b/ci/run-linux32-build.sh
@@ -6,6 +6,12 @@
 #   run-linux32-build.sh [host-user-id]
 #
 
+if TAG=$(git describe --exact-match HEAD 2>/dev/null)
+then
+	echo "The tip of the branch is exactly at $TAG"
+	exit 0
+fi
+
 # Update packages to the latest available versions
 linux32 --32bit i386 sh -c '
     apt update >/dev/null &&
diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh
index 2d98f6b2f9..74d4819d74 100755
--- a/ci/run-windows-build.sh
+++ b/ci/run-windows-build.sh
@@ -12,6 +12,12 @@ test -z "$GFW_CI_TOKEN" && echo "GFW_CI_TOKEN not defined" && exit
 BRANCH=$1
 COMMIT=$2
 
+if TAG=$(git describe --exact-match "$COMMIT" 2>/dev/null)
+then
+	echo "Tip of $BRANCH exactly at $TAG"
+	exit 0
+fi
+
 gfwci () {
 	local CURL_ERROR_CODE HTTP_CODE
 	CONTENT_FILE=$(mktemp -t "git-windows-ci-XXXXXX")

Re: Reducing redundant build at Travis?

[Lars Schneider]

On Thu, Jul 13, 2017 at 1:44 AM, Junio C Hamano <gitster@pobox.com> wrote:
> I usually try to stay as late as possible to finish all the
> integration branches in order before pushing out the result; it is
> more efficient to be able to batch things (for humans).
>
> I however noticed that This often means we would have multiple build
> jobs at Travis for branches and builds on Windows often fails

The Windows build has some kind of problem since June 22.
Somehow building gitk-git just blocks the build and waits until
the timeout. I had no time, yet, to investigate this further.


> waiting for its response.  Since I tagged the tip of 'maint', and I
> wanted to give all the build a fair chance to succeed without other
> build jobs starving it of resources, I pushed out 'maint' and the
> tag before others, even though I already have all the other
> integration branches ready.
>
> Unfortunately, https://travis-ci.org/git/git/builds/ shows that it
> does not care if it spawned a job to build the tip of 'maint' and
> another for 'v2.13.3' that point at the same thing.

That is indeed suprising and wasteful. Looks like other people
did run into the same issue. How about something like this?
https://github.com/mockito/mockito/blob/release/2.x/.travis.yml#L26-L29


- Lars

Re: Reducing redundant build at Travis?

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

> On Thu, Jul 13, 2017 at 1:44 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> I usually try to stay as late as possible to finish all the
>> integration branches in order before pushing out the result; it is
>> more efficient to be able to batch things (for humans).
>>
>> I however noticed that This often means we would have multiple build
>> jobs at Travis for branches and builds on Windows often fails
>
> The Windows build has some kind of problem since June 22.
> Somehow building gitk-git just blocks the build and waits until
> the timeout. I had no time, yet, to investigate this further.
>
>
>> waiting for its response.  Since I tagged the tip of 'maint', and I
>> wanted to give all the build a fair chance to succeed without other
>> build jobs starving it of resources, I pushed out 'maint' and the
>> tag before others, even though I already have all the other
>> integration branches ready.
>>
>> Unfortunately, https://travis-ci.org/git/git/builds/ shows that it
>> does not care if it spawned a job to build the tip of 'maint' and
>> another for 'v2.13.3' that point at the same thing.
>
> That is indeed suprising and wasteful. Looks like other people
> did run into the same issue. How about something like this?
> https://github.com/mockito/mockito/blob/release/2.x/.travis.yml#L26-L29

That unfortunately is exactly what I wanted to avoid.

We'd want to test tagged releases, and we'd want to test usual
updates to integration branches.  It just is that sometimes the tips
of integration branches happen to be at the tagged release, so I'd
prefer to always build tags but skip a branch build if it happens to
be also tagged.  After all, none of the integration branches may
directly point at a tagged release when the tag is pushed out.

Re: Reducing redundant build at Travis?

[Jeff King]

On Thu, Jul 13, 2017 at 02:53:17PM -0700, Junio C Hamano wrote:

> >> Unfortunately, https://travis-ci.org/git/git/builds/ shows that it
> >> does not care if it spawned a job to build the tip of 'maint' and
> >> another for 'v2.13.3' that point at the same thing.
> >
> > That is indeed suprising and wasteful. Looks like other people
> > did run into the same issue. How about something like this?
> > https://github.com/mockito/mockito/blob/release/2.x/.travis.yml#L26-L29
> 
> That unfortunately is exactly what I wanted to avoid.
> 
> We'd want to test tagged releases, and we'd want to test usual
> updates to integration branches.  It just is that sometimes the tips
> of integration branches happen to be at the tagged release, so I'd
> prefer to always build tags but skip a branch build if it happens to
> be also tagged.  After all, none of the integration branches may
> directly point at a tagged release when the tag is pushed out.

Right, I think the right solution is some amount of peeling. Recognizing
that the commit sha1 is the same, or better yet, not bothering to retest
trees which have already been tested.

We used a hacked-up copy of Jenkins for our in-house CI, and it skips
already-tested trees.  Besides the discussed cases, this also saves time
when pushing noop rebases (e.g., when you just changed commit messages)
and noop merges (e.g., if you already back-merged master to your topic,
tested it, and now do a "merge --no-ff" to merge the topic in). I don't
think either of those are common in our workflows, though.

If we had some kind of persistent storage, we could do a quick:

  tree=$(git rev-parse HEAD^{tree})
  if test "$(get_from_storage status-$tree)" = "good"
  then
	echo "Already tested $tree, skipping"
	exit 0
  fi
  ... run actual tests ...
  put_into_storage status-$tree good

The "git test" script[1] uses this strategy with git-notes as the
storage, and I've found it quite useful. I don't think we can rely on
git-notes, but I think Travis gives us some storage options. Even just a
best-effort cache directory would probably be sufficient (this is an
optimization, after all).

-Peff

[1] https://github.com/mhagger/git-test

Re: Reducing redundant build at Travis?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Right, I think the right solution is some amount of peeling. Recognizing
> that the commit sha1 is the same, or better yet, not bothering to retest
> trees which have already been tested.

Yup, I also have a hack to avoid testing a version that is only
different in insignificant way (e.g. the only difference being
GIT-VERSION-GEN or Documentation/RelNotes/*) from an installed one
in the script I use after each integration attempt I do, which I use
a few times a day (that's "Meta/Dothem" if anybody is interested).

> If we had some kind of persistent storage, we could do a quick:
> ...
> The "git test" script[1] uses this strategy with git-notes as the
> storage, and I've found it quite useful. I don't think we can rely on
> git-notes, but I think Travis gives us some storage options. Even just a
> best-effort cache directory would probably be sufficient (this is an
> optimization, after all).

We do seem to use some persistence to order prove tests already, but
I do not think it helps the common case, where my end-of-day push
pushes out 'maint' and 'v2.13.3' at the same time, because the push
is made with "git push --follow-tags $there maint master next pu"
and the new tag happens to be at 'maint'.  It would be nice if
Travis runs were sequential, but I often observe that it creates
jobs for these multiple branches and tags pushed at the same time,
and start running a few of them.

Re: Reducing redundant build at Travis?

[Jeff King]

On Fri, Jul 14, 2017 at 07:54:16AM -0700, Junio C Hamano wrote:

> > The "git test" script[1] uses this strategy with git-notes as the
> > storage, and I've found it quite useful. I don't think we can rely on
> > git-notes, but I think Travis gives us some storage options. Even just a
> > best-effort cache directory would probably be sufficient (this is an
> > optimization, after all).
> 
> We do seem to use some persistence to order prove tests already, but
> I do not think it helps the common case, where my end-of-day push
> pushes out 'maint' and 'v2.13.3' at the same time, because the push
> is made with "git push --follow-tags $there maint master next pu"
> and the new tag happens to be at 'maint'.  It would be nice if
> Travis runs were sequential, but I often observe that it creates
> jobs for these multiple branches and tags pushed at the same time,
> and start running a few of them.

Ah, right, I didn't think about how these are racing. You'd need storage
which allows some kind of atomic operation to "claim" the tree as a
work-in-progress (and anybody who loses the race to get the lock would
have to spin waiting for the winner to tell them the real status).

I don't know if Travis's cache storage is up to that challenge. We could
probably build such a lock on top of third-party storage, but things are
rapidly getting more complex.

-Peff

Re: Reducing redundant build at Travis?

[Lars Schneider]

> On 14 Jul 2017, at 17:32, Jeff King <peff@peff.net> wrote:
> 
> On Fri, Jul 14, 2017 at 07:54:16AM -0700, Junio C Hamano wrote:
> 
>>> The "git test" script[1] uses this strategy with git-notes as the
>>> storage, and I've found it quite useful. I don't think we can rely on
>>> git-notes, but I think Travis gives us some storage options. Even just a
>>> best-effort cache directory would probably be sufficient (this is an
>>> optimization, after all).
>> 
>> We do seem to use some persistence to order prove tests already, but
>> I do not think it helps the common case, where my end-of-day push
>> pushes out 'maint' and 'v2.13.3' at the same time, because the push
>> is made with "git push --follow-tags $there maint master next pu"
>> and the new tag happens to be at 'maint'.  It would be nice if
>> Travis runs were sequential, but I often observe that it creates
>> jobs for these multiple branches and tags pushed at the same time,
>> and start running a few of them.
> 
> Ah, right, I didn't think about how these are racing. You'd need storage
> which allows some kind of atomic operation to "claim" the tree as a
> work-in-progress (and anybody who loses the race to get the lock would
> have to spin waiting for the winner to tell them the real status).
> 
> I don't know if Travis's cache storage is up to that challenge. We could
> probably build such a lock on top of third-party storage, but things are
> rapidly getting more complex.

I think we shouldn't go there because of the complexity. I reached out
to TravisCI and asked about the "hash build twice" problem [1]. Unfortunately,
I got no response, yet. The issue could also be considered a feature as you
could perform different actions in your TravisCI configuration based on
the branch name.

I think Junio's original suggestions for the Windows build makes a lot
of sense because it saves Dscho's compute resources:

--- a/ci/run-windows-build.sh
+++ b/ci/run-windows-build.sh
@@ -12,6 +12,12 @@ test -z "$GFW_CI_TOKEN" && echo "GFW_CI_TOKEN not defined" && exit
BRANCH=$1
COMMIT=$2

+if TAG=$(git describe --exact-match "$COMMIT" 2>/dev/null)
+then
+	echo "Tip of $BRANCH exactly at $TAG"
+	exit 0
+fi
+
gfwci () {
	local CURL_ERROR_CODE HTTP_CODE
	CONTENT_FILE=$(mktemp -t "git-windows-ci-XXXXXX")


However, I don't think we need to do the same for the builds that
use TravisCI resources. If they would be concerned about that then 
they wouldn't build the same hash twice in the first place.

- Lars


[1] https://twitter.com/kit3bus/status/885902189692112896

Re: Reducing redundant build at Travis?

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

>> On 14 Jul 2017, at 17:32, Jeff King <peff@peff.net> wrote:
>> 
>> I don't know if Travis's cache storage is up to that challenge. We could
>> probably build such a lock on top of third-party storage, but things are
>> rapidly getting more complex.
>
> I think we shouldn't go there because of the complexity. I reached out
> to TravisCI and asked about the "hash build twice" problem [1]. Unfortunately,
> I got no response, yet. The issue could also be considered a feature as you
> could perform different actions in your TravisCI configuration based on
> the branch name.

Oh, no doubt that it is a feature, and a very useful one at that.
With that, we can change actions depending on the branch name in
such a way that normally we do our build and test, but when we are
on a branch (not testing a tag) and its tip is tagged, we become
no-op to avoid the cost of testing.  That is the feature we exactly
want.

The question I had, and wanted a help from you, was if there was a
way we can write that "are we on a branch (not testing a tag) and is
its tip tagged?" test only once in .travis.yml, even though we have
quite a many items in the matrix.  With the current way .travis.yml
is laid out, without such a facility, we'd need the logic sprinkled
to at the beginning of all "before_install:" or something like that,
which is not quite optimal.

> I think Junio's original suggestions for the Windows build makes a lot
> of sense because it saves Dscho's compute resources:
>
> --- a/ci/run-windows-build.sh
> +++ b/ci/run-windows-build.sh
> @@ -12,6 +12,12 @@ test -z "$GFW_CI_TOKEN" && echo "GFW_CI_TOKEN not defined" && exit
> BRANCH=$1
> COMMIT=$2
>
> +if TAG=$(git describe --exact-match "$COMMIT" 2>/dev/null)
> +then
> +	echo "Tip of $BRANCH exactly at $TAG"
> +	exit 0
> +fi
> +
> gfwci () {
> 	local CURL_ERROR_CODE HTTP_CODE
> 	CONTENT_FILE=$(mktemp -t "git-windows-ci-XXXXXX")
>
> However, I don't think we need to do the same for the builds that
> use TravisCI resources. If they would be concerned about that then 
> they wouldn't build the same hash twice in the first place.

But I do care ;-) It would be nice for me not to have to wait and
keep worrying about MacOSX builds.  It also is not nice that
branches for other tests are blocked and have to wait only because
'maint' and 'vX.Y.Z' are both tested even though we know they are
the same tree.  This is where my question earlier comes from---is
there a good way to do the "not test a branch if its at a tagged
commit, because that tag will be tested anyway" test only at one
place in the .travis.yml we have?---because it's not like we only
care about Windows.

Re: Reducing redundant build at Travis?

[Lars Schneider]

> On 20 Jul 2017, at 17:16, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Lars Schneider <larsxschneider@gmail.com> writes:
> 
>>> On 14 Jul 2017, at 17:32, Jeff King <peff@peff.net> wrote:
>>> 
>>> I don't know if Travis's cache storage is up to that challenge. We could
>>> probably build such a lock on top of third-party storage, but things are
>>> rapidly getting more complex.
>> 
>> I think we shouldn't go there because of the complexity. I reached out
>> to TravisCI and asked about the "hash build twice" problem [1]. Unfortunately,
>> I got no response, yet. The issue could also be considered a feature as you
>> could perform different actions in your TravisCI configuration based on
>> the branch name.
> 
> Oh, no doubt that it is a feature, and a very useful one at that.
> With that, we can change actions depending on the branch name in
> such a way that normally we do our build and test, but when we are
> on a branch (not testing a tag) and its tip is tagged, we become
> no-op to avoid the cost of testing.  That is the feature we exactly
> want.
> 
> The question I had, and wanted a help from you, was if there was a
> way we can write that "are we on a branch (not testing a tag) and is
> its tip tagged?" test only once in .travis.yml, even though we have
> quite a many items in the matrix.  With the current way .travis.yml
> is laid out, without such a facility, we'd need the logic sprinkled
> to at the beginning of all "before_install:" or something like that,
> which is not quite optimal.

To answer your question: I don't see an easy solution to the problem.

However, maybe it is solvable with a creative usage of "build-stages"?
https://blog.travis-ci.com/2017-05-11-introducing-build-stages

Sorry,
Lars

Re: Reducing redundant build at Travis?

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

> To answer your question: I don't see an easy solution to the problem.

That's OK.  Thanks for digging.

I am wondering if the attached would be acceptable as a minimum
impact patch to address this issue.  

I think I got the "are we building a tag, or are we building a
branch that happens to be at a tag?" logic right, but I have no idea
what I am writing in the "script" sections (I am just assuming that
these lines are squashed into a line by removing line-breaks and
become a single loooong shell script), and can certainly use guiding
hands.  I didn't bother skipping the work done in before_script.

Thanks.

diff --git a/.travis.yml b/.travis.yml
index 278943d14a..55af619830 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -53,6 +53,7 @@ matrix:
       script:
         - >
           test "$TRAVIS_REPO_SLUG" != "git/git" ||
+          ci/skip-branch-tip-with-tag.sh ||
           ci/run-windows-build.sh $TRAVIS_BRANCH $(git rev-parse HEAD)
       after_failure:
     - env: Linux32
@@ -65,6 +66,7 @@ matrix:
       before_script:
       script:
         - >
+          ci/skip-branch-tip-with-tag.sh ||
           docker run
           --interactive
           --env DEVELOPER
@@ -145,9 +147,10 @@ before_script: make --jobs=2
 
 script:
   - >
+    ci/skip-branch-tip-with-tag.sh || {
     mkdir -p $HOME/travis-cache;
     ln -s $HOME/travis-cache/.prove t/.prove;
-    make --quiet test;
+    make --quiet test; }
 
 after_failure:
   - >
diff --git a/ci/skip-branch-tip-with-tag.sh b/ci/skip-branch-tip-with-tag.sh
new file mode 100755
index 0000000000..a57e724b35
--- /dev/null
+++ b/ci/skip-branch-tip-with-tag.sh
@@ -0,0 +1,23 @@
+#!/bin/sh
+
+# Sometimes, a branch is pushed at the same time the tag that points
+# at the same commit as the tip of the branch is pushed, and building
+# both at the same time is a waste.
+#
+# Travis gives a tagname e.g. v2.14.0 in $TRAVIS_BRANCH when
+# the build is triggered by a push to a tag.  Let's see if
+# $TRAVIS_BRANCH is exactly at a tag, and if so, if it is 
+# different from $TRAVIS_BRANCH.  That way, we can tell if
+# we are building the tip of a branch that is tagged---and
+# we can skip the build because we won't be skipping a build
+# of a tag.
+
+if TAG=$(git describe --exact-match "$TRAVIS_BRANCH" 2>/dev/null) &&
+	$TAG != $TRAVIS_BRANCH
+then
+	echo "Tip of $TRAVIS_BRANCH is exactly at $TAG"
+	exit 0
+else
+	exit 1
+fi
+

Re: Reducing redundant build at Travis?

[Lars Schneider]

> On 21 Jul 2017, at 18:44, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Lars Schneider <larsxschneider@gmail.com> writes:
> 
>> To answer your question: I don't see an easy solution to the problem.
> 
> That's OK.  Thanks for digging.
> 
> I am wondering if the attached would be acceptable as a minimum
> impact patch to address this issue.  

Your patch would still run a number of expensive operations for the
same hashes (e.g. static analysis and documentation builds). I started 
to work on a patch that moves all TravisCI logic into scripts located 
in the `ci` folder. These scripts share a `lib-travisci.sh` for common 
functions such as `skip_branch_tip_with_tag ()` executed at the
beginning of every script.

Does this sound sensible to you? I am a bit busy with non Git related 
work right now but I try to post the patch for you to review ASAP.

- Lars

Re: Reducing redundant build at Travis?

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

> ... I started 
> to work on a patch that moves all TravisCI logic into scripts located 
> in the `ci` folder. These scripts share a `lib-travisci.sh` for common 
> functions such as `skip_branch_tip_with_tag ()` executed at the
> beginning of every script.
>
> Does this sound sensible to you? I am a bit busy with non Git related 
> work right now but I try to post the patch for you to review ASAP.

Absolutely fantastic.

Take your time; there is no need to rush and haste makes waste.

Thanks for working on this.