[Buildroot] [PATCH v2] support/download/git: Prioritize remote archive

[Buildroot] [PATCH v2] support/download/git: Prioritize remote archive

[Benjamin Kamath]

From: Benjamin Kamath <bkamath@spaceflight.com>

Attempt to do a remote archive since it shortcuts us past a few steps when
available. Additionally. if the git server has uploadArchive.allowUnreachable
set to true, then this method can also work on arbitrary sha1s, offering a huge
speed advantage over a full clone.

Signed-off-by: Benjamin Kamath <bkamath@spaceflight.com>
---
Changes v1 -> v2:
    - Skip remote archive for repositories with submodules (suggested by Yann)

 support/download/git | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/support/download/git b/support/download/git
index 416cd1b..d060cbd 100755
--- a/support/download/git
+++ b/support/download/git
@@ -36,6 +36,17 @@ _git() {
     eval ${GIT} "${@}"
 }
 
+# Try a remote archive, since it is as fast as a shallow clone and can give us
+# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
+# on the remote, this will also work for arbitrary sha1s, and will offer a
+# considerable speedup over a full clone. This will not work for submodules.
+if [ ${recurse} -eq 0 ]; then
+    printf "Doing remote archive\n"
+    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then
+        exit 0
+    fi
+fi
+
 # Try a shallow clone, since it is faster than a full clone - but that only
 # works if the version is a ref (tag or branch). Before trying to do a shallow
 # clone we check if ${cset} is in the list provided by git ls-remote. If not
-- 
2.7.4

[Buildroot] [PATCH v2] support/download/git: Prioritize remote archive

[Yann E. MORIN]

Benjmin, All,

On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
> From: Benjamin Kamath <bkamath@spaceflight.com>
> 
> Attempt to do a remote archive since it shortcuts us past a few steps when
> available. Additionally. if the git server has uploadArchive.allowUnreachable
> set to true, then this method can also work on arbitrary sha1s, offering a huge
> speed advantage over a full clone.

Do you know of a publicly-accessible server that allows remote archives?

I've tried with a github-hosted package, but githib won;t allow remote
archives, it seems...

Regards,
Yann E. MORIN.

> Signed-off-by: Benjamin Kamath <bkamath@spaceflight.com>
> ---
> Changes v1 -> v2:
>     - Skip remote archive for repositories with submodules (suggested by Yann)
> 
>  support/download/git | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/support/download/git b/support/download/git
> index 416cd1b..d060cbd 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -36,6 +36,17 @@ _git() {
>      eval ${GIT} "${@}"
>  }
>  
> +# Try a remote archive, since it is as fast as a shallow clone and can give us
> +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
> +# on the remote, this will also work for arbitrary sha1s, and will offer a
> +# considerable speedup over a full clone. This will not work for submodules.
> +if [ ${recurse} -eq 0 ]; then
> +    printf "Doing remote archive\n"
> +    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then
> +        exit 0
> +    fi
> +fi
> +
>  # Try a shallow clone, since it is faster than a full clone - but that only
>  # works if the version is a ref (tag or branch). Before trying to do a shallow
>  # clone we check if ${cset} is in the list provided by git ls-remote. If not
> -- 
> 2.7.4
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2] support/download/git: Prioritize remote archive

[Benjamin Kamath]

On Wed, Aug 17, 2016 at 2:53 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Benjmin, All,
>
> On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
>> From: Benjamin Kamath <bkamath@spaceflight.com>
>>
>> Attempt to do a remote archive since it shortcuts us past a few steps when
>> available. Additionally. if the git server has uploadArchive.allowUnreachable
>> set to true, then this method can also work on arbitrary sha1s, offering a huge
>> speed advantage over a full clone.
>
> Do you know of a publicly-accessible server that allows remote archives?
>
> I've tried with a github-hosted package, but githib won;t allow remote
> archives, it seems...
I just tried quite a few and it doesn't seem to be frequently
supported. gitlab-ce does seem to have it enabled by default:

git at gitlab.com:gitlab-org/gitlab-ce.git

>
> Regards,
> Yann E. MORIN.
>
>> Signed-off-by: Benjamin Kamath <bkamath@spaceflight.com>
>> ---
>> Changes v1 -> v2:
>>     - Skip remote archive for repositories with submodules (suggested by Yann)
>>
>>  support/download/git | 11 +++++++++++
>>  1 file changed, 11 insertions(+)
>>
>> diff --git a/support/download/git b/support/download/git
>> index 416cd1b..d060cbd 100755
>> --- a/support/download/git
>> +++ b/support/download/git
>> @@ -36,6 +36,17 @@ _git() {
>>      eval ${GIT} "${@}"
>>  }
>>
>> +# Try a remote archive, since it is as fast as a shallow clone and can give us
>> +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
>> +# on the remote, this will also work for arbitrary sha1s, and will offer a
>> +# considerable speedup over a full clone. This will not work for submodules.
>> +if [ ${recurse} -eq 0 ]; then
>> +    printf "Doing remote archive\n"
>> +    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then
>> +        exit 0
>> +    fi
>> +fi
>> +
>>  # Try a shallow clone, since it is faster than a full clone - but that only
>>  # works if the version is a ref (tag or branch). Before trying to do a shallow
>>  # clone we check if ${cset} is in the list provided by git ls-remote. If not
>> --
>> 2.7.4
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2] support/download/git: Prioritize remote archive

[Yann E. MORIN]

Benjamin, All,

On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly:
> On Wed, Aug 17, 2016 at 2:53 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > Benjmin, All,
> >
> > On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
> >> From: Benjamin Kamath <bkamath@spaceflight.com>
> >>
> >> Attempt to do a remote archive since it shortcuts us past a few steps when
> >> available. Additionally. if the git server has uploadArchive.allowUnreachable
> >> set to true, then this method can also work on arbitrary sha1s, offering a huge
> >> speed advantage over a full clone.
> >
> > Do you know of a publicly-accessible server that allows remote archives?
> >
> > I've tried with a github-hosted package, but githib won;t allow remote
> > archives, it seems...
> I just tried quite a few and it doesn't seem to be frequently
> supported. gitlab-ce does seem to have it enabled by default:
> 
> git at gitlab.com:gitlab-org/gitlab-ce.git

OK, it does work, as long as one uses the ssh protocol. It does not work
via http:// or git://

Which means we have no package in Buildroot that can use this.

Do you have a need for that for an internal package?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2] support/download/git: Prioritize remote archive

[Benjamin Kamath]

On Thu, Aug 18, 2016 at 2:10 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Benjamin, All,
>
> On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly:
>> On Wed, Aug 17, 2016 at 2:53 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>> > Benjmin, All,
>> >
>> > On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
>> >> From: Benjamin Kamath <bkamath@spaceflight.com>
>> >>
>> >> Attempt to do a remote archive since it shortcuts us past a few steps when
>> >> available. Additionally. if the git server has uploadArchive.allowUnreachable
>> >> set to true, then this method can also work on arbitrary sha1s, offering a huge
>> >> speed advantage over a full clone.
>> >
>> > Do you know of a publicly-accessible server that allows remote archives?
>> >
>> > I've tried with a github-hosted package, but githib won;t allow remote
>> > archives, it seems...
>> I just tried quite a few and it doesn't seem to be frequently
>> supported. gitlab-ce does seem to have it enabled by default:
>>
>> git at gitlab.com:gitlab-org/gitlab-ce.git
>
> OK, it does work, as long as one uses the ssh protocol. It does not work
> via http:// or git://
>
OK. I had not noticed that before, and as such that makes this patch
much less useful.

> Which means we have no package in Buildroot that can use this.
>
> Do you have a need for that for an internal package?
Yes, but we can create a local patch for it. On dev-branches of our
project we frequently have internal packages that point directly at
sha1s. I mistakenly thought more servers would support remote archive.
I suppose we should reject this patch.

>
> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2] support/download/git: Prioritize remote archive

[Yann E. MORIN]

Benjamin, All,

On 2016-08-18 02:26 -0700, Benjamin Kamath spake thusly:
> On Thu, Aug 18, 2016 at 2:10 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > Benjamin, All,
> >
> > On 2016-08-17 15:17 -0700, Benjamin Kamath spake thusly:
> >> On Wed, Aug 17, 2016 at 2:53 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> >> > Benjmin, All,
> >> >
> >> > On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
> >> >> From: Benjamin Kamath <bkamath@spaceflight.com>
> >> >>
> >> >> Attempt to do a remote archive since it shortcuts us past a few steps when
> >> >> available. Additionally. if the git server has uploadArchive.allowUnreachable
> >> >> set to true, then this method can also work on arbitrary sha1s, offering a huge
> >> >> speed advantage over a full clone.
> >> >
> >> > Do you know of a publicly-accessible server that allows remote archives?
> >> >
> >> > I've tried with a github-hosted package, but githib won;t allow remote
> >> > archives, it seems...
> >> I just tried quite a few and it doesn't seem to be frequently
> >> supported. gitlab-ce does seem to have it enabled by default:
> >>
> >> git at gitlab.com:gitlab-org/gitlab-ce.git
> >
> > OK, it does work, as long as one uses the ssh protocol. It does not work
> > via http:// or git://
> >
> OK. I had not noticed that before, and as such that makes this patch
> much less useful.
> 
> > Which means we have no package in Buildroot that can use this.
> >
> > Do you have a need for that for an internal package?
> Yes, but we can create a local patch for it. On dev-branches of our
> project we frequently have internal packages that point directly at
> sha1s. I mistakenly thought more servers would support remote archive.
> I suppose we should reject this patch.

I'm OK with adding this, as long as it does not break existing
behaviour. From my little tests, it shows that it works.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2] support/download/git: Prioritize remote archive

[Yann E. MORIN]

Benjamin, All,

On 2016-08-17 14:25 -0700, Benjamin Kamath spake thusly:
> From: Benjamin Kamath <bkamath@spaceflight.com>
> 
> Attempt to do a remote archive since it shortcuts us past a few steps when
> available. Additionally. if the git server has uploadArchive.allowUnreachable
> set to true, then this method can also work on arbitrary sha1s, offering a huge
> speed advantage over a full clone.
> 
> Signed-off-by: Benjamin Kamath <bkamath@spaceflight.com>
> ---
> Changes v1 -> v2:
>     - Skip remote archive for repositories with submodules (suggested by Yann)
> 
>  support/download/git | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/support/download/git b/support/download/git
> index 416cd1b..d060cbd 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -36,6 +36,17 @@ _git() {
>      eval ${GIT} "${@}"
>  }
>  
> +# Try a remote archive, since it is as fast as a shallow clone and can give us
> +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
> +# on the remote, this will also work for arbitrary sha1s, and will offer a
> +# considerable speedup over a full clone. This will not work for submodules.
> +if [ ${recurse} -eq 0 ]; then
> +    printf "Doing remote archive\n"
> +    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} -o ${output} ${cset} 2>&1; then
> +        exit 0

I would have also done:

        else
            printf "Remote archive failed, falling back to cloning\n"

Like we do for the shallow clone.

Also, why do you redirect stderr? We don't do it in the clone commands.
We only do it to fetch the special references (gitHub PRs of Gerrit
changes) because it does not make sense to display an spurious message
when the reference is in fact not special.

Regards,
Yann E. MORIN.

> +    fi
> +fi
> +
>  # Try a shallow clone, since it is faster than a full clone - but that only
>  # works if the version is a ref (tag or branch). Before trying to do a shallow
>  # clone we check if ${cset} is in the list provided by git ls-remote. If not
> -- 
> 2.7.4
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v3] support/download/git: Prioritize remote archive

[Benjamin Kamath]

Attempt to do a remote archive since it shortcuts us past a few steps when
available. Additionally. if the git server has uploadArchive.allowUnreachable
set to true, then this method can also work on arbitrary sha1s, offering a huge
speed advantage over a full clone.

Signed-off-by: Benjamin Kamath <kamath.ben@gmail.com>
---
Changes v2 -> v3:
    - Indicate to user if the remote archive fails (suggested by Yann)
    - redirect all errors from git archive to /dev/null since this
      operation fails on most public git servers
Changes v1 -> v2:
    - Skip remote archive for repositories with submodules (suggested by Yann)

 support/download/git | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/support/download/git b/support/download/git
index 416cd1b..ec49057 100755
--- a/support/download/git
+++ b/support/download/git
@@ -36,6 +36,22 @@ _git() {
     eval ${GIT} "${@}"
 }
 
+# Try a remote archive, since it is as fast as a shallow clone and can give us
+# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
+# on the remote, this will also work for arbitrary sha1s, and will offer a
+# considerable speedup over a full clone. This will not work for submodules.
+# Since this command fails on most public git servers, consign the output to 
+# oblivion but still trace as a warning.
+if [ ${recurse} -eq 0 ]; then
+    printf "Doing remote archive\n"
+    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} \
+        -o ${output} ${cset} >/dev/null 2>&1; then
+        exit 0
+    else
+        printf "Remote archive failed, falling back to cloning\n"
+    fi
+fi
+
 # Try a shallow clone, since it is faster than a full clone - but that only
 # works if the version is a ref (tag or branch). Before trying to do a shallow
 # clone we check if ${cset} is in the list provided by git ls-remote. If not
-- 
2.7.4

[Buildroot] [PATCH v3] support/download/git: Prioritize remote archive

[Arnout Vandecappelle]

 Hi Benjamin,


On 24-08-16 19:47, Benjamin Kamath wrote:
> Attempt to do a remote archive since it shortcuts us past a few steps when
> available. Additionally. if the git server has uploadArchive.allowUnreachable
> set to true, then this method can also work on arbitrary sha1s, offering a huge
> speed advantage over a full clone.
> 
> Signed-off-by: Benjamin Kamath <kamath.ben@gmail.com>

 After some discussion at the Buildroot developer meeting, we decided not to
accept this patch, for the following reasons.

* We want to keep the download/git script as simple as reasonably possible.

* Not many upstream sites support this option.

* We now have good support for shallow clones; in most cases, the shallow clone
handling will give equal benefits.


 So I've marked this patch as rejected in patchwork.

 If you feel it is important to include this feature in Buildroot after all,
feel free to argue your case.

 And finally, sorry that it took so long before we reacted to your patch.

 Regards,
 Arnout


> ---
> Changes v2 -> v3:
>     - Indicate to user if the remote archive fails (suggested by Yann)
>     - redirect all errors from git archive to /dev/null since this
>       operation fails on most public git servers
> Changes v1 -> v2:
>     - Skip remote archive for repositories with submodules (suggested by Yann)
> 
>  support/download/git | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/support/download/git b/support/download/git
> index 416cd1b..ec49057 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -36,6 +36,22 @@ _git() {
>      eval ${GIT} "${@}"
>  }
>  
> +# Try a remote archive, since it is as fast as a shallow clone and can give us
> +# an archive directly. Also, if uploadArchive.allowUnreachable is set to true
> +# on the remote, this will also work for arbitrary sha1s, and will offer a
> +# considerable speedup over a full clone. This will not work for submodules.
> +# Since this command fails on most public git servers, consign the output to 
> +# oblivion but still trace as a warning.
> +if [ ${recurse} -eq 0 ]; then
> +    printf "Doing remote archive\n"
> +    if _git archive --format=tar.gz --prefix=${basename}/ --remote=${repo} \
> +        -o ${output} ${cset} >/dev/null 2>&1; then
> +        exit 0
> +    else
> +        printf "Remote archive failed, falling back to cloning\n"
> +    fi
> +fi
> +
>  # Try a shallow clone, since it is faster than a full clone - but that only
>  # works if the version is a ref (tag or branch). Before trying to do a shallow
>  # clone we check if ${cset} is in the list provided by git ls-remote. If not
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF