[Buildroot] [PATCH] git fetcher: do not remove the tree after initial clone

[Buildroot] [PATCH] git fetcher: do not remove the tree after initial clone

[Nicolas Dechesne]

With the current implementation of the 'git fetcher' the project git tree is
cloned for every build (if the commit ID has changed, otherwise the .tar.gz
would already exist). This appears to me quite inefficient since rebuilding
with a more recent commit is a common use case.

With this patch I am changing how the 'git fetch algorithm' works as follows:

- if project <foobar> uses git, the tree will be cloned in dl/git/foobar
- the tree is cloned using 'git clone -n' to make sure that we can reliably
  update the tree later (--bare does not create the correct refspec, and
  --mirror would work but is doing more than necessary)
- the clone is done only the first time, e.g. when dl/git/foobar does not
  exist
- when making another build, if dl/git/foobar exists, the tree is not
  cloned, but instead we run 'git fetch --all --tags' to update the existing
  tree. that can save a lot of time, especially with large trees since only
  the delta is downloaded. '--all --tags' is required to make sure that all
  remotes commits are downloaded.

Note1: git fetch is actually called even the first time, right after the
       clone, this is obviously not required, but it made the code much
       simpler in the makefile, and it does not hurt too much

Note2: similiar mechanism might be implemented for other SCM supported by
       buildroot

Signed-off-by: Nicolas Dechesne <n-dechesne@ti.com>
---
 package/Makefile.package.in |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
index 868bf28..2a2cc40 100644
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -1,4 +1,4 @@
-################################################################################
+###############################################################################
 # Generic package infrastructure
 #
 # This file implements an infrastructure that eases development of
@@ -105,13 +105,13 @@ endif
 
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	$(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
-	pushd $($(PKG)_BASE_NAME) > /dev/null && \
+	(mkdir -p $(DL_DIR)/git && pushd $(DL_DIR)/git > /dev/null && \
+        (test -d $($(PKG)_NAME) ||  $(GIT) clone -n $($(PKG)_SITE) $($(PKG)_NAME) ) && \
+	pushd $($(PKG)_NAME) > /dev/null && \
+        $(GIT) fetch --tags --all && \
 	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
 		gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
 	popd > /dev/null && \
-	rm -rf $($(PKG)_DL_DIR) && \
 	popd > /dev/null)
 endef
 
-- 
1.7.5.4

[Buildroot] [PATCH] git fetcher: do not remove the tree after initial clone

[Luca Ceresoli]

Hi Nicolas,

Nicolas Dechesne wrote:
> With the current implementation of the 'git fetcher' the project git tree is
> cloned for every build (if the commit ID has changed, otherwise the .tar.gz
> would already exist). This appears to me quite inefficient since rebuilding
> with a more recent commit is a common use case.
>
> With this patch I am changing how the 'git fetch algorithm' works as follows:
>
> - if project<foobar>  uses git, the tree will be cloned in dl/git/foobar
> - the tree is cloned using 'git clone -n' to make sure that we can reliably
>    update the tree later (--bare does not create the correct refspec, and
>    --mirror would work but is doing more than necessary)
> - the clone is done only the first time, e.g. when dl/git/foobar does not
>    exist
> - when making another build, if dl/git/foobar exists, the tree is not
>    cloned, but instead we run 'git fetch --all --tags' to update the existing
>    tree. that can save a lot of time, especially with large trees since only
>    the delta is downloaded. '--all --tags' is required to make sure that all
>    remotes commits are downloaded.

The git downloader was the subject of a pretty long discussion around
July 2010, and two different use cases were conceived with respect to
git-hosted packages:

- integrate in BR an externally-developed package that is not
   released in tarballs, but only in a git repository (use BR for
   integration);
- integrate in BR a package that is internally under development,
   so that it can be easily tested inside the entire root fs (use BR for
   integration and development).

In the first case updates to the BR package would happen rarely,
typically once per each new upstream release.

In the second case updates would be frequent, probably many per day,
on the developer workstation.

Mine vision at the time was more close to the second case, as I do
develop software that I test within buildroot. This is also your idea
with this proposed patch.
In fact the implementation I submitted at that time was similar to
yours in principle: clone once, fetch the following times.

Instead the choice for mainline BR was to privilege the first
use case (use BR for integration).

I was not enthusiast of course, but I found that in my case it was not
a big problem anyway since I clone from local servers, which makes
clones relatively fast. This would be different if the repositories you
want to clone are large or remote (or both!).

Using BR for development has not been completely neglected anyway.
Recently, Thomas Petazzoni submitted a patch that would allow to use a
local directory to build the package. This is much different from using
git fetches, but is expected to serve the "use BR for development" use
case effectively.
You might want to test it and comment to the author. Here it is:
http://lists.busybox.net/pipermail/buildroot/2011-July/044479.html

Luca

[Buildroot] [PATCH] git fetcher: do not remove the tree after initial clone

[Thomas Petazzoni]

Le Mon, 29 Aug 2011 09:48:47 +0200,
Luca Ceresoli <luca@lucaceresoli.net> a ?crit :

> Using BR for development has not been completely neglected anyway.
> Recently, Thomas Petazzoni submitted a patch that would allow to use a
> local directory to build the package. This is much different from using
> git fetches, but is expected to serve the "use BR for development" use
> case effectively.
> You might want to test it and comment to the author. Here it is:
> http://lists.busybox.net/pipermail/buildroot/2011-July/044479.html

I still think that this local directory mechanism is a better solution.

I have a new version of my patch set, which I hope to submit soon, at
least when the 2011.11 development process will open (by the end of the
month, so should be in the next few days).

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH] git fetcher: do not remove the tree after initial clone

[Dechesne, Nicolas]

Lucas, Thomas,

On Mon, Aug 29, 2011 at 2:23 PM, Thomas Petazzoni <
thomas.petazzoni@free-electrons.com> wrote:

> > Using BR for development has not been completely neglected anyway.
> > Recently, Thomas Petazzoni submitted a patch that would allow to use a
> > local directory to build the package. This is much different from using
> > git fetches, but is expected to serve the "use BR for development" use
> > case effectively.
> > You might want to test it and comment to the author. Here it is:
> > http://lists.busybox.net/pipermail/buildroot/2011-July/044479.html
>
> I still think that this local directory mechanism is a better solution.
>
> I have a new version of my patch set, which I hope to submit soon, at
> least when the 2011.11 development process will open (by the end of the
> month, so should be in the next few days).
>

i like the 'local directory' method as well, really much. but i still
believe that the 2 things are complementary..

i happen to use BR in a project using large git tree which are not close to
me, so I care about git fetch vs git clone.
on top of that there are many users that regularly make rebuild.

the way i see what i propose is *just* an optimization of how BR will use
git, not a new workflow. for those of us who use
BR for development, i fully agree that local method is a great addition.

fwiw, i am coming from OE which such a git clone/fetch optimization is
already implemented.

any reason why we can't have both?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20110829/5426ef47/attachment.html>

[Buildroot] [PATCH] git fetcher: do not remove the tree after initial clone

[Luca Ceresoli]

Dechesne, Nicolas wrote:
> Lucas, Thomas,
>
> On Mon, Aug 29, 2011 at 2:23 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com
> <mailto:thomas.petazzoni@free-electrons.com>> wrote:
>
>      > Using BR for development has not been completely neglected anyway.
>      > Recently, Thomas Petazzoni submitted a patch that would allow to
>     use a
>      > local directory to build the package. This is much different from
>     using
>      > git fetches, but is expected to serve the "use BR for
>     development" use
>      > case effectively.
>      > You might want to test it and comment to the author. Here it is:
>      > http://lists.busybox.net/pipermail/buildroot/2011-July/044479.html
>
>     I still think that this local directory mechanism is a better solution.
>
>     I have a new version of my patch set, which I hope to submit soon, at
>     least when the 2011.11 development process will open (by the end of the
>     month, so should be in the next few days).
>
>
> i like the 'local directory' method as well, really much. but i still
> believe that the 2 things are complementary..

Thomas' local directory mechanism has an advantage: it works also for
non-git (and even non-versioned) packages.

You could somehow use this method for your needs too: git clone in a
local directory once, then git pull before each BR rebuild. Not as
straightforward as you would like, but maybe a good compromise.

>
> i happen to use BR in a project using large git tree which are not close
> to me, so I care about git fetch vs git clone.
> on top of that there are many users that regularly make rebuild.
>
> the way i see what i propose is *just* an optimization of how BR will
> use git, not a new workflow.

Still it has the disadvantage of using a lot of disk space, and it needs
to connect to the server at every build only to discover if there are
new commits.

This would hurt people not having your needs.

If you would add an option to choose between the always-clone git
downloader and the clone-once-fetch-many version, your work would become
more interesting.
But this is only my opinion.

Luca

[Buildroot] [PATCH] git fetcher: do not remove the tree after initial clone

[Dechesne, Nicolas]

On Tue, Aug 30, 2011 at 10:20 AM, Luca Ceresoli <luca@lucaceresoli.net>wrote:

>    I have a new version of my patch set, which I hope to submit soon, at
>>    least when the 2011.11 development process will open (by the end of the
>>    month, so should be in the next few days).
>>
>>
>> i like the 'local directory' method as well, really much. but i still
>> believe that the 2 things are complementary..
>>
>
> Thomas' local directory mechanism has an advantage: it works also for
> non-git (and even non-versioned) packages.
>
> You could somehow use this method for your needs too: git clone in a
> local directory once, then git pull before each BR rebuild. Not as
> straightforward as you would like, but maybe a good compromise.


ok. i am not planning to use this feature to improve the 'I am a developer'
situation.
in this case the local method is definetly better.

my use case is:
- i am using BR for a large project and all our programs are hosted on many
(internal) git servers
and these servers are far away (multisite company)
- many developers would often need to build a specific release (without
making changes in the code), just
fetch new/updated recipes and rebuild


>
>
>
>> i happen to use BR in a project using large git tree which are not close
>> to me, so I care about git fetch vs git clone.
>> on top of that there are many users that regularly make rebuild.
>>
>> the way i see what i propose is *just* an optimization of how BR will
>> use git, not a new workflow.
>>
>
> Still it has the disadvantage of using a lot of disk space, and it needs
> to connect to the server at every build only to discover if there are
> new commits.
>

agreed on disk space. since we don't remove the clone, we have to store
them.
but on the 2nd point it would connect to server *only* if the recipe has
changed and uses
a new commit. otherwise it wouldn't connect again since after a successful
build the tarball would be
there already (and the extracted source code).

i could make one optimization. run git fetch only if the commit object I am
looking for is not
there already.

>
> This would hurt people not having your needs.
>

> If you would add an option to choose between the always-clone git
> downloader and the clone-once-fetch-many version, your work would become
> more interesting.
> But this is only my opinion.
>

is that true? i could make the update. i would just need to create two
variants of DOWNLOAD_GIT and
choose based on the config option...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20110830/22d0a829/attachment.html>