[Buildroot] [PATCH 0/5] Introduce alternative archive format

[Buildroot] [PATCH 0/5] Introduce alternative archive format

[Jérôme Pouiller]

Hello,

As suggested by Arnout [1], this series provide an alternative archive
format. This new format contains a shallowed version of upstream
repository. This format is a little bigger and a little longer to
create but it allow a better workflow with upstream. I describe some
good practice in patch 2.

Notice projects hosted by github don't yet benefit of this feature
since I have not found any elegant way to do it :-(. 

During my tests, I have noticed current shallow clone is mostly broken
(at least with git < 2.5 [2]). Indeed, shallow clone only work with
symbolic references (HEAD, a tag or a branch). However, we avoid use of
symbolic references in VERSION.

[1] http://lists.busybox.net/pipermail/buildroot/2015-November/144296.html
[2] http://stackoverflow.com/a/30701724/301717


J?r?me Pouiller (5):
  pkg-download: do not test SITE_METHOD
  download/git: allow to create archives containing shallowed git repos
  pkg-generic: allow to populate build directory from a git archive
  pkg-generic: provide an option to use git archives
  pkg-generic: tag sources if git is used

 Config.in               | 13 +++++++++++
 package/pkg-download.mk |  7 +-----
 package/pkg-generic.mk  | 60 ++++++++++++++++++++++++++++++++++---------------
 support/download/git    | 30 ++++++++++++++++++++++---
 4 files changed, 83 insertions(+), 27 deletions(-)

-- 
2.1.4

[Buildroot] [PATCH 1/5] pkg-download: do not test SITE_METHOD

[Jérôme Pouiller]

SITE_METHOD is always set in inner-generic-package. So, it is useless
to test it in pkg-download.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/pkg-download.mk | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index c86b9ab..757bafe 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -222,12 +222,7 @@ define DOWNLOAD_INNER
 		exit 1 ; \
 	fi ; \
 	if test -n "$(1)" ; then \
-		if test -z "$($(PKG)_SITE_METHOD)" ; then \
-			scheme="$(call geturischeme,$(1))" ; \
-		else \
-			scheme="$($(PKG)_SITE_METHOD)" ; \
-		fi ; \
-		case "$$scheme" in \
+		case "$($(PKG)_SITE_METHOD)" in \
 			git) $($(3)_GIT) && exit ;; \
 			svn) $($(3)_SVN) && exit ;; \
 			cvs) $($(3)_CVS) && exit ;; \
-- 
2.1.4

[Buildroot] [PATCH 2/5] download/git: allow to create archives containing shallowed git repos

[Jérôme Pouiller]

We introduce a new tarball format for projects managed with git. This
new format is a shallowed version of upstream repository.

This format is a little larger (20 to 50% in my tests) than standard
one and contains information about upstream git repository.

That simplify workflow with upstream. It is now possible to reconnect
a build directory with upstream repository using:

   git fetch --unshallow

If you want to also get remote tags and branch (as in a standard
clone), run:

   git config remote.origin.fetch refs/heads/*:refs/remotes/origin/*
   git fetch

For large repository, you can use a local repository as reference in
order to speedup unshallow process (read warnings about '--reference'
option in git-clone manual page):

   echo "repo_path/.git/objects" > .git/objects/info/alternates
   git fetch --unshallow

Note 1: some repository does not support unshallowing (especially
anonscm.debian.org)

Note 2: I have tested this patch with git 2.1 (from Debian Jessie).

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 support/download/git | 30 +++++++++++++++++++++++++++---
 1 file changed, 27 insertions(+), 3 deletions(-)

diff --git a/support/download/git b/support/download/git
index 0e6103b..6755f65 100755
--- a/support/download/git
+++ b/support/download/git
@@ -46,7 +46,31 @@ if [ ${git_done} -eq 0 ]; then
     ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
 fi
 
-GIT_DIR="${basename}" \
-${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}"
+if [[ ${output} != *.git.tar.gz* ]]; then
+    GIT_DIR="${basename}" \
+    ${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}"
 
-gzip <"${output}.tmp" >"${output}"
+    gzip <"${output}.tmp" >"${output}"
+fi
+if [[ ${output} = *.git.tar.gz* ]]; then
+    GIT_DIR="${basename}" ${GIT} reset --soft ${cset}
+
+    # Git cannot shallow a repository on an arbitrary commit-id. It must be a
+    # symbolic reference of a remote repository (HEAD, tag or branch) (problem
+    # seems to be resolved in git 2.5).
+    # However, we want to be able to shallow on any commit-id.
+    # We didn't find any other way than use an intermediate repository in order
+    # to work around this limitation.
+    # (In add, this method make easier pruning of packed objects)
+    #
+    # Note 1: --depth is ignored in local clones. Git suggest to use file://
+    # in order to force copy of objects. We have to use an absolute path with
+    # these URIs
+    #
+    # Note 2: ${basename}.git will be removed by dl-wrapper (since it is in
+    # same directory than ${basename})
+    ${GIT} clone "file://$(readlink -f ${basename})" "${basename}.git" --bare --depth 1
+    GIT_DIR="${basename}.git" ${GIT} remote set-url origin "${repo}"
+
+    tar czf "${output}" "${basename}.git"
+fi
-- 
2.1.4

[Buildroot] [PATCH 3/5] pkg-generic: allow to populate build directory from a git archive

[Jérôme Pouiller]

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/pkg-generic.mk | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 08f4f31..846d8e9 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -515,12 +515,26 @@ $(2)_TARGET_SOURCE =		$$($(2)_DIR)/.stamp_downloaded
 $(2)_TARGET_DIRCLEAN =		$$($(2)_DIR)/.stamp_dircleaned
 
 # default extract command
+ifdef $(2)_SOURCE
+ifeq ($$(filter %.git.tar.gz,$$($(2)_SOURCE)),)
 $(2)_EXTRACT_CMDS ?= \
-	$$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$(DL_DIR)/$$($(2)_SOURCE) | \
+	$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$(DL_DIR)/$$($(2)_SOURCE) | \
 	$$(TAR) --strip-components=$$($(2)_STRIP_COMPONENTS) \
 		-C $$($(2)_DIR) \
 		$$(foreach x,$$($(2)_EXCLUDES),--exclude='$$(x)' ) \
-		$$(TAR_OPTIONS) -)
+		$$(TAR_OPTIONS) -
+else
+# $(2)_STRIP_COMPONENTS != 1 and $(2)_EXCLUDES are not supported
+$(2)_EXTRACT_CMDS ?= \
+	mkdir -p $$($(2)_DIR)/.git && \
+	$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$(DL_DIR)/$$($(2)_SOURCE) | \
+	$$(TAR) --strip-components=1 \
+		-C $$($(2)_DIR)/.git \
+		$$(TAR_OPTIONS) - && \
+	$(GIT) -C $$($(2)_DIR) config --local --bool core.bare false && \
+	$(GIT) -C $$($(2)_DIR) reset --hard
+endif
+endif
 
 # pre/post-steps hooks
 $(2)_PRE_DOWNLOAD_HOOKS         ?=
-- 
2.1.4

[Buildroot] [PATCH 4/5] pkg-generic: provide an option to use git archives

[Jérôme Pouiller]

Note: In order to make this work, $(2)_SITE_METHOD must be set before
$(2)_SOURCE

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 Config.in              | 13 +++++++++++++
 package/pkg-generic.mk | 36 ++++++++++++++++++++----------------
 2 files changed, 33 insertions(+), 16 deletions(-)

diff --git a/Config.in b/Config.in
index d795361..6b9482c 100644
--- a/Config.in
+++ b/Config.in
@@ -158,6 +158,19 @@ config BR2_HOST_DIR
 
 menu "Mirrors and Download locations"
 
+config BR2_USE_GIT_ARCHIVES
+	bool "Place git repository in archives"
+	help
+	  In case a package use a git repository as source. Enabling this
+	  option will archive (shallowed) bare git repository instead of plain
+	  files. In order to distinguish these archives from classical ones,
+	  their names end with .git.tar.gz.
+
+	  Enable this option if you work with upstream and are interrested to
+	  have a better workflow with upstream team. Notice archive generation
+	  is about 25% slower and archives are 20 to 50% bigger. Time to extract
+	  archives are identical with or without this option.
+
 config BR2_PRIMARY_SITE
 	string "Primary download site"
 	default ""
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 846d8e9..30155a6 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -412,11 +412,30 @@ ifneq ($$($(2)_OVERRIDE_SRCDIR),)
 $(2)_VERSION = custom
 endif
 
+ifndef $(2)_SITE
+ ifdef $(3)_SITE
+  $(2)_SITE = $$($(3)_SITE)
+ endif
+endif
+
+ifndef $(2)_SITE_METHOD
+ ifdef $(3)_SITE_METHOD
+  $(2)_SITE_METHOD = $$($(3)_SITE_METHOD)
+ else
+  # Try automatic detection using the scheme part of the URI
+  $(2)_SITE_METHOD = $$(call geturischeme,$$($(2)_SITE))
+ endif
+endif
+
 ifndef $(2)_SOURCE
  ifdef $(3)_SOURCE
   $(2)_SOURCE = $$($(3)_SOURCE)
  else
-  $(2)_SOURCE			?= $$($(2)_RAWNAME)-$$($(2)_VERSION).tar.gz
+  ifeq ($(BR2_USE_GIT_ARCHIVES)$$($(2)_SITE_METHOD),ygit)
+   $(2)_SOURCE			?= $$($(2)_RAWNAME)-$$($(2)_VERSION).git.tar.gz
+  else
+   $(2)_SOURCE			?= $$($(2)_RAWNAME)-$$($(2)_VERSION).tar.gz
+  endif
  endif
 endif
 
@@ -431,21 +450,6 @@ $(2)_ALL_DOWNLOADS = \
 		$$(if $$(findstring ://,$$(p)),$$(p),\
 			$$($(2)_SITE)/$$(p)))
 
-ifndef $(2)_SITE
- ifdef $(3)_SITE
-  $(2)_SITE = $$($(3)_SITE)
- endif
-endif
-
-ifndef $(2)_SITE_METHOD
- ifdef $(3)_SITE_METHOD
-  $(2)_SITE_METHOD = $$($(3)_SITE_METHOD)
- else
-	# Try automatic detection using the scheme part of the URI
-	$(2)_SITE_METHOD = $$(call geturischeme,$$($(2)_SITE))
- endif
-endif
-
 ifeq ($$($(2)_SITE_METHOD),local)
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 $(2)_OVERRIDE_SRCDIR = $$($(2)_SITE)
-- 
2.1.4

[Buildroot] [PATCH 5/5] pkg-generic: tag sources if git is used

[Jérôme Pouiller]

In order to simplify patch updating, tag original version of sources.
We may also provide a feature to automaticaly update patches in future.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/pkg-generic.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 30155a6..6d7677b 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -152,6 +152,12 @@ $(BUILD_DIR)/%/.stamp_patched:
 	@$(call step_start,patch)
 	@$(call MESSAGE,"Patching")
 	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
+	$(Q)( \
+	if test -d $(@D)/.git; then \
+	  GIT_DIR=$(@D)/.git $(GIT) tag upstream; \
+	  GIT_DIR=$(@D)/.git $(GIT) tag $(NAMEVER); \
+	fi; \
+	)
 	$(foreach p,$($(PKG)_PATCH),$(APPLY_PATCHES) $(@D) $(DL_DIR) $(notdir $(p))$(sep))
 	$(Q)( \
 	for D in $(PATCH_BASE_DIRS); do \
-- 
2.1.4

[Buildroot] [PATCH 0/5] Introduce alternative archive format

[Thomas Petazzoni]

J?r?me,

On Thu, 19 Nov 2015 11:36:01 +0100, J?r?me Pouiller wrote:

> As suggested by Arnout [1], this series provide an alternative archive
> format. This new format contains a shallowed version of upstream
> repository. This format is a little bigger and a little longer to
> create but it allow a better workflow with upstream. I describe some
> good practice in patch 2.
> 
> Notice projects hosted by github don't yet benefit of this feature
> since I have not found any elegant way to do it :-(. 
> 
> During my tests, I have noticed current shallow clone is mostly broken
> (at least with git < 2.5 [2]). Indeed, shallow clone only work with
> symbolic references (HEAD, a tag or a branch). However, we avoid use of
> symbolic references in VERSION.

Thanks for this contribution.

Your justification in PATCH 2 is just "That simplify workflow with
upstream". However, we already have the <pkg>_OVERRIDE_SRCDIR mechanism
(and <pkg>_SITE_METHOD = local, which is the same) to specifically
address this use case.

The idea with <pkg>_OVERRIDE_SRCDIR is that if you are actively
developing on a software component, then it should not be Buildroot's
responsibility to download/extract/patch it, but it should instead use
a locally available source directory, which is managed completely
separately from Buildroot. There you can do whatever Git, Subversion or
Mercurial version control you want, Buildroot will simply rsync to the
build directory.

I think doing development in the build directory, as encouraged by your
patch, is a bad practice. The build directory is a temporary location,
people should not be encouraged to work from there.

And I fail to see what your solution brings compared to using
<pkg>_OVERRIDE_SRCDIR or <pkg>_SITE_METHOD = local. To me, the existing
solutions are in fact more flexible and don't encourage the practice of
hacking in the build directory.

Of course, if there is a specific workflow that you could describe that
doesn't work with <pkg>_OVERRIDE_SRCDIR, then I'm definitely
interested, and from this discussion we can decide whether improvements
to OVERRIDE_SRCDIR are needed, or if a completely different solution is
needed.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 0/5] Introduce alternative archive format

[Jérôme Pouiller]

On Thursday 19 November 2015 13:02:53 Thomas Petazzoni wrote:
> J?r?me,
> 
> On Thu, 19 Nov 2015 11:36:01 +0100, J?r?me Pouiller wrote:
> > As suggested by Arnout [1], this series provide an alternative
> > archive format. This new format contains a shallowed version of
> > upstream repository. This format is a little bigger and a little
> > longer to create but it allow a better workflow with upstream. I
> > describe some good practice in patch 2.
> > 
> > Notice projects hosted by github don't yet benefit of this feature
> > since I have not found any elegant way to do it :-(.
> > 
> > During my tests, I have noticed current shallow clone is mostly
> > broken (at least with git < 2.5 [2]). Indeed, shallow clone only
> > work with symbolic references (HEAD, a tag or a branch). However,
> > we avoid use of symbolic references in VERSION.
> 
> Thanks for this contribution.
> 
> Your justification in PATCH 2 is just "That simplify workflow with
> upstream". However, we already have the <pkg>_OVERRIDE_SRCDIR
> mechanism (and <pkg>_SITE_METHOD = local, which is the same) to
> specifically address this use case.
> 
> The idea with <pkg>_OVERRIDE_SRCDIR is that if you are actively
> developing on a software component, then it should not be Buildroot's
> responsibility to download/extract/patch it, but it should instead use
> a locally available source directory, which is managed completely
> separately from Buildroot. There you can do whatever Git, Subversion
> or Mercurial version control you want, Buildroot will simply rsync to
> the build directory.
> 
> I think doing development in the build directory, as encouraged by
> your patch, is a bad practice. The build directory is a temporary
> location, people should not be encouraged to work from there.
> 
> And I fail to see what your solution brings compared to using
> <pkg>_OVERRIDE_SRCDIR or <pkg>_SITE_METHOD = local. To me, the
> existing solutions are in fact more flexible and don't encourage the
> practice of hacking in the build directory.

So I am guilty :-) . I do hack in the build directory. 

However, I think in 70% of cases, I just retrieve information from build 
directory without touching anything. Sometime, I run "make 
<pkg>-extract" only to quickly check sources of a package even if I 
don't use it in my current configuration (else I have to retrieve 
upstream URL, download tarball, extract it in a temporary directory, 
apply BR patches and don't forget to remove temporary directory).

I think it is valuable to have upstream history easily available. 

In add, I have in my drafts a patch to use "git am" to apply patches 
when possible. I think it would make patch rebasing easier when versions 
changes. It would be also easier to cherry-pick upstream patches.


> Of course, if there is a specific workflow that you could describe
> that doesn't work with <pkg>_OVERRIDE_SRCDIR, then I'm definitely
> interested, and from this discussion we can decide whether
> improvements to OVERRIDE_SRCDIR are needed, or if a completely
> different solution is needed.



Regards,

-- 
J?r?me Pouiller

[Buildroot] [PATCH 1/5] pkg-download: do not test SITE_METHOD

[Yann E. MORIN]

J?r?me, All,

On 2015-11-19 11:36 +0100, J?r?me Pouiller spake thusly:
> SITE_METHOD is always set in inner-generic-package. So, it is useless
> to test it in pkg-download.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
>  package/pkg-download.mk | 7 +------
>  1 file changed, 1 insertion(+), 6 deletions(-)
> 
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index c86b9ab..757bafe 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -222,12 +222,7 @@ define DOWNLOAD_INNER
>  		exit 1 ; \
>  	fi ; \
>  	if test -n "$(1)" ; then \
> -		if test -z "$($(PKG)_SITE_METHOD)" ; then \
> -			scheme="$(call geturischeme,$(1))" ; \
> -		else \
> -			scheme="$($(PKG)_SITE_METHOD)" ; \
> -		fi ; \
> -		case "$$scheme" in \
> +		case "$($(PKG)_SITE_METHOD)" in \
>  			git) $($(3)_GIT) && exit ;; \
>  			svn) $($(3)_SVN) && exit ;; \
>  			cvs) $($(3)_CVS) && exit ;; \
> -- 
> 2.1.4
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 0/5] Introduce alternative archive format

[Yann E. MORIN]

J?r?me, Thomas, All,

On 2015-11-19 13:02 +0100, Thomas Petazzoni spake thusly:
> On Thu, 19 Nov 2015 11:36:01 +0100, J?r?me Pouiller wrote:
> > As suggested by Arnout [1], this series provide an alternative archive
> > format. This new format contains a shallowed version of upstream
> > repository. This format is a little bigger and a little longer to
> > create but it allow a better workflow with upstream. I describe some
> > good practice in patch 2.
> > 
> > Notice projects hosted by github don't yet benefit of this feature
> > since I have not found any elegant way to do it :-(. 
> > 
> > During my tests, I have noticed current shallow clone is mostly broken
> > (at least with git < 2.5 [2]). Indeed, shallow clone only work with
> > symbolic references (HEAD, a tag or a branch). However, we avoid use of
> > symbolic references in VERSION.
> 
> Thanks for this contribution.
> 
> Your justification in PATCH 2 is just "That simplify workflow with
> upstream". However, we already have the <pkg>_OVERRIDE_SRCDIR mechanism
> (and <pkg>_SITE_METHOD = local, which is the same) to specifically
> address this use case.
> 
> The idea with <pkg>_OVERRIDE_SRCDIR is that if you are actively
> developing on a software component, then it should not be Buildroot's
> responsibility to download/extract/patch it, but it should instead use
> a locally available source directory, which is managed completely
> separately from Buildroot. There you can do whatever Git, Subversion or
> Mercurial version control you want, Buildroot will simply rsync to the
> build directory.
> 
> I think doing development in the build directory, as encouraged by your
> patch, is a bad practice. The build directory is a temporary location,
> people should not be encouraged to work from there.
> 
> And I fail to see what your solution brings compared to using
> <pkg>_OVERRIDE_SRCDIR or <pkg>_SITE_METHOD = local. To me, the existing
> solutions are in fact more flexible and don't encourage the practice of
> hacking in the build directory.
> 
> Of course, if there is a specific workflow that you could describe that
> doesn't work with <pkg>_OVERRIDE_SRCDIR, then I'm definitely
> interested, and from this discussion we can decide whether improvements
> to OVERRIDE_SRCDIR are needed, or if a completely different solution is
> needed.

I have to agree with Thomas: we already have one mechanism to do actual
development on packages, and I believe that it is the best we can offer:

  - it is not removed on 'make clean'  (the killing feature for it)

  - it can be managed however the developer wants to  (that too is a
    killing feature)

  - it is not too complex to setup

Also, I believe it covers all use-cases we can imagine.

The one thing that is made a bit more complex is gdb-ing, because path
in the build directory are referenced, rather than in the actual source
dir. IMHO, that's a minor annoyance, and it is easy to mentally match
the former to the latter.

So, except for the first patch which is indeed a nice cleanup, I'm not
too favourable to that series as a whole...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 0/5] Introduce alternative archive format

[Arnout Vandecappelle]

On 19-11-15 11:36, J?r?me Pouiller wrote:
> Hello,
> 
> As suggested by Arnout [1], this series provide an alternative archive
> format. This new format contains a shallowed version of upstream
> repository. This format is a little bigger and a little longer to
> create but it allow a better workflow with upstream. I describe some
> good practice in patch 2.

 The use case I had in mind with [1] was not so much working with upstream (for
that, OVERRIDE_SRCDIR is much more appropriate), but rather to make it cheaper
to get different upstream versions. Particularly for linux repositories, if you
downloaded one of them, you can reuse a large part of it when you get another
version. When you can do a shallow clone there is often not so much to gain from
that, but in many of our defconfigs a shallow clone is not possible because we
specify a sha. In that case, an incremental download really saves a lot of time.


 Regards,
 Arnout

> 
> Notice projects hosted by github don't yet benefit of this feature
> since I have not found any elegant way to do it :-(. 
> 
> During my tests, I have noticed current shallow clone is mostly broken
> (at least with git < 2.5 [2]). Indeed, shallow clone only work with
> symbolic references (HEAD, a tag or a branch). However, we avoid use of
> symbolic references in VERSION.
> 
> [1] http://lists.busybox.net/pipermail/buildroot/2015-November/144296.html
> [2] http://stackoverflow.com/a/30701724/301717
> 
> 
> J?r?me Pouiller (5):
>   pkg-download: do not test SITE_METHOD
>   download/git: allow to create archives containing shallowed git repos
>   pkg-generic: allow to populate build directory from a git archive
>   pkg-generic: provide an option to use git archives
>   pkg-generic: tag sources if git is used
> 
>  Config.in               | 13 +++++++++++
>  package/pkg-download.mk |  7 +-----
>  package/pkg-generic.mk  | 60 ++++++++++++++++++++++++++++++++++---------------
>  support/download/git    | 30 ++++++++++++++++++++++---
>  4 files changed, 83 insertions(+), 27 deletions(-)
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 0/5] Introduce alternative archive format

[Jérôme Pouiller]

Hello Yann,

On Sunday 29 November 2015 19:02:27 Yann E. MORIN wrote:
> J?r?me, Thomas, All,
> 
> On 2015-11-19 13:02 +0100, Thomas Petazzoni spake thusly:
> > On Thu, 19 Nov 2015 11:36:01 +0100, J?r?me Pouiller wrote:
> > > As suggested by Arnout [1], this series provide an alternative
> > > archive format. This new format contains a shallowed version of
> > > upstream repository. This format is a little bigger and a little
> > > longer to create but it allow a better workflow with upstream. I
> > > describe some good practice in patch 2.
> > > 
> > > Notice projects hosted by github don't yet benefit of this feature
> > > since I have not found any elegant way to do it :-(.
> > > 
> > > During my tests, I have noticed current shallow clone is mostly
> > > broken (at least with git < 2.5 [2]). Indeed, shallow clone only
> > > work with symbolic references (HEAD, a tag or a branch). However,
> > > we avoid use of symbolic references in VERSION.
> > 
> > Thanks for this contribution.
> > 
> > Your justification in PATCH 2 is just "That simplify workflow with
> > upstream". However, we already have the <pkg>_OVERRIDE_SRCDIR
> > mechanism (and <pkg>_SITE_METHOD = local, which is the same) to
> > specifically address this use case.
> > 
> > The idea with <pkg>_OVERRIDE_SRCDIR is that if you are actively
> > developing on a software component, then it should not be
> > Buildroot's
> > responsibility to download/extract/patch it, but it should instead
> > use a locally available source directory, which is managed
> > completely separately from Buildroot. There you can do whatever
> > Git, Subversion or Mercurial version control you want, Buildroot
> > will simply rsync to the build directory.
> > 
> > I think doing development in the build directory, as encouraged by
> > your patch, is a bad practice. The build directory is a temporary
> > location, people should not be encouraged to work from there.
> > 
> > And I fail to see what your solution brings compared to using
> > <pkg>_OVERRIDE_SRCDIR or <pkg>_SITE_METHOD = local. To me, the
> > existing solutions are in fact more flexible and don't encourage
> > the practice of hacking in the build directory.
> > 
> > Of course, if there is a specific workflow that you could describe
> > that doesn't work with <pkg>_OVERRIDE_SRCDIR, then I'm definitely
> > interested, and from this discussion we can decide whether
> > improvements to OVERRIDE_SRCDIR are needed, or if a completely
> > different solution is needed.
> 
> I have to agree with Thomas: we already have one mechanism to do
> actual development on packages, and I believe that it is the best we
> can offer:
> 
>   - it is not removed on 'make clean'  (the killing feature for it)
> 
>   - it can be managed however the developer wants to  (that too is a
>     killing feature)
> 
>   - it is not too complex to setup
> 
> Also, I believe it covers all use-cases we can imagine.
> 
> The one thing that is made a bit more complex is gdb-ing, because path
> in the build directory are referenced, rather than in the actual
> source dir. IMHO, that's a minor annoyance, and it is easy to
> mentally match the former to the latter.
> 
> So, except for the first patch which is indeed a nice cleanup, I'm not
> too favourable to that series as a whole...

hmm... You suggest to use <pkg>_OVERRIDE_SRCDIR even to do small fixes 
like build failures or version changes? In this case, I offer to provide 
a script to do same thing than in my proposal (get git repo if available 
and apply BR patches) but in an external directory. But I don't like 
this approach because it break concept that everything is launched from 
"make"


Regards,

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH 1/5] pkg-download: do not test SITE_METHOD

[Thomas Petazzoni]

Dear J?r?me Pouiller,

On Thu, 19 Nov 2015 11:36:02 +0100, J?r?me Pouiller wrote:
> SITE_METHOD is always set in inner-generic-package. So, it is useless
> to test it in pkg-download.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  package/pkg-download.mk | 7 +------
>  1 file changed, 1 insertion(+), 6 deletions(-)

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 0/5] Introduce alternative archive format

[Yann E. MORIN]

J??me, All,

On 2015-11-19 11:36 +0100, J?r?me Pouiller spake thusly:
> As suggested by Arnout [1], this series provide an alternative archive
> format. This new format contains a shallowed version of upstream
> repository. This format is a little bigger and a little longer to
> create but it allow a better workflow with upstream. I describe some
> good practice in patch 2.
> 
> Notice projects hosted by github don't yet benefit of this feature
> since I have not found any elegant way to do it :-(. 
> 
> During my tests, I have noticed current shallow clone is mostly broken
> (at least with git < 2.5 [2]). Indeed, shallow clone only work with
> symbolic references (HEAD, a tag or a branch). However, we avoid use of
> symbolic references in VERSION.
> 
> [1] http://lists.busybox.net/pipermail/buildroot/2015-November/144296.html
> [2] http://stackoverflow.com/a/30701724/301717
> 
> 
> J?r?me Pouiller (5):
>   pkg-download: do not test SITE_METHOD
>   download/git: allow to create archives containing shallowed git repos
>   pkg-generic: allow to populate build directory from a git archive
>   pkg-generic: provide an option to use git archives
>   pkg-generic: tag sources if git is used

Thanks for the proposal. :-)

Following the previous replies from Thomas and I, we've further
discussed this tonight on IRC, and we still believe this is not how
development (active or debugging) should be done, and we already provide
the _OVERRIDE_SRCDIR for that.

As such, I've marked patches 2..5 of your series as rejected in
Patchwork.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'