[Buildroot] [pull request] core/download: add support for git sub-modules

[Buildroot] [pull request] core/download: add support for git sub-modules

[Yann E. MORIN]

Hello All!

This little series adds support for packages coming from a git clone,
with git sub-modules.

The commit logs are pretty detailed, but roughly, it goes as thus;

  - we can't get sub-modules in a bare clone, so we no longer use bare
    clones, but use full clones (with a working copy)

  - we enhance the git download wrapper to retrieve sub-modules if
    needed,

  - we expose a new package variable to request sub-modules.

Regards,
Yann E. MORIN.


The following changes since commit 0b7b84310c3781c346eea4c114c8cb4e97bbd8ea:

  olimex_imx233_olinuxino_defconfig: genimage support (2016-03-11 13:14:54 +0100)

are available in the git repository at:

  git://git.busybox.net/~ymorin/git/buildroot yem/git

for you to fetch changes up to 8386db648bd2fde9ca492c57b2b92c09cdcfa18d:

  core/pkg-infra: downlaod git submodules if the package wants them (2016-03-11 19:28:45 +0100)

----------------------------------------------------------------
Yann E. MORIN (3):
      support/download/git: do not use bare clones
      support/download/git: add support for submodules
      core/pkg-infra: downlaod git submodules if the package wants them

 package/pkg-download.mk |  1 +
 package/pkg-generic.mk  |  8 ++++++++
 support/download/git    | 46 ++++++++++++++++++++++++++++++++++++++++------
 3 files changed, 49 insertions(+), 6 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/3] support/download/git: do not use bare clones

[Yann E. MORIN]

Currently, we are using bare clones, so as to minimise the disk usage,
most notably for largeish repositories such as the one for the Linux
kernel, which can go beyond the 1GiB barrier.

However, this precludes updating (and thus using) the submodules, if
any, of the repositories, as a working copy is required to use
submodules (becaue we need to know the list of submodules, where to find
them, where to clone them, what cset to checkout, and all those is
dependent upon the checked out cset of the father repository).

Switch to using /plain/ clones with a working copy.

This means that the extra refs used by some forges (like pull-requests
for Github, or changes for gerrit...) are no longer fetched as part of
the clone, because git does not offer to do a mirror clone when there is
a working copy.

Instead, we have to fetch those special refs by hand. Since there is no
easy solution to know whether the cset the user asked for is such a
special ref or not, we just try to always fetch the cset requested by
the user; if this fails, we assume that this is not a special ref (most
probably, it is a sha1) and we defer the check to the checkout, which
would fail if the requested cset is missing anyway.

Furthermore, we can no longer rely on git to generate reproducible
tarballs, so we have to handle it all manually:
  - get the date of the commit to store in the archive,
  - store only numeric owners,
  - store owner and group as 0 (zero, although any arbitrary value
    would have been fine, as long as it's a constant),
  - sort the files to store in the archive.

Finally, we get rid of the .git directory as it is not very usefull in a
tarball.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

---
Note about removing .git : yes, we could keep it, at the expense of much
larger size of the generated archive. Some people would like the .git to
stay, to speed-up later downloads. However, there's no easy way we could
do that. For example:
  - clone foo-12345, keep the .git n the tarball
  - update buildroot
  - clone foo-98765
For that second clone, how could we know we have to extract foo-12345
first? So, the .git in the archive is pretty much useless for Buildroot.
---
 support/download/git | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/support/download/git b/support/download/git
index 314b388..5672217 100755
--- a/support/download/git
+++ b/support/download/git
@@ -41,7 +41,7 @@ _git() {
 git_done=0
 if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
     printf "Doing shallow clone\n"
-    if _git clone ${verbose} --depth 1 -b "'${cset}'" --bare "'${repo}'" "'${basename}'"; then
+    if _git clone ${verbose} --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
         git_done=1
     else
         printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -49,10 +49,33 @@ if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
 fi
 if [ ${git_done} -eq 0 ]; then
     printf "Doing full clone\n"
-    _git clone ${verbose} --mirror "'${repo}'" "'${basename}'"
+    _git clone ${verbose} "'${repo}'" "'${basename}'"
 fi
 
-GIT_DIR="${basename}" \
-_git archive --prefix="'${basename}/'" -o "'${output}.tmp'" --format=tar "'${cset}'"
+pushd "${basename}" >/dev/null
 
+# Try to get the special refs exposed by some forges (pull-requests for
+# github, changes for gerrit...). There is no easy way to know whether
+# the cset the user passed us is such a special ref or a tag or a sha1
+# or whatever else. We'll eventually fail at checking out that cset,
+# below, if there is an issue anyway. Since most of the cset we're gonna
+# have to clone are not such special refs, consign the output to oblivion
+# so as not to alarm unsuspecting users, but still trace it as a warning.
+if ! _git fetch "'${cset}:${cset}'" >/dev/null 2>&1; then
+    printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
+fi
+
+# Checkout the required changeset.
+_git checkout -q "'${cset}'"
+
+# Get date of commit to generate a reproducible archive.
+date="$( _git show --no-patch --pretty=format:%cD )"
+
+# We do not need the .git dir to generate the tarball
+rm -rf .git
+
+popd >/dev/null
+
+tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" \
+         -T <(find "${basename}" -not -type d |sort) >"${output}.tmp"
 gzip -n <"${output}.tmp" >"${output}"
-- 
1.9.1

[Buildroot] [PATCH 2/3] support/download/git: add support for submodules

[Yann E. MORIN]

Some git repositories may be split into a master repository and
submodules. Up until now, we did not have support for submodules,
because we wer using bare clones, in which it is not possible to
update the list of submodules.

Now that we are using plain clones with a working copy, we can retrieve
the submdoules.

Add an option to the git downlaod helper to kick the update of
submodules, so that they are only fetched for those packages that
require them. Also document the existing -q option at the same time.

Submodules have a .git file at their root, which contains the path to
the real .git directory of the master repository. Since we remove it,
there is no point in keeping those .git files either.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 support/download/git | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/support/download/git b/support/download/git
index 5672217..8df3934 100755
--- a/support/download/git
+++ b/support/download/git
@@ -6,15 +6,20 @@ set -e
 # Download helper for git, to be called from the download wrapper script
 #
 # Call it as:
-#   .../git [-q] OUT_FILE REPO_URL CSET BASENAME
+#   .../git [-q] [-r] OUT_FILE REPO_URL CSET BASENAME
+#
+#   -q  Be quiet.
+#   -r  Clone and archive sub-modules.
 #
 # Environment:
 #   GIT      : the git command to call
 
 verbose=
-while getopts :q OPT; do
+recurse=0
+while getopts :qr OPT; do
     case "${OPT}" in
     q)  verbose=-q; exec >/dev/null;;
+    r)  recurse=1;;
     \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
     esac
 done
@@ -61,18 +66,24 @@ pushd "${basename}" >/dev/null
 # below, if there is an issue anyway. Since most of the cset we're gonna
 # have to clone are not such special refs, consign the output to oblivion
 # so as not to alarm unsuspecting users, but still trace it as a warning.
-if ! _git fetch "'${cset}:${cset}'" >/dev/null 2>&1; then
+if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
     printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
 fi
 
-# Checkout the required changeset.
+# Checkout the required changeset, so that we can update the required
+# submodules.
 _git checkout -q "'${cset}'"
 
 # Get date of commit to generate a reproducible archive.
 date="$( _git show --no-patch --pretty=format:%cD )"
 
-# We do not need the .git dir to generate the tarball
-rm -rf .git
+# There might be submodules, so fetch them.
+if [ ${recurse} -eq 1 ]; then
+    _git submodule update --init --recursive
+fi
+
+# We do not need the .git dir and files to generate the tarball
+find . -name .git -exec rm -rf {} +
 
 popd >/dev/null
 
-- 
1.9.1

[Buildroot] [PATCH 3/3] core/pkg-infra: downlaod git submodules if the package wants them

[Yann E. MORIN]

Add a new package variable that packages can set to specify that they
need git submodules.

Only accept this option if the downlaod method is git, as we can not get
submodules via an http download (via wget).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-download.mk | 1 +
 package/pkg-generic.mk  | 8 ++++++++
 2 files changed, 9 insertions(+)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 1332e66..2324a07 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -76,6 +76,7 @@ export BR_NO_CHECK_HASH_FOR =
 define DOWNLOAD_GIT
 	$(EXTRA_ENV) $(DL_WRAPPER) -b git \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
+		$(if $($(PKG)_GIT_SUBMODULES),-r) \
 		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 3904c09..fee7eb0 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -453,6 +453,14 @@ ifndef $(2)_SITE_METHOD
  endif
 endif
 
+# Do not accept to download git submodule if not using the git method
+ifneq ($$($(2)_GIT_SUBMODULES),)
+ ifneq ($$($(2)_SITE_METHOD),git)
+  $$(error $(2) declares having git sub-modules, but does not use the \
+	   'git' method (uses '$$($(2)_SITE_METHOD)' instead))
+ endif
+endif
+
 ifeq ($$($(2)_SITE_METHOD),local)
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 $(2)_OVERRIDE_SRCDIR = $$($(2)_SITE)
-- 
1.9.1