[Buildroot] [PATCH 01/13] pkg-infra: DOWNLOAD is never called with two arguments

[Buildroot] [PATCH 01/13] pkg-infra: DOWNLOAD is never called with two arguments

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Currently, the DOWNLOAD helper accepts one or two arguments:
  - the source URI
  - the source filename; if missing, it is derived from the URI

But we have no caller of DOWNLOAD that passes a second argument.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/pkg-download.mk | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 6d4cbcd..85842de 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -238,14 +238,13 @@ endef
 # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
 #
 # Argument 1 is the source location
-# Argument 2 is the source filename
 #
 # E.G. use like this:
-# $(call DOWNLOAD,$(FOO_SITE),$(FOO_SOURCE))
+# $(call DOWNLOAD,$(FOO_SITE))
 ################################################################################
 
 define DOWNLOAD
-	$(call DOWNLOAD_INNER,$(1),$(if $(2),$(2),$(notdir $(1))))
+	$(call DOWNLOAD_INNER,$(1),$(notdir $(1)))
 endef
 
 define DOWNLOAD_INNER
-- 
1.8.3.2

[Buildroot] [PATCH 02/13] Makefile: rename USER_HOOKS_EXTRA_ENV to EXTRA_ENV

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

This variable contains extra environment variables that we can not export
since they are clashing with some build systems (eg. BUILD_DIR with
u-boot).

So, we may need these variables for uses other than the user's hooks
for instrumentation. For example, we'll use them later on to export
BUILD_DIR to the download helper scripts.

Fix comment, too.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 Makefile               | 4 ++--
 package/Makefile.in    | 4 ++--
 package/pkg-generic.mk | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/Makefile b/Makefile
index ae868da..b18f8db 100644
--- a/Makefile
+++ b/Makefile
@@ -577,7 +577,7 @@ endif
 
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_BUILD_SCRIPT)), \
 		$(call MESSAGE,"Executing post-build script $(s)"); \
-		$(USER_HOOKS_EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
 LOCALE_WHITELIST=$(BUILD_DIR)/locales.nopurge
@@ -623,7 +623,7 @@ endif
 target-post-image: $(TARGETS_ROOTFS) target-finalize
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \
-		$(USER_HOOKS_EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+		$(EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
 toolchain-eclipse-register:
 	./support/scripts/eclipse-register-toolchain `readlink -f $(O)` $(notdir $(TARGET_CROSS)) $(BR2_ARCH)
diff --git a/package/Makefile.in b/package/Makefile.in
index 454f614..2c0b8da 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -277,11 +277,11 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \
 		PKG_CONFIG_SYSROOT_DIR="/" \
 		PKG_CONFIG_LIBDIR="$(HOST_DIR)/usr/lib/pkgconfig"
 
-# This extra environment we can not export ourselves (eg. because some
+# This is extra environment we can not export ourselves (eg. because some
 # packages use that variable internally, eg. uboot), so we have to
 # explicitly pass it to user-supplied external hooks (eg. post-build,
 # post-images)
-USER_HOOKS_EXTRA_ENV=\
+EXTRA_ENV=\
 	BUILD_DIR=$(BUILD_DIR)
 
 ################################################################################
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 339c3eb..3618332 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -58,7 +58,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # User-supplied script
 define step_user
 	@$(foreach user_hook, $(BR2_INSTRUMENTATION_SCRIPTS), \
-		$(USER_HOOKS_EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep))
+		$(EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep))
 endef
 ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),)
 GLOBAL_INSTRUMENTATION_HOOKS += step_user
-- 
1.8.3.2

[Buildroot] [PATCH 03/13] Makefile: add BR2_DL_DIR to EXTRA_ENV

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Also export BR2_DL_DIR for incoming download helper scripts.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/Makefile.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/Makefile.in b/package/Makefile.in
index 2c0b8da..5460caa 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -282,6 +282,7 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \
 # explicitly pass it to user-supplied external hooks (eg. post-build,
 # post-images)
 EXTRA_ENV=\
+	BR2_DL_DIR=$(BR2_DL_DIR) \
 	BUILD_DIR=$(BUILD_DIR)
 
 ################################################################################
-- 
1.8.3.2

[Buildroot] [PATCH 04/13] pkg-infra: move the git download helper to a script

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

The git download helper is getting a bit more complex. Fixing it in the
Makefile when it breaks (like the recent breakage with a non-existing
sha1-cset) proves to be challenging, to say the least.

Move it into a shell script in support/download/git, which will make
it much easier to read, maintain, fix and enhance in the future.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
 package/pkg-download.mk | 17 +++--------------
 support/download/git    | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+), 14 deletions(-)
 create mode 100755 support/download/git

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 85842de..15fc1d6 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -12,7 +12,7 @@ WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
 SVN := $(call qstrip,$(BR2_SVN))
 CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
-GIT := $(call qstrip,$(BR2_GIT))
+export GIT := $(call qstrip,$(BR2_GIT))
 HG := $(call qstrip,$(BR2_HG)) $(QUIET)
 SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
 SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
@@ -84,19 +84,8 @@ github = https://github.com/$(1)/$(2)/tarball/$(3)
 # problems
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	 ((test "`git ls-remote $($(PKG)_SITE) $($(PKG)_DL_VERSION)`" && \
-	   echo "Doing shallow clone" && \
-	   $(GIT) clone --depth 1 -b $($(PKG)_DL_VERSION) --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME)) || \
-	  (echo "Doing full clone" && \
-	   $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME))) && \
-	pushd $($(PKG)_BASE_NAME) > /dev/null && \
-	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ -o $(DL_DIR)/.$($(PKG)_SOURCE).tmp $($(PKG)_DL_VERSION) && \
-	gzip -c $(DL_DIR)/.$($(PKG)_SOURCE).tmp > $(DL_DIR)/$($(PKG)_SOURCE) && \
-	rm -f $(DL_DIR)/.$($(PKG)_SOURCE).tmp && \
-	popd > /dev/null && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(EXTRA_ENV) support/download/git $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+					  $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
 endef
 
 # TODO: improve to check that the given PKG_DL_VERSION exists on the remote
diff --git a/support/download/git b/support/download/git
new file mode 100755
index 0000000..96db3a9
--- /dev/null
+++ b/support/download/git
@@ -0,0 +1,37 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for git
+# Call it with:
+#   $1: git repo
+#   $2: git cset
+#   $3: package's basename (eg. foobar-1.2.3)
+#   $4: output file
+# And this environment:
+#   BR2_DL_DIR: path to Buildroot's download dir
+#   GIT       : the git command to call
+
+repo="${1}"
+cset="${2}"
+basename="${3}"
+output="${4}"
+
+repodir="${BR2_DL_DIR}/${basename}"
+
+if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
+    printf "Doing shallow clone\n"
+    ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${repodir}"
+else
+    printf "Doing full clone\n"
+    ${GIT} clone --bare "${repo}" "${repodir}"
+fi
+
+pushd "${repodir}"
+${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}"
+gzip -c "${output}.tmp" >"${output}"
+rm -f "${output}.tmp"
+popd
+
+rm -rf "${repodir}"
-- 
1.8.3.2

[Buildroot] [PATCH 05/13] pkg-infra: move the svn download helper to a script

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-download.mk |  9 +++------
 support/download/svn    | 25 +++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 6 deletions(-)
 create mode 100755 support/download/svn

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 15fc1d6..7fcbfed 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -9,7 +9,7 @@
 
 # Download method commands
 WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
-SVN := $(call qstrip,$(BR2_SVN))
+export SVN := $(call qstrip,$(BR2_SVN))
 CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
@@ -133,11 +133,8 @@ endef
 
 define DOWNLOAD_SVN
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	$(SVN) export -r $($(PKG)_DL_VERSION) $($(PKG)_SITE) $($(PKG)_DL_DIR) && \
-	$(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(EXTRA_ENV) support/download/svn $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+					  $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
 endef
 
 define SOURCE_CHECK_SVN
diff --git a/support/download/svn b/support/download/svn
new file mode 100755
index 0000000..c3ab32c
--- /dev/null
+++ b/support/download/svn
@@ -0,0 +1,25 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for svn
+# Call it with:
+#   $1: svn repo
+#   $2: svn revision
+#   $3: package's basename (eg. foobar-1.2.3)
+#   $4: output file
+# And this environment:
+#   SVN       : the svn command to call
+#   BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+rev="${2}"
+basename="${3}"
+output="${4}"
+
+pushd "${BR2_DL_DIR}"
+${SVN} export -r "${rev}" "${repo}" "${basename}"
+tar czf "${output}" "${basename}"
+rm -rf "${basename}"
+popd
-- 
1.8.3.2

[Buildroot] [PATCH 06/13] pkg-infra: move the cvs download helper to a script

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-download.mk | 11 ++++-------
 support/download/cvs    | 27 +++++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 7 deletions(-)
 create mode 100755 support/download/cvs

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 7fcbfed..b83c7df 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -10,7 +10,7 @@
 # Download method commands
 WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
 export SVN := $(call qstrip,$(BR2_SVN))
-CVS := $(call qstrip,$(BR2_CVS))
+export CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
 HG := $(call qstrip,$(BR2_HG)) $(QUIET)
@@ -114,12 +114,9 @@ endef
 
 define DOWNLOAD_CVS
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	$(CVS) -z3 -d:pserver:anonymous@$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
-		co -d $($(PKG)_BASE_NAME) -r :$($(PKG)_DL_VERSION) -P $($(PKG)_RAWNAME) && \
-	$(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(EXTRA_ENV) support/download/cvs $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
+					  $($(PKG)_DL_VERSION) $($(PKG)_RAWNAME) \
+					  $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
 endef
 
 # Not all CVS servers support ls/rls, use login to see if we can connect
diff --git a/support/download/cvs b/support/download/cvs
new file mode 100755
index 0000000..06b8647
--- /dev/null
+++ b/support/download/cvs
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for cvs
+# Call it with:
+#   $1: cvs repo
+#   $2: cvs revision
+#   $3: package's name (eg. foobar)
+#   $4: package's basename (eg. foobar-1.2.3)
+#   $5: output file
+# And this environment:
+#   CVS       : the cvs command to call
+#   BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+rev="${2}"
+rawname="${3}"
+basename="${4}"
+output="${5}"
+
+cd "${BR2_DL_DIR}"
+${CVS} -z3 -d":pserver:anonymous@${repo}" \
+       co -d "${basename}" -r ":${rev}" -P "${rawname}"
+tar czf "${output}" "${basename}"
+rm -rf "${basename}"
-- 
1.8.3.2

[Buildroot] [PATCH 07/13] pkg-infra: move the hg download helper to a script

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-download.mk | 11 +++--------
 support/download/hg     | 25 +++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 8 deletions(-)
 create mode 100755 support/download/hg

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index b83c7df..36dcfac 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -13,7 +13,7 @@ export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
-HG := $(call qstrip,$(BR2_HG)) $(QUIET)
+export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
 SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
 SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
 LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
@@ -161,13 +161,8 @@ endef
 
 define DOWNLOAD_HG
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
-	(pushd $(DL_DIR) > /dev/null && \
-	rm -rf $($(PKG)_BASE_NAME) && \
-	$(HG) clone --noupdate --rev $($(PKG)_DL_VERSION) $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
-	$(HG) archive --repository $($(PKG)_BASE_NAME) --type tgz --prefix $($(PKG)_BASE_NAME)/ \
-	              --rev $($(PKG)_DL_VERSION) $(DL_DIR)/$($(PKG)_SOURCE) && \
-	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	$(EXTRA_ENV) support/download/hg $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+					  $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
 endef
 
 # TODO: improve to check that the given PKG_DL_VERSION exists on the remote
diff --git a/support/download/hg b/support/download/hg
new file mode 100755
index 0000000..70b49cf
--- /dev/null
+++ b/support/download/hg
@@ -0,0 +1,25 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for hg
+# Call it with:
+#   $1: hg repo
+#   $2: hg cset
+#   $3: package's basename (eg. foobar-1.2.3)
+#   $4: output file
+# And this environment:
+#   HG        : the hg command to call
+#   BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+cset="${2}"
+basename="${3}"
+output="${4}"
+
+cd "${BR2_DL_DIR}"
+${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
+${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \
+              --rev "${cset}" "${output}"
+rm -rf "${basename}"
-- 
1.8.3.2

[Buildroot] [PATCH 0/13 v2] some download-related changes

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Hello All!

This series contains those download-related changes:

  - Patches 1..3  : some cleanups and preparatory works in the
    download infra,

  - Patches 4..8  : move download helpers to shell scripts, one
    such script for each download helper

  - Patches 9..10 : don't use DL_DIR as scratchpad for temporary VCS
    checkouts, since DL_DIR is a precious location

  - Patches 11..13: check hashes of downloaded files, to ensure the
    files are genuine


Changes v1 -> v2:
  - don't do any behavioural change when switching to shell scripts,
    just reproduce exactly what was in the Makefile  (Luca, Arnout)
  - support more than one hash algorithm  (Arnout, Gustavo)
  - typoes  (Luca, Baruch)


Regards,
Yann E. MORIN.


----------------------------------------------------------------
Yann E. MORIN (13):
      pkg-infra: DOWNLOAD is never called with two arguments
      Makefile: rename USER_HOOKS_EXTRA_ENV to EXTRA_ENV
      Makefile: add BR2_DL_DIR to EXTRA_ENV
      pkg-infra: move the git download helper to a script
      pkg-infra: move the svn download helper to a script
      pkg-infra: move the cvs download helper to a script
      pkg-infra: move the hg download helper to a script
      pkg-infra: move the wget download helper to a script
      pkg-infra: don't use DL_DIR as scratchpad for temporary VCS checkouts
      pkg-infra: also set PKGDIR for the download step
      pkg-infra: add possiblity to check downloaded files against known hashes
      manual: add documentation about packages' hashes
      package/ca-certificates: add tarball's hashes

 Makefile                                     |  4 +-
 docs/manual/adding-packages-directory.txt    | 57 +++++++++++++++++++
 package/Makefile.in                          |  5 +-
 package/ca-certificates/ca-certificates.hash |  3 +
 package/pkg-download.mk                      | 84 +++++++++++-----------------
 package/pkg-generic.mk                       |  3 +-
 support/download/check-hash                  | 71 +++++++++++++++++++++++
 support/download/cvs                         | 31 ++++++++++
 support/download/git                         | 40 +++++++++++++
 support/download/hg                          | 29 ++++++++++
 support/download/svn                         | 27 +++++++++
 support/download/wget                        | 23 ++++++++
 12 files changed, 322 insertions(+), 55 deletions(-)
 create mode 100644 package/ca-certificates/ca-certificates.hash
 create mode 100755 support/download/check-hash
 create mode 100755 support/download/cvs
 create mode 100755 support/download/git
 create mode 100755 support/download/hg
 create mode 100755 support/download/svn
 create mode 100755 support/download/wget

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 08/13] pkg-infra: move the wget download helper to a script

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-download.mk |  6 ++----
 support/download/wget   | 21 +++++++++++++++++++++
 2 files changed, 23 insertions(+), 4 deletions(-)
 create mode 100755 support/download/wget

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 36dcfac..c94ecba 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -8,7 +8,7 @@
 ################################################################################
 
 # Download method commands
-WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
+export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
 export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
 BZR := $(call qstrip,$(BR2_BZR))
@@ -183,9 +183,7 @@ endef
 # the next time the download is tried.
 define DOWNLOAD_WGET
 	test -e $(DL_DIR)/$(2) || \
-	($(WGET) -O $(DL_DIR)/$(2).tmp '$(call qstrip,$(1))' && \
-	 mv $(DL_DIR)/$(2).tmp $(DL_DIR)/$(2)) || \
-	(rm -f $(DL_DIR)/$(2).tmp ; exit 1)
+	$(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2)
 endef
 
 define SOURCE_CHECK_WGET
diff --git a/support/download/wget b/support/download/wget
new file mode 100755
index 0000000..7865517
--- /dev/null
+++ b/support/download/wget
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for wget
+# Call it with:
+#   $1: URL
+#   $2: output file
+# And this environment:
+#   WGET      : the wget command to call
+
+url="${1}"
+output="${2}"
+
+if ${WGET} -O "${output}.tmp" "${url}"; then
+    mv "${output}.tmp" "${output}"
+else
+    rm -f "${output}.tmp"
+    exit 1
+fi
-- 
1.8.3.2

[Buildroot] [PATCH 09/13] pkg-infra: don't use DL_DIR as scratchpad for temporary VCS checkouts

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

DL_DIR can be a very precious place for some users: they use it to store
all the downloaded archives to share across all their Buildroot (and
maybe non-Buildroot) builds.

We do not want to trash this location with our temporary VCS (git, Hg,
svn, cvs) repository clones/checkouts.

Turns out that we already have some kind of scratchpad, the BUILD_DIR.
Although it is not really a disposable location, that's the best we have
so far.

Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared
locations, sometime with little place (eg. tmpfs), and some of the
repositories we clone/checkout can be very big.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 support/download/cvs  | 16 ++++++++++------
 support/download/git  |  9 ++++++---
 support/download/hg   | 16 ++++++++++------
 support/download/svn  | 14 ++++++++------
 support/download/wget | 10 ++++++----
 5 files changed, 40 insertions(+), 25 deletions(-)

diff --git a/support/download/cvs b/support/download/cvs
index 06b8647..a0adfc3 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -11,8 +11,8 @@ set -e
 #   $4: package's basename (eg. foobar-1.2.3)
 #   $5: output file
 # And this environment:
-#   CVS       : the cvs command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   CVS      : the cvs command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 rev="${2}"
@@ -20,8 +20,12 @@ rawname="${3}"
 basename="${4}"
 output="${5}"
 
-cd "${BR2_DL_DIR}"
+repodir="${basename}.tmp-cvs-checkout"
+
+cd "${BUILD_DIR}"
+rm -rf "${repodir}" # Remove leftovers from a previous failed run
+
 ${CVS} -z3 -d":pserver:anonymous@${repo}" \
-       co -d "${basename}" -r ":${rev}" -P "${rawname}"
-tar czf "${output}" "${basename}"
-rm -rf "${basename}"
+       co -d "${repodir}" -r ":${rev}" -P "${rawname}"
+tar czf "${output}" "${repodir}"
+rm -rf "${repodir}"
diff --git a/support/download/git b/support/download/git
index 96db3a9..6321448 100755
--- a/support/download/git
+++ b/support/download/git
@@ -10,15 +10,18 @@ set -e
 #   $3: package's basename (eg. foobar-1.2.3)
 #   $4: output file
 # And this environment:
-#   BR2_DL_DIR: path to Buildroot's download dir
-#   GIT       : the git command to call
+#   GIT      : the git command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 cset="${2}"
 basename="${3}"
 output="${4}"
 
-repodir="${BR2_DL_DIR}/${basename}"
+repodir="${basename}.tmp-git-checkout"
+
+cd "${BUILD_DIR}"
+rm -rf "${repodir}" # Remove leftovers from a previous failed run
 
 if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
     printf "Doing shallow clone\n"
diff --git a/support/download/hg b/support/download/hg
index 70b49cf..f166548 100755
--- a/support/download/hg
+++ b/support/download/hg
@@ -10,16 +10,20 @@ set -e
 #   $3: package's basename (eg. foobar-1.2.3)
 #   $4: output file
 # And this environment:
-#   HG        : the hg command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   HG       : the hg command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 cset="${2}"
 basename="${3}"
 output="${4}"
 
-cd "${BR2_DL_DIR}"
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
-${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \
+repodir="${basename}.tmp-hg-checkout"
+
+cd "${BUILD_DIR}"
+rm -rf "${repodir}" # Remove leftovers from a previous failed run
+
+${HG} clone --noupdate --rev "${cset}" "${repo}" "${repodir}"
+${HG} archive --repository "${repodir}" --type tgz --prefix "${basename}" \
               --rev "${cset}" "${output}"
-rm -rf "${basename}"
+rm -rf "${repodir}"
diff --git a/support/download/svn b/support/download/svn
index c3ab32c..ada934b 100755
--- a/support/download/svn
+++ b/support/download/svn
@@ -10,16 +10,18 @@ set -e
 #   $3: package's basename (eg. foobar-1.2.3)
 #   $4: output file
 # And this environment:
-#   SVN       : the svn command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   SVN      : the svn command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 rev="${2}"
 basename="${3}"
 output="${4}"
 
-pushd "${BR2_DL_DIR}"
-${SVN} export -r "${rev}" "${repo}" "${basename}"
-tar czf "${output}" "${basename}"
-rm -rf "${basename}"
+repodir="${basename}.tmp-svn-checkout"
+
+pushd "${BUILD_DIR}"
+${SVN} export -r "${rev}" "${repo}" "${repodir}"
+tar czf "${output}" "${repodir}"
+rm -rf "${repodir}"
 popd
diff --git a/support/download/wget b/support/download/wget
index 7865517..a5b4f3a 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -8,14 +8,16 @@ set -e
 #   $1: URL
 #   $2: output file
 # And this environment:
-#   WGET      : the wget command to call
+#   WGET     : the wget command to call
 
 url="${1}"
 output="${2}"
 
-if ${WGET} -O "${output}.tmp" "${url}"; then
-    mv "${output}.tmp" "${output}"
+tmpfile="${BUILD_DIR}/${output##*/}.tmp-download"
+
+if ${WGET} -O "${tmpfile}" "${url}"; then
+    mv "${tmpfile}" "${output}"
 else
-    rm -f "${output}.tmp"
+    rm -f "${tmpfile}"
     exit 1
 fi
-- 
1.8.3.2

[Buildroot] [PATCH 10/13] pkg-infra: also set PKGDIR for the download step

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

This will be needed to get the hash file, to check the
downloaded files.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/pkg-generic.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 3618332..80389ea 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -534,6 +534,7 @@ $$($(2)_TARGET_PATCH):			RAWNAME=$(patsubst host-%,%,$(1))
 $$($(2)_TARGET_PATCH):			PKGDIR=$(pkgdir)
 $$($(2)_TARGET_EXTRACT):		PKG=$(2)
 $$($(2)_TARGET_SOURCE):			PKG=$(2)
+$$($(2)_TARGET_SOURCE):			PKGDIR=$(pkgdir)
 $$($(2)_TARGET_DIRCLEAN):		PKG=$(2)
 
 # Compute the name of the Kconfig option that correspond to the
-- 
1.8.3.2

[Buildroot] [PATCH 11/13] pkg-infra: add possiblity to check downloaded files against known hashes

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Some of the packages that Buildroot might build are sensitive packages,
related to security: openssl, dropbear, ca-certificates...

Some of those packages are downloaded over plain http, because there is
no way to get them over a secure channel, such as https.

In these dark times of pervasive surveillance, the potential for harm that
a tampered-with package could generate, we may want to check the integrity
of those sensitive packages.

So, each package may now provide a list of hashes for all files that needs
to be downloaded, and Buildroot will just fail if any downloaded file does
not match its known hash.

Hashes can be any of the sha1 or sha2 variants, and will be checked even if
the file was pre-downloaded.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Baruch Siach <baruch@tkos.co.il>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
Note: this is not a bullet-proof solution, since Buildroot may itself be
compromised. But since we do sign our releases, then we secure the list of
hashes at the same time. Only random snapshots from the repository may be
at risk of tampering, although this is highly doubtfull, given how git
stores its data.
---
 package/pkg-download.mk     | 27 +++++++++++------
 support/download/check-hash | 71 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 89 insertions(+), 9 deletions(-)
 create mode 100755 support/download/check-hash

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index c94ecba..adaccef 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -58,6 +58,17 @@ domainseparator=$(if $(1),$(1),/)
 # github(user,package,version): returns site of github repository
 github = https://github.com/$(1)/$(2)/tarball/$(3)
 
+# Helper for checking a tarball's checksum
+# If the hash does not match, remove the incorrect file
+# $(1): the path to the fies with the hashes
+# $(2): the full path to the file to check
+define VERIFY_HASH
+	if ! support/download/check-hash $(1) $(2); then \
+		rm -f $(1); \
+		exit 1; \
+	fi
+endef
+
 ################################################################################
 # The DOWNLOAD_* helpers are in charge of getting a working copy
 # of the source repository for their corresponding SCM,
@@ -147,7 +158,8 @@ endef
 # to prepend the path with a slash: scp://[user@]host:/absolutepath
 define DOWNLOAD_SCP
 	test -e $(DL_DIR)/$(2) || \
-	$(SCP) '$(call stripurischeme,$(call qstrip,$(1)))' $(DL_DIR)/$(2)
+	$(SCP) '$(call stripurischeme,$(call qstrip,$(1)))' $(DL_DIR)/$(2) && \
+	$(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
 endef
 
 define SOURCE_CHECK_SCP
@@ -175,15 +187,11 @@ define SHOW_EXTERNAL_DEPS_HG
   echo $($(PKG)_SOURCE)
 endef
 
-# Download a file using wget. Only download the file if it doesn't
-# already exist in the download directory. If the download fails,
-# remove the file (because wget -O creates a 0-byte file even if the
-# download fails).  To handle an interrupted download as well, download
-# to a temporary file first.  The temporary file will be overwritten
-# the next time the download is tried.
+
 define DOWNLOAD_WGET
 	test -e $(DL_DIR)/$(2) || \
-	$(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2)
+	$(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2) && \
+	$(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
 endef
 
 define SOURCE_CHECK_WGET
@@ -196,7 +204,8 @@ endef
 
 define DOWNLOAD_LOCALFILES
 	test -e $(DL_DIR)/$(2) || \
-		$(LOCALFILES) $(call stripurischeme,$(call qstrip,$(1))) $(DL_DIR)
+	$(LOCALFILES) $(call stripurischeme,$(call qstrip,$(1))) $(DL_DIR) && \
+	$(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
 endef
 
 define SOURCE_CHECK_LOCALFILES
diff --git a/support/download/check-hash b/support/download/check-hash
new file mode 100755
index 0000000..ae62fc9
--- /dev/null
+++ b/support/download/check-hash
@@ -0,0 +1,71 @@
+#!/bin/sh
+set -e
+
+# Helper to check a file matches its known hash
+# Call it with:
+#   $1: the full path to the file to check
+#   $2: the path of the file containing all the the expected hashes
+
+h_file="${1}"
+file="${2}"
+
+# Does the hash-file exist?
+if [ ! -f "${h_file}" ]; then
+    exit 0
+fi
+
+# Check one hash for a file
+# $1: known hash
+# $2: file (full path)
+check_one_hash() {
+    _known="${1}"
+    _file="${2}"
+
+    # What type of hash is it?
+    # Note: md5 is supported, but un-documented on-purpose.
+    # Note: sha3 is not supported, since there is currently no implemetation
+    #       (the NIST has yet to publish the parameters).
+    _h=
+    case "${#_known}" in
+        32) _h=md5;;
+        40) _h=sha1;;
+        56) _h=sha224;;
+        64) _h=sha256;;
+        96) _h=sha384;;
+       128) _h=sha512;;
+         *) # Unknown hash, exit with error
+            printf "ERROR: malformed hash '%s' for '%s'\n"  \
+                   "${_known}" "${_file##*/}" >&2
+            exit 1
+            ;;
+    esac
+
+    # Do the hashes match?
+    _hash=$( ${_h}sum "${_file}" |cut -d ' ' -f 1 )
+    if [ "${_hash}" = "${_known}" ]; then
+        printf "%s: OK (%s: %s)\n" "${_file##*/}" "${_h}" "${_hash}"
+        return 0
+    fi
+
+    printf "ERROR: %s has wrong %s hash:\n" "${_file##*/}" "${_h}" >&2
+    printf "ERROR: expected: %s\n" "${_known}" >&2
+    printf "ERROR: got     : %s\n" "${_hash}" >&2
+    printf "ERROR: Incomplete download, or MITM attack\n" >&2
+
+    exit 1
+}
+
+# Do we know one or more hashes for that file?
+while read h f; do
+    case "${h}" in
+        ''|'#'*)
+            # Skip comments and empty lines
+            continue
+            ;;
+        *)
+            if [ "${f}" = "${file##*/}" ]; then
+                check_one_hash "${h}" "${file}"
+            fi
+            ;;
+    esac
+done <"${h_file}"
-- 
1.8.3.2

[Buildroot] [PATCH 12/13] manual: add documentation about packages' hashes

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Baruch Siach <baruch@tkos.co.il>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Samuel Martin <s.martin49@gmail.com>
---
 docs/manual/adding-packages-directory.txt | 57 +++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

diff --git a/docs/manual/adding-packages-directory.txt b/docs/manual/adding-packages-directory.txt
index e56e59a..11f5c58 100644
--- a/docs/manual/adding-packages-directory.txt
+++ b/docs/manual/adding-packages-directory.txt
@@ -346,3 +346,60 @@ different way, using different infrastructures:
 
 Further formatting details: see xref:writing-rules-mk[the writing
 rules].
+
+The +.hash+ file
+~~~~~~~~~~~~~~~~
+[[adding-packages-hash]]
+
+Optionally, you can add a third file, named +libfoo.hash+, that contains
+the hashes of the downloaded files for the +libfoo+ package.
+
+The hashes stored in that file are used to validate the integrity of the
+downloaded files.
+
+The format for this file is one line for each file for which to check the
+hash, each line being space-separated, with these two fields:
+
+* the hash of the file, one of:
+** +sha1+, 40 hexa-decimal characters
+** +sha224+, 56 hexa-decimal characters
+** +sha256+, 64 hexa-decimal characters
+** +sha384+, 96 hexa-decimal characters
+** +sha512+, 128 hexa-decimal characters
+* the name of the file, without any directory component
+
+Lines starting with a +#+ sign are considered comments, and ignored. Empty
+lines are ignored.
+
+There can be more than one hash for a single file, each of its own line. In
+this case, all hashes must match.
+
+Ideally, the hashes stored in this file should match the hashes published by
+upstream, eg, on their website, in the email announcement... If no such info
+is available, then compute them yourself, and state so in a comment line
+above the hashes.
+
+*Note:* the number of spaces does not matter, so one can use spaces to
+properly align the different fields.
+
+The example below defines a +sha1+ and a +sha256+ published by upstream for
+the main +libfoo-1.2.3.tar.bz2+ tarball, plus two locally-computed hashes,
+a +sha256+ for a downloaded patch, and a +sha1+ for a downloaded binary blob:
+
+----
+# Hashes from: http://www.foosoftware.org/download/libfoo-1.2.3.tar.bz2.{sha1,sha256}:
+486fb55c3efa71148fe07895fd713ea3a5ae343a                         libfoo-1.2.3.tar.bz2
+efc8103cc3bcb06bda6a781532d12701eb081ad83e8f90004b39ab81b65d4369 libfoo-1.2.3.tar.bz2
+
+# No upstream hashes for the following:
+ff52101fb90bbfc3fe9475e425688c660f46216d7e751c4bbdb1dc85cdccacb9 libfoo-fix-blabla.patch
+2d608f3c318c6b7557d551a5a09314f03452f1a1                         libfoo-data.bin
+----
+
+If the +.hash+ file is present, and there is one or more hashes for a
+downloaded file, the hash(es) of the downloaded file must match the
+hash(es) stored in the +.hash+ file. If one hash does not match, this is
+considered an error, and Buildroot aborts.
+
+If the +.hash+ file is present, but there is no hash for a downloaded file,
+or if the +.hash+ file is missing, then no check is done.
-- 
1.8.3.2

[Buildroot] [PATCH 13/13] package/ca-certificates: add tarball's hashes

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

ca-certificates contains sensitive security-related information,
and we want to ensure the archive that we download has not been
compromised.

Add the sha1 and sha256 hashes from Debian's packaging.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Martin Bark <martin@barkynet.com>
---
 package/ca-certificates/ca-certificates.hash | 3 +++
 1 file changed, 3 insertions(+)
 create mode 100644 package/ca-certificates/ca-certificates.hash

diff --git a/package/ca-certificates/ca-certificates.hash b/package/ca-certificates/ca-certificates.hash
new file mode 100644
index 0000000..13b0c30
--- /dev/null
+++ b/package/ca-certificates/ca-certificates.hash
@@ -0,0 +1,3 @@
+# sha1 and sha256 from: $(CA_CERTIFICATES_SITE)/ca-certificates_20130906.dsc :
+7f197c1bf7c7fc82e9f8f2fec6d8cc65f6a6187b                         ca-certificates_20130906.tar.gz
+dd10520091d469e95e11e5fafb7422d3be0a66071984d09009ed3e0232cb277d ca-certificates_20130906.tar.gz
-- 
1.8.3.2

[Buildroot] [PATCH 01/13] pkg-infra: DOWNLOAD is never called with two arguments

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sun,  2 Mar 2014 18:51:34 +0100, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> Currently, the DOWNLOAD helper accepts one or two arguments:
>   - the source URI
>   - the source filename; if missing, it is derived from the URI
> 
> But we have no caller of DOWNLOAD that passes a second argument.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
>  package/pkg-download.mk | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 11/13] pkg-infra: add possiblity to check downloaded files against known hashes

[Samuel Martin]

Yann,

On Sun, Mar 2, 2014 at 6:51 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Some of the packages that Buildroot might build are sensitive packages,
> related to security: openssl, dropbear, ca-certificates...
>
> Some of those packages are downloaded over plain http, because there is
> no way to get them over a secure channel, such as https.
>
> In these dark times of pervasive surveillance, the potential for harm that
> a tampered-with package could generate, we may want to check the integrity
> of those sensitive packages.
>
> So, each package may now provide a list of hashes for all files that needs
> to be downloaded, and Buildroot will just fail if any downloaded file does
> not match its known hash.
>
> Hashes can be any of the sha1 or sha2 variants, and will be checked even if
> the file was pre-downloaded.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Baruch Siach <baruch@tkos.co.il>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> ---
> Note: this is not a bullet-proof solution, since Buildroot may itself be
> compromised. But since we do sign our releases, then we secure the list of
> hashes at the same time. Only random snapshots from the repository may be
> at risk of tampering, although this is highly doubtfull, given how git
> stores its data.
> ---
>  package/pkg-download.mk     | 27 +++++++++++------
>  support/download/check-hash | 71 +++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 89 insertions(+), 9 deletions(-)
>  create mode 100755 support/download/check-hash
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index c94ecba..adaccef 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -58,6 +58,17 @@ domainseparator=$(if $(1),$(1),/)
>  # github(user,package,version): returns site of github repository
>  github = https://github.com/$(1)/$(2)/tarball/$(3)
>
> +# Helper for checking a tarball's checksum
> +# If the hash does not match, remove the incorrect file
> +# $(1): the path to the fies with the hashes

nit:
s/fies/file/


Regards,


-- 
Samuel

[Buildroot] [PATCH 0/13 v2] some download-related changes

[Samuel Martin]

Yann, all

On Sun, Mar 2, 2014 at 6:51 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Hello All!
>
> This series contains those download-related changes:
>
>   - Patches 1..3  : some cleanups and preparatory works in the
>     download infra,
>
>   - Patches 4..8  : move download helpers to shell scripts, one
>     such script for each download helper
>
>   - Patches 9..10 : don't use DL_DIR as scratchpad for temporary VCS
>     checkouts, since DL_DIR is a precious location
>
>   - Patches 11..13: check hashes of downloaded files, to ensure the
>     files are genuine
>
>
> Changes v1 -> v2:
>   - don't do any behavioural change when switching to shell scripts,
>     just reproduce exactly what was in the Makefile  (Luca, Arnout)
>   - support more than one hash algorithm  (Arnout, Gustavo)
>   - typoes  (Luca, Baruch)
>
>
> Regards,
> Yann E. MORIN.
>

[Samuel: for the whole series, with the (nit) fix in patch 11/13]
Reviewed-by: Samuel Martin <s.martin49@gmail.com>

Regards,

-- 
Samuel

[Buildroot] [PATCH 0/13 v2] some download-related changes

[Thomas De Schampheleire]

Samuel Martin <s.martin49@gmail.com> schreef:
>Yann, all
>
>On Sun, Mar 2, 2014 at 6:51 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>>
>> Hello All!
>>
>> This series contains those download-related changes:
>>
>>   - Patches 1..3  : some cleanups and preparatory works in the
>>     download infra,
>>
>>   - Patches 4..8  : move download helpers to shell scripts, one
>>     such script for each download helper
>>
>>   - Patches 9..10 : don't use DL_DIR as scratchpad for temporary VCS
>>     checkouts, since DL_DIR is a precious location
>>
>>   - Patches 11..13: check hashes of downloaded files, to ensure the
>>     files are genuine
>>
>>
>> Changes v1 -> v2:
>>   - don't do any behavioural change when switching to shell scripts,
>>     just reproduce exactly what was in the Makefile  (Luca, Arnout)
>>   - support more than one hash algorithm  (Arnout, Gustavo)
>>   - typoes  (Luca, Baruch)
>>
>>

In this same area (but can be in separate patches) we should protect all download helpers against interrupted downloads. This is already some for wget and git, but not for hg and the others...

Best regards,
Thomas

[Buildroot] [PATCH 11/13] pkg-infra: add possiblity to check downloaded files against known hashes

[Yann E. MORIN]

Samuel, All,

On 2014-03-02 21:10 +0100, Samuel Martin spake thusly:
> Yann,
> 
> On Sun, Mar 2, 2014 at 6:51 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >
> > Some of the packages that Buildroot might build are sensitive packages,
> > related to security: openssl, dropbear, ca-certificates...
> >
> > Some of those packages are downloaded over plain http, because there is
> > no way to get them over a secure channel, such as https.
> >
> > In these dark times of pervasive surveillance, the potential for harm that
> > a tampered-with package could generate, we may want to check the integrity
> > of those sensitive packages.
> >
> > So, each package may now provide a list of hashes for all files that needs
> > to be downloaded, and Buildroot will just fail if any downloaded file does
> > not match its known hash.
> >
> > Hashes can be any of the sha1 or sha2 variants, and will be checked even if
> > the file was pre-downloaded.
> >
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Baruch Siach <baruch@tkos.co.il>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > ---
> > Note: this is not a bullet-proof solution, since Buildroot may itself be
> > compromised. But since we do sign our releases, then we secure the list of
> > hashes at the same time. Only random snapshots from the repository may be
> > at risk of tampering, although this is highly doubtfull, given how git
> > stores its data.
> > ---
> >  package/pkg-download.mk     | 27 +++++++++++------
> >  support/download/check-hash | 71 +++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 89 insertions(+), 9 deletions(-)
> >  create mode 100755 support/download/check-hash
> >
> > diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> > index c94ecba..adaccef 100644
> > --- a/package/pkg-download.mk
> > +++ b/package/pkg-download.mk
> > @@ -58,6 +58,17 @@ domainseparator=$(if $(1),$(1),/)
> >  # github(user,package,version): returns site of github repository
> >  github = https://github.com/$(1)/$(2)/tarball/$(3)
> >
> > +# Helper for checking a tarball's checksum
> > +# If the hash does not match, remove the incorrect file
> > +# $(1): the path to the fies with the hashes
> 
> nit:
> s/fies/file/

Already spotted by Gustavo on IRC. Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 0/13 v2] some download-related changes

[Yann E. MORIN]

Thomas, All,

On 2014-03-02 21:55 +0100, Thomas De Schampheleire spake thusly:
> Samuel Martin <s.martin49@gmail.com> schreef:
> >Yann, all
> >
> >On Sun, Mar 2, 2014 at 6:51 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> >> This series contains those download-related changes:
[--SNIP--]
> In this same area (but can be in separate patches) we should protect
> all download helpers against interrupted downloads. This is already
> some for wget and git, but not for hg and the others...

For all download helpers (except wget), we force-remove the local copy
(clone or checkout) before cloning/checking out, in case there were some
leftovers from a previous run.

The only phase that can still be interrupted is creating the archive
from the local copy. Even for git, this is not correctly handled:

    cvs:    tar czf "${output}" "${repodir}"
    git:    gzip -c "${output}.tmp" >"${output}"
    hg:     ${HG} archive [...] "${output}"
    svn:    tar czf "${output}" "${repodir}"

which leaves open a (small) window were we could have a problem.

Only wget gets it right:
    mv "${tmpfile}" "${output}"

So, yes, we can do better. I'll fix and resubmit.

Thank you!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 03/13] Makefile: add BR2_DL_DIR to EXTRA_ENV

[Arnout Vandecappelle]

On 02/03/14 18:51, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> Also export BR2_DL_DIR for incoming download helper scripts.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

> ---
>  package/Makefile.in | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/Makefile.in b/package/Makefile.in
> index 2c0b8da..5460caa 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -282,6 +282,7 @@ HOST_MAKE_ENV=PATH=$(HOST_PATH) \
>  # explicitly pass it to user-supplied external hooks (eg. post-build,
>  # post-images)
>  EXTRA_ENV=\
> +	BR2_DL_DIR=$(BR2_DL_DIR) \
>  	BUILD_DIR=$(BUILD_DIR)
>  
>  ################################################################################
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 04/13] pkg-infra: move the git download helper to a script

[Arnout Vandecappelle]

On 02/03/14 18:51, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> The git download helper is getting a bit more complex. Fixing it in the
> Makefile when it breaks (like the recent breakage with a non-existing
> sha1-cset) proves to be challenging, to say the least.
> 
> Move it into a shell script in support/download/git, which will make
> it much easier to read, maintain, fix and enhance in the future.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Arnout Vandecappelle <arnout@mind.be>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>


 Regards,
 Arnout

> ---
>  package/pkg-download.mk | 17 +++--------------
>  support/download/git    | 37 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 40 insertions(+), 14 deletions(-)
>  create mode 100755 support/download/git
[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F