[Buildroot] [PATCH 00/16] Enable hash checking for git downloads

[Buildroot] [PATCH 00/16] Enable hash checking for git downloads

[Arnout Vandecappelle]

Since our git downloads are reproducible since a while now, we can
actually check the hash of git downloads. This does however still
require a number of fixes first.

Patch 1 fixes the case when tar is compiled to generated Posix format
tarballs.

Patch 2 is just a cleanup. This patch is optional.

Patch 3-5 fix incorrect hash files, which weren't noticed before
because hashes weren't checked.

Patch 6-8 adds a hash for packages that actually have a hash file but
set the hash type to 'none'. These patches are optional.

Patches 9-15 add exclusions for the different packages with CUSTOM_GIT
configuration.

Patch 16, finally, enables the hash check.

Regards,
Arnout

[PATCH 01/16] download/git: create GNU format tar files
[PATCH 02/16] aer-inject: remove redundant _SITE_METHOD
[PATCH 03/16] fmc: correct hash file
[PATCH 04/16] linux-firmware: correct hash
[PATCH 05/16] squashfs: correct hash
[PATCH 06/16] ubus: add hash
[PATCH 07/16] uhttpd: add hash
[PATCH 08/16] vboot-utils: add hash
[PATCH 09/16] linux: exclude from hash check except for latest
[PATCH 10/16] linux-headers: rework hash exclusion
[PATCH 11/16] uboot: exclude from hash check except for latest
[PATCH 12/16] barebox: exclude from hash check except for latest
[PATCH 13/16] at91bootstrap3: exclude from hash when downloading from
[PATCH 14/16] mxs-bootlets: exclude from hash when downloading from
[PATCH 15/16] arm-trusted-firmware: exclude from hash when
[PATCH 16/16] pkg-download: enable hash check for git downloads

[Buildroot] [PATCH 01/16] download/git: create GNU format tar files

[Arnout Vandecappelle]

On most distros, the tar format defaults to GNU. However, at build time
the default format may be changed to posix. Also, future versions of
tar will default to posix.

Since we want the tarballs created by the git download method to be
reproducible (so their hash can be checked), we should explicitly
specify the format. Since existing tarballs on sources.buildroot.org
use the GNU format, and also the existing hashes in the *.hash files
are based on GNU format tarballs, we use the GNU format.

In addition, the Posix format encodes atime and ctime as well as mtime,
but tar offers no option like --mtime to override them. In the GNU
format, atime and ctime are only encoded if the --incremental option is
given.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Peter Seiderer <ps.report@gmx.net>
---
Peter, can you test if this solves the issue for you?
---
 support/download/git | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/support/download/git b/support/download/git
index 7921411835..056057c700 100755
--- a/support/download/git
+++ b/support/download/git
@@ -94,6 +94,8 @@ popd >/dev/null
 # Generate the archive, sort with the C locale so that it is reproducible
 find "${basename}" -not -type d >"${basename}.list"
 LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
-tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" \
+# Create GNU-format tarballs, since that's the format of the tarballs on
+# sources.buildroot.org and used in the *.hash files
+tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
          -T "${basename}.list.sorted" >"${output}.tar"
 gzip -n <"${output}.tar" >"${output}"
-- 
2.11.0

[Buildroot] [PATCH 01/16] download/git: create GNU format tar files

[Thomas Petazzoni]

Hello,

On Tue, 21 Mar 2017 01:06:57 +0100, Arnout Vandecappelle
(Essensium/Mind) wrote:
> On most distros, the tar format defaults to GNU. However, at build time
> the default format may be changed to posix. Also, future versions of
> tar will default to posix.
> 
> Since we want the tarballs created by the git download method to be
> reproducible (so their hash can be checked), we should explicitly
> specify the format. Since existing tarballs on sources.buildroot.org
> use the GNU format, and also the existing hashes in the *.hash files
> are based on GNU format tarballs, we use the GNU format.
> 
> In addition, the Posix format encodes atime and ctime as well as mtime,
> but tar offers no option like --mtime to override them. In the GNU
> format, atime and ctime are only encoded if the --incremental option is
> given.
> 
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: Peter Seiderer <ps.report@gmx.net>
> ---
> Peter, can you test if this solves the issue for you?
> ---
>  support/download/git | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)

Applied to master, thanks. We require tar 1.17 at least, and I checked
that tar 1.17 supports --format=gnu.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 01/16] download/git: create GNU format tar files

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> writes:

 > On most distros, the tar format defaults to GNU. However, at build time
 > the default format may be changed to posix. Also, future versions of
 > tar will default to posix.

 > Since we want the tarballs created by the git download method to be
 > reproducible (so their hash can be checked), we should explicitly
 > specify the format. Since existing tarballs on sources.buildroot.org
 > use the GNU format, and also the existing hashes in the *.hash files
 > are based on GNU format tarballs, we use the GNU format.

 > In addition, the Posix format encodes atime and ctime as well as mtime,
 > but tar offers no option like --mtime to override them. In the GNU
 > format, atime and ctime are only encoded if the --incremental option is
 > given.

 > Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
 > Cc: Peter Seiderer <ps.report@gmx.net>
 > ---
 > Peter, can you test if this solves the issue for you?

Committed to 2017.02.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 02/16] aer-inject: remove redundant _SITE_METHOD

[Arnout Vandecappelle]

The _SITE has a git:// URL, so _SITE_METHOD is not needed.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Tiago Brusamarello <tiago.brusamarello@datacom.ind.br>
---
 package/aer-inject/aer-inject.mk | 1 -
 1 file changed, 1 deletion(-)

diff --git a/package/aer-inject/aer-inject.mk b/package/aer-inject/aer-inject.mk
index 69189b1668..e00570e272 100644
--- a/package/aer-inject/aer-inject.mk
+++ b/package/aer-inject/aer-inject.mk
@@ -6,7 +6,6 @@
 
 AER_INJECT_VERSION = 9bd5e2c7886fca72f139cd8402488a2235957d41
 AER_INJECT_SITE = git://git.kernel.org/pub/scm/linux/kernel/git/gong.chen/aer-inject.git
-AER_INJECT_SITE_METHOD = git
 AER_INJECT_LICENSE = GPLv2
 AER_INJECT_LICENSE_FILES = README
 AER_INJECT_DEPENDENCIES = host-flex host-bison
-- 
2.11.0

[Buildroot] [PATCH 03/16] fmc: correct hash file

[Arnout Vandecappelle]

It had sha256sum instead of sha256. This wasn't noticed before since
hashes for git downloads are not checked.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Matthew Weber <matthew.weber@rockwellcollins.com>
---
 package/fmc/fmc.hash | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/fmc/fmc.hash b/package/fmc/fmc.hash
index 2402831855..13d9d77ee5 100644
--- a/package/fmc/fmc.hash
+++ b/package/fmc/fmc.hash
@@ -1,2 +1,2 @@
-# Locally Computed
-sha256sum	a91e0c9b7c7f238634c64a755c05671f33f2acdb6ae2d09cad4d683b364ee8e4	fmc-fsl-sdk-v2.0.tar.gz
+# Locally calculated
+sha256	a91e0c9b7c7f238634c64a755c05671f33f2acdb6ae2d09cad4d683b364ee8e4	fmc-fsl-sdk-v2.0.tar.gz
-- 
2.11.0

[Buildroot] [PATCH 03/16] fmc: correct hash file

[Matthew Weber]

Arnout,

On Mon, Mar 20, 2017 at 7:06 PM, Arnout Vandecappelle (Essensium/Mind)
<arnout@mind.be> wrote:
> It had sha256sum instead of sha256. This wasn't noticed before since
> hashes for git downloads are not checked.
>
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Cc: Matthew Weber <matthew.weber@rockwellcollins.com>

Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>

> ---
>  package/fmc/fmc.hash | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/package/fmc/fmc.hash b/package/fmc/fmc.hash
> index 2402831855..13d9d77ee5 100644
> --- a/package/fmc/fmc.hash
> +++ b/package/fmc/fmc.hash
> @@ -1,2 +1,2 @@
> -# Locally Computed
> -sha256sum      a91e0c9b7c7f238634c64a755c05671f33f2acdb6ae2d09cad4d683b364ee8e4        fmc-fsl-sdk-v2.0.tar.gz
> +# Locally calculated
> +sha256 a91e0c9b7c7f238634c64a755c05671f33f2acdb6ae2d09cad4d683b364ee8e4        fmc-fsl-sdk-v2.0.tar.gz
> --
> 2.11.0
>



-- 
Matthew L Weber / Pr Software Engineer
Airborne Information Systems / Security Systems and Software / Secure Platforms
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
www.rockwellcollins.com

Note: Any Export License Required Information and License Restricted
Third Party Intellectual Property (TPIP) content must be encrypted and
sent to matthew.weber at corp.rockwellcollins.com.

[Buildroot] [PATCH 03/16] fmc: correct hash file

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> writes:

 > It had sha256sum instead of sha256. This wasn't noticed before since
 > hashes for git downloads are not checked.

 > Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
 > Cc: Matthew Weber <matthew.weber@rockwellcollins.com>

Committed to 2017.02.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 04/16] linux-firmware: correct hash

[Arnout Vandecappelle]

When linux-firmware was bumped to 6d3bc8886 in commit 3ff5896ff, a hash
file was added but the hash was wrong. This wasn't noticed before since
hashes are not checked for git downloads.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Adam Duskett <aduskett@codeblue.com>
---
 package/linux-firmware/linux-firmware.hash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/linux-firmware/linux-firmware.hash b/package/linux-firmware/linux-firmware.hash
index e8ee2b72a1..164603947e 100644
--- a/package/linux-firmware/linux-firmware.hash
+++ b/package/linux-firmware/linux-firmware.hash
@@ -1,2 +1,2 @@
 # Locally calculated
-sha256 256378564e36b8f222adb8fbe746529e73413c6cbe2b0e5574f41dfd62d92577 linux-firmware-6d3bc8886517d171068fd1263176b8b5c51df204.tar.gz
+sha256 1cd14477a392fe8c7c44e62623b85466df90ed72a55ae213b637e1c4335739fd linux-firmware-6d3bc8886517d171068fd1263176b8b5c51df204.tar.gz
-- 
2.11.0

[Buildroot] [PATCH 04/16] linux-firmware: correct hash

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> writes:

 > When linux-firmware was bumped to 6d3bc8886 in commit 3ff5896ff, a hash
 > file was added but the hash was wrong. This wasn't noticed before since
 > hashes are not checked for git downloads.

 > Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
 > Cc: Adam Duskett <aduskett@codeblue.com>

Committed to 2017.02.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 05/16] squashfs: correct hash

[Arnout Vandecappelle]

When squashfs was bumped to 3de1687d in commit ee90313c64, the hash
file was not updated. This wasn't noticed before since hashes are not
checked for git downloads.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
 package/squashfs/squashfs.hash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/squashfs/squashfs.hash b/package/squashfs/squashfs.hash
index eb456be364..74fdb44354 100644
--- a/package/squashfs/squashfs.hash
+++ b/package/squashfs/squashfs.hash
@@ -1,2 +1,2 @@
 # Locally computed
-sha256	e807202ab4dcfeea214df49d3a728b9a85644a7e7262fc6fcd875a0def99d4ed	squashfs-9c1db6d13a51a2e009f0027ef336ce03624eac0d.tar.gz
+sha256	2c49392816615b73bf5d6cdce9776cb66a01f5a1494b1d9af580667e32bab001  squashfs-3de1687d7432ea9b302c2db9521996f506c140a3.tar.gz
-- 
2.11.0

[Buildroot] [PATCH 05/16] squashfs: correct hash

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> writes:

 > When squashfs was bumped to 3de1687d in commit ee90313c64, the hash
 > file was not updated. This wasn't noticed before since hashes are not
 > checked for git downloads.

 > Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
 > Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

Committed to 2017.02.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 06/16] ubus: add hash

[Arnout Vandecappelle]

ubus had a hash file, but is says "none" because it is a git download.
Since our git downloads are not reproducible, we can actually add a
hash.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/ubus/ubus.hash | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/ubus/ubus.hash b/package/ubus/ubus.hash
index af9b652b0e..4e6a755c7f 100644
--- a/package/ubus/ubus.hash
+++ b/package/ubus/ubus.hash
@@ -1,2 +1,2 @@
-# No hash for this git snapshot
-none  xxx  ubus-259450f414d8c9ee41896e8e6d6bc57ec00e2b63.tar.gz
+# Locally calculated
+sha256  0dbf75c2103f3f567b9bf1bb719ea9776d18401bbf2dfd8bed2c2ebbdd1021da  ubus-259450f414d8c9ee41896e8e6d6bc57ec00e2b63.tar.gz
-- 
2.11.0

[Buildroot] [PATCH 06/16] ubus: add hash

[Thomas Petazzoni]

Hello,

On Tue, 21 Mar 2017 01:07:02 +0100, Arnout Vandecappelle
(Essensium/Mind) wrote:
> ubus had a hash file, but is says "none" because it is a git download.
> Since our git downloads are not reproducible, we can actually add a

I guess you meant "are *now* reproducible", right? Ditto for the next
commits. No need to resend just for that, it can be fixed up when
applying.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 06/16] ubus: add hash

[Arnout Vandecappelle]

On 21-03-17 10:47, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 21 Mar 2017 01:07:02 +0100, Arnout Vandecappelle
> (Essensium/Mind) wrote:
>> ubus had a hash file, but is says "none" because it is a git download.
>> Since our git downloads are not reproducible, we can actually add a
> 
> I guess you meant "are *now* reproducible", right? 

 Right. Looks like my fingers were quicker than my brains again.


> Ditto for the next
> commits. 

 Ah, the joys of 'git commit -c HEAD' :-)

 Regards,
 Arnout


> No need to resend just for that, it can be fixed up when
> applying.
> 
> Thomas
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 07/16] uhttpd: add hash

[Arnout Vandecappelle]

uhttpd had a hash file, but is says "none" because it is a git download.
Since our git downloads are not reproducible, we can actually add a
hash.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/uhttpd/uhttpd.hash | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/uhttpd/uhttpd.hash b/package/uhttpd/uhttpd.hash
index 59e54ad49a..bfeb5571c5 100644
--- a/package/uhttpd/uhttpd.hash
+++ b/package/uhttpd/uhttpd.hash
@@ -1,2 +1,2 @@
-# No hash for this git snapshot
-none	xxx	uhttpd-59e0c739634f46a164d939e54416287b91ff8a9b.tar.gz
+# Locally calculated
+sha256 b01aec584daaa367b85eddd950da27e3a41bcb265507cf42f79ff2ebcf0972d4  uhttpd-59e0c739634f46a164d939e54416287b91ff8a9b.tar.gz
-- 
2.11.0

[Buildroot] [PATCH 08/16] vboot-utils: add hash

[Arnout Vandecappelle]

vboot-utils had a hash file, but is says "none" because it is a git
download.  Since our git downloads are not reproducible, we can actually
add a hash.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/vboot-utils/vboot-utils.hash | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/vboot-utils/vboot-utils.hash b/package/vboot-utils/vboot-utils.hash
index e16142389b..9a5b334324 100644
--- a/package/vboot-utils/vboot-utils.hash
+++ b/package/vboot-utils/vboot-utils.hash
@@ -1,2 +1,2 @@
-# Git shapshot
-none	xxx	bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz
+# Locally computed
+sha256  e119782a374655117e3d9a4e667b0056c76961c4593ba907f860d1310f6fbc2a  vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz
-- 
2.11.0

[Buildroot] [PATCH 09/16] linux: exclude from hash check except for latest version

[Arnout Vandecappelle]

We don't have a hash for any linux version. We currently also don't
have a hash for the latest version, but if we ever add a hash, it
will only be for the latest version.

This simplifies the code a little because soon we will add hash checks
for git as well, so we also need an exclusion in that case. It's much
easier to always exclude except in the single case where we do have a
hash.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 linux/linux.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/linux/linux.mk b/linux/linux.mk
index 7f4432e7b1..a52083f00a 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -20,7 +20,6 @@ ifeq ($(BR2_LINUX_KERNEL_CUSTOM_TARBALL),y)
 LINUX_TARBALL = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION))
 LINUX_SITE = $(patsubst %/,%,$(dir $(LINUX_TARBALL)))
 LINUX_SOURCE = $(notdir $(LINUX_TARBALL))
-BR_NO_CHECK_HASH_FOR += $(LINUX_SOURCE)
 else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_GIT),y)
 LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_SITE_METHOD = git
@@ -32,9 +31,6 @@ LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_SITE_METHOD = svn
 else
 LINUX_SOURCE = linux-$(LINUX_VERSION).tar.xz
-ifeq ($(BR2_LINUX_KERNEL_CUSTOM_VERSION),y)
-BR_NO_CHECK_HASH_FOR += $(LINUX_SOURCE)
-endif
 # In X.Y.Z, get X and Y. We replace dots and dashes by spaces in order
 # to use the $(word) function. We support versions such as 4.0, 3.1,
 # 2.6.32, 2.6.32-rc1, 3.0-rc6, etc.
@@ -51,6 +47,10 @@ LINUX_SITE := $(LINUX_SITE)/testing
 endif # -rc
 endif
 
+ifeq ($(BR2_LINUX_KERNEL)$(BR2_LINUX_KERNEL_LATEST_VERSION),y)
+BR_NO_CHECK_HASH_FOR += $(LINUX_SOURCE)
+endif
+
 LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))
 
 # We rely on the generic package infrastructure to download and apply
-- 
2.11.0

[Buildroot] [PATCH 10/16] linux-headers: rework hash exclusion

[Arnout Vandecappelle]

Although we currently don't have a .hash file for linux-headers, there
already are exclusions for the BR2_KERNEL_HEADERS_AS_KERNEL case (copied
from linux.mk).  However, there is no exclusion for the
BR2_KERNEL_HEADERS_VERSION case.

For the BR2_KERNEL_HEADERS_AS_KERNEL case, the exclusion is actually
not needed. Indeed, KERNEL_HEADERS_SOURCE is computed to be the same
value as LINUX_SOURCE, and linux.mk already adds LINUX_SOURCE to
BR_NO_CHECK_HASH_FOR.

For the other cases, we should exclude the BR2_KERNEL_HEADERS_VERSION
case because there the user supplies the version so it can't be
included in the .hash file.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/linux-headers/linux-headers.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/package/linux-headers/linux-headers.mk b/package/linux-headers/linux-headers.mk
index 8f339cfec5..d83b9b0b02 100644
--- a/package/linux-headers/linux-headers.mk
+++ b/package/linux-headers/linux-headers.mk
@@ -16,7 +16,6 @@ ifeq ($(BR2_LINUX_KERNEL_CUSTOM_TARBALL),y)
 LINUX_HEADERS_TARBALL = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION))
 LINUX_HEADERS_SITE = $(patsubst %/,%,$(dir $(LINUX_HEADERS_TARBALL)))
 LINUX_HEADERS_SOURCE = $(notdir $(LINUX_HEADERS_TARBALL))
-BR_NO_CHECK_HASH_FOR += $(LINUX_HEADERS_SOURCE)
 else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_GIT),y)
 LINUX_HEADERS_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_HEADERS_SITE_METHOD = git
@@ -29,9 +28,6 @@ LINUX_HEADERS_SITE_METHOD = hg
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.gz
 else
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
-ifeq ($(BR2_LINUX_KERNEL_CUSTOM_VERSION),y)
-BR_NO_CHECK_HASH_FOR += $(LINUX_HEADERS_SOURCE)
-endif
 # In X.Y.Z, get X and Y. We replace dots and dashes by spaces in order
 # to use the $(word) function. We support versions such as 4.0, 3.1,
 # 2.6.32, 2.6.32-rc1, 3.0-rc6, etc.
@@ -80,6 +76,10 @@ LINUX_HEADERS_SITE = $(BR2_KERNEL_MIRROR)/linux/kernel/v4.x
 endif
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
 
+ifeq ($(BR2_KERNEL_HEADERS_VERSION),y)
+BR_NO_CHECK_HASH_FOR += $(LINUX_HEADERS_SOURCE)
+endif
+
 endif # ! BR2_KERNEL_HEADERS_AS_KERNEL
 
 LINUX_HEADERS_LICENSE = GPLv2
-- 
2.11.0

[Buildroot] [PATCH 11/16] uboot: exclude from hash check except for latest version

[Arnout Vandecappelle]

Instead of excluding the hash specifically for the custom tarball and
custom version cases, exclude it always except in the one case where
we do have a hash: the latest version.

This simplifies the code a little because soon we will add hash checks
for git as well, so we also need an exclusion in that case.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 boot/uboot/uboot.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/boot/uboot/uboot.mk b/boot/uboot/uboot.mk
index 1e22eaa8ea..705863dd6a 100644
--- a/boot/uboot/uboot.mk
+++ b/boot/uboot/uboot.mk
@@ -17,7 +17,6 @@ ifeq ($(UBOOT_VERSION),custom)
 UBOOT_TARBALL = $(call qstrip,$(BR2_TARGET_UBOOT_CUSTOM_TARBALL_LOCATION))
 UBOOT_SITE = $(patsubst %/,%,$(dir $(UBOOT_TARBALL)))
 UBOOT_SOURCE = $(notdir $(UBOOT_TARBALL))
-BR_NO_CHECK_HASH_FOR += $(UBOOT_SOURCE)
 else ifeq ($(BR2_TARGET_UBOOT_CUSTOM_GIT),y)
 UBOOT_SITE = $(call qstrip,$(BR2_TARGET_UBOOT_CUSTOM_REPO_URL))
 UBOOT_SITE_METHOD = git
@@ -31,9 +30,10 @@ else
 # Handle stable official U-Boot versions
 UBOOT_SITE = ftp://ftp.denx.de/pub/u-boot
 UBOOT_SOURCE = u-boot-$(UBOOT_VERSION).tar.bz2
-ifeq ($(BR2_TARGET_UBOOT_CUSTOM_VERSION),y)
-BR_NO_CHECK_HASH_FOR += $(UBOOT_SOURCE)
 endif
+
+ifeq ($(BR2_TARGET_UBOOT)$(BR2_TARGET_UBOOT_LATEST_VERSION),y)
+BR_NO_CHECK_HASH_FOR += $(UBOOT_SOURCE)
 endif
 
 ifeq ($(BR2_TARGET_UBOOT_FORMAT_BIN),y)
-- 
2.11.0

[Buildroot] [PATCH 12/16] barebox: exclude from hash check except for latest version

[Arnout Vandecappelle]

Instead of excluding the hash specifically for the custom tarball and
custom version cases, exclude it always except in the one case where
we do have a hash: the latest version.

This simplifies the code a little because soon we will add hash checks
for git as well, so we also need an exclusion in that case.

It is not needed to exclude it twice for barebox and barebox-aux,
because they use the same source tarball and barebox-aux can only be
enabled if barebox is enabled. So simplify even further by pulling
the exclusion logic out of inner-barebox-package.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 boot/barebox/barebox.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/boot/barebox/barebox.mk b/boot/barebox/barebox.mk
index 7a110d2145..dc19224c76 100644
--- a/boot/barebox/barebox.mk
+++ b/boot/barebox/barebox.mk
@@ -22,7 +22,6 @@ ifeq ($$($(1)_VERSION),custom)
 $(1)_TARBALL = $$(call qstrip,$$(BR2_TARGET_BAREBOX_CUSTOM_TARBALL_LOCATION))
 $(1)_SITE = $$(patsubst %/,%,$$(dir $$($(1)_TARBALL)))
 $(1)_SOURCE = $$(notdir $$($(1)_TARBALL))
-BR_NO_CHECK_HASH_FOR += $$($(1)_SOURCE)
 else ifeq ($$(BR2_TARGET_BAREBOX_CUSTOM_GIT),y)
 $(1)_SITE = $$(call qstrip,$$(BR2_TARGET_BAREBOX_CUSTOM_GIT_REPO_URL))
 $(1)_SITE_METHOD = git
@@ -30,9 +29,6 @@ else
 # Handle stable official Barebox versions
 $(1)_SOURCE = barebox-$$($(1)_VERSION).tar.bz2
 $(1)_SITE = http://www.barebox.org/download
-ifeq ($$(BR2_TARGET_BAREBOX_CUSTOM_VERSION),y)
-BR_NO_CHECK_HASH_FOR += $$($(1)_SOURCE)
-endif
 endif
 
 $(1)_DEPENDENCIES = host-lzop
@@ -155,3 +151,7 @@ barebox-package=$(call inner-barebox-package,$(call UPPERCASE,$(pkgname)))
 
 include boot/barebox/barebox/barebox.mk
 include boot/barebox/barebox-aux/barebox-aux.mk
+
+ifeq ($(BR2_TARGET_BAREBOX)$(BR2_TARGET_BAREBOX_LATEST_VERSION),y)
+BR_NO_CHECK_HASH_FOR += $(BAREBOX_SOURCE)
+endif
-- 
2.11.0

[Buildroot] [PATCH 13/16] at91bootstrap3: exclude from hash when downloading from git

[Arnout Vandecappelle]

Since we will enable hash checks for git downloads soon, the hash check
for the custom git download should be disabled.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 boot/at91bootstrap3/at91bootstrap3.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/boot/at91bootstrap3/at91bootstrap3.mk b/boot/at91bootstrap3/at91bootstrap3.mk
index 32732f4758..685be5bbd7 100644
--- a/boot/at91bootstrap3/at91bootstrap3.mk
+++ b/boot/at91bootstrap3/at91bootstrap3.mk
@@ -9,6 +9,7 @@ AT91BOOTSTRAP3_VERSION = $(call qstrip,$(BR2_TARGET_AT91BOOTSTRAP3_VERSION))
 ifeq ($(BR2_TARGET_AT91BOOTSTRAP3_CUSTOM_GIT),y)
 AT91BOOTSTRAP3_SITE = $(call qstrip,$(BR2_TARGET_AT91BOOTSTRAP3_CUSTOM_REPO_URL))
 AT91BOOTSTRAP3_SITE_METHOD = git
+BR_NO_CHECK_HASH_FOR += $(AT91BOOTSTRAP3_SOURCE)
 else
 AT91BOOTSTRAP3_SITE = $(call github,linux4sam,at91bootstrap,$(AT91BOOTSTRAP3_VERSION))
 endif
-- 
2.11.0

[Buildroot] [PATCH 14/16] mxs-bootlets: exclude from hash when downloading from git

[Arnout Vandecappelle]

Since we will enable hash checks for git downloads soon, the hash check
for the custom git download should be disabled.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
Since mxs-bootlets is simpler, it is easier to do the exclusion
explicitly for these two cases, and not use the more generic approach
like for linux and uboot.
---
 boot/mxs-bootlets/mxs-bootlets.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/boot/mxs-bootlets/mxs-bootlets.mk b/boot/mxs-bootlets/mxs-bootlets.mk
index 17af6dc158..c7dd534ec0 100644
--- a/boot/mxs-bootlets/mxs-bootlets.mk
+++ b/boot/mxs-bootlets/mxs-bootlets.mk
@@ -13,6 +13,7 @@ else ifeq ($(BR2_TARGET_MXS_BOOTLETS_CUSTOM_GIT),y)
 MXS_BOOTLETS_SITE = $(BR2_TARGET_MXS_BOOTLETS_CUSTOM_GIT_URL)
 MXS_BOOTLETS_SITE_METHOD = git
 MXS_BOOTLETS_VERSION = $(call qstrip,$(BR2_TARGET_MXS_BOOTLETS_CUSTOM_GIT_VERSION))
+BR_NO_CHECK_HASH_FOR += $(MXS_BOOTLETS_SOURCE)
 else
 MXS_BOOTLETS_VERSION = 10.12.01
 MXS_BOOTLETS_SITE = http://download.ossystems.com.br/bsp/freescale/source
-- 
2.11.0

[Buildroot] [PATCH 15/16] arm-trusted-firmware: exclude from hash when downloading from git

[Arnout Vandecappelle]

Since we will enable hash checks for git downloads soon, the hash check
for the custom git download should be disabled.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
Since arm-trusted-firmware is simpler, it is easier to do the exclusion
explicitly for these two cases, and not use the more generic approach
like for linux and uboot.
---
 boot/arm-trusted-firmware/arm-trusted-firmware.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/boot/arm-trusted-firmware/arm-trusted-firmware.mk b/boot/arm-trusted-firmware/arm-trusted-firmware.mk
index 5acfab391f..b9fb3d84d3 100644
--- a/boot/arm-trusted-firmware/arm-trusted-firmware.mk
+++ b/boot/arm-trusted-firmware/arm-trusted-firmware.mk
@@ -19,6 +19,7 @@ BR_NO_CHECK_HASH_FOR += $(ARM_TRUSTED_FIRMWARE_SOURCE)
 else ifeq ($(BR2_TARGET_ARM_TRUSTED_FIRMWARE_CUSTOM_GIT),y)
 ARM_TRUSTED_FIRMWARE_SITE = $(call qstrip,$(BR2_TARGET_ARM_TRUSTED_FIRMWARE_CUSTOM_REPO_URL))
 ARM_TRUSTED_FIRMWARE_SITE_METHOD = git
+BR_NO_CHECK_HASH_FOR += $(ARM_TRUSTED_FIRMWARE_SOURCE)
 else
 ARM_TRUSTED_FIRMWARE_SITE = $(call github,ARM-software,arm-trusted-firmware,$(ARM_TRUSTED_FIRMWARE_VERSION))
 endif
-- 
2.11.0

[Buildroot] [PATCH 16/16] pkg-download: enable hash check for git downloads

[Arnout Vandecappelle]

For a while already, our git downloaded tarballs are reproducible, so
we can actually check the hash for them.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
Tested 'make pkg-source' for all git-downloaded packages (with and
without hash files).
---
 package/pkg-download.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index cfc550e77b..fbbc2d7391 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -77,6 +77,7 @@ define DOWNLOAD_GIT
 	$(EXTRA_ENV) $(DL_WRAPPER) -b git \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		$(if $($(PKG)_GIT_SUBMODULES),-r) \
+		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
 		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
@@ -225,7 +226,7 @@ define SOURCE_CHECK
 endef
 
 define DOWNLOAD_INNER
-	$(Q)$(if $(filter bzr cvs git hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
+	$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
 	if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
 		case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \
 			file) $(call $(3)_LOCALFILES,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
-- 
2.11.0

[Buildroot] [PATCH 00/16] Enable hash checking for git downloads

[Ricardo Martincoski]

Arnout,

----- Original Message -----
> From: "Arnout Vandecappelle"
> Sent: Monday, March 20, 2017 9:06:56 PM
> Subject: [Buildroot] [PATCH 00/16] Enable hash checking for git downloads
[snip]
> [PATCH 01/16] download/git: create GNU format tar files
> [PATCH 02/16] aer-inject: remove redundant _SITE_METHOD
> [PATCH 03/16] fmc: correct hash file
> [PATCH 04/16] linux-firmware: correct hash
> [PATCH 05/16] squashfs: correct hash
> [PATCH 06/16] ubus: add hash
> [PATCH 07/16] uhttpd: add hash
> [PATCH 08/16] vboot-utils: add hash
> [PATCH 09/16] linux: exclude from hash check except for latest
> [PATCH 10/16] linux-headers: rework hash exclusion
> [PATCH 11/16] uboot: exclude from hash check except for latest
> [PATCH 12/16] barebox: exclude from hash check except for latest
> [PATCH 13/16] at91bootstrap3: exclude from hash when downloading from
> [PATCH 14/16] mxs-bootlets: exclude from hash when downloading from
> [PATCH 15/16] arm-trusted-firmware: exclude from hash when
> [PATCH 16/16] pkg-download: enable hash check for git downloads

fmc-source works fine for me but host-vboot-utils-source does not.
Also arm-trusted-firmware-source falls back to sources.buildroot.net for me.
See logs below.

git clean -ffdx && make defconfig
make fmc-source
----->8-----
>>> fmc fsl-sdk-v2.0 Downloading
Doing shallow clone
Cloning into 'fmc-fsl-sdk-v2.0'...
remote: Counting objects: 69, done.
remote: Compressing objects: 100% (65/65), done.
remote: Total 69 (delta 9), reused 26 (delta 2)
Receiving objects: 100% (69/69), 276.00 KiB | 138.00 KiB/s, done.
Resolving deltas: 100% (9/9), done.
Note: checking out 'a079d2c844edd85dff85a317a63198e7988bcd09'.
[snip git detached HEAD warning]
warning: refname 'fsl-sdk-v2.0' is ambiguous.
fmc-fsl-sdk-v2.0.tar.gz: OK (sha256: a91e0c9b7c7f238634c64a755c05671f33f2acdb6ae2d09cad4d683b364ee8e4)
----->8-----

make host-vboot-utils-source
----->8-----
>>> host-vboot-utils bbdd62f9b030db7ad8eef789aaf58a7ff9a25656 Downloading
Doing full clone
Cloning into 'vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656'...
remote: Sending approximately 38.99 MiB ...
remote: Total 21863 (delta 14206), reused 21863 (delta 14206)
Receiving objects: 100% (21863/21863), 38.99 MiB | 1.55 MiB/s, done.
Resolving deltas: 100% (14206/14206), done.
warning: refname 'bbdd62f9b030db7ad8eef789aaf58a7ff9a25656' is ambiguous.
[snip git warning]
ERROR: vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz has wrong sha256 hash:
ERROR: expected: e119782a374655117e3d9a4e667b0056c76961c4593ba907f860d1310f6fbc2a
ERROR: got     : d95b64b1f1de4a3ffa5c2e446d7c8e92aa197aee10de24206b2ea2deb5a8b947
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
--2017-03-21 09:14:38--  http://sources.buildroot.net/vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 35645166 (34M) [application/x-gzip]
[snip long lines]
ERROR: vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz has wrong sha256 hash:
ERROR: expected: e119782a374655117e3d9a4e667b0056c76961c4593ba907f860d1310f6fbc2a
ERROR: got     : 2c71c3d04b9397ccb4b18202ca83d507f227e1e39c2bab6c9be2c3859155a52b
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
----->8-----

git clean -ffdx
make arm_juno_defconfig
make arm-trusted-firmware-source
----->8-----
Doing shallow clone
Cloning into 'arm-trusted-firmware-v1.2'...
remote: Counting objects: 645, done.
remote: Compressing objects: 100% (550/550), done.
remote: Total 645 (delta 240), reused 288 (delta 46), pack-reused 0
Receiving objects: 100% (645/645), 1.96 MiB | 827.00 KiB/s, done.
Resolving deltas: 100% (240/240), done.
Note: checking out 'd0c104e1e1ad0102f0f4c70997b7ee6e6fbbe273'.
[snip git warning]
warning: refname 'v1.2' is ambiguous.
ERROR: arm-trusted-firmware-v1.2.tar.gz has wrong sha256 hash:
ERROR: expected: cbdd9b770ec1ab4933fc7f9f520daea5a364bb4dc964820fb017a0cf8c7df556
ERROR: got     : 0eeba7a89028392a97fd64fc9052a36391af388ff716bd7c884cd50098a2f50c
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
--2017-03-21 09:04:41--  http://sources.buildroot.net/arm-trusted-firmware-v1.2.tar.gz
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1808700 (1,7M) [application/x-gzip]
Saving to: ?/tmp/git-hash/output/build/.arm-trusted-firmware-v1.2.tar.gz.clvxYW/output?
[snip long lines]
arm-trusted-firmware-v1.2.tar.gz: OK (sha256: cbdd9b770ec1ab4933fc7f9f520daea5a364bb4dc964820fb017a0cf8c7df556)
arm-trusted-firmware-v1.2.tar.gz: OK (md5: fac2c08bd74337fec2e14a98fc9f748f)
----->8-----

cat /etc/os-release | grep VERSION= ; git --version ; tar --version | grep tar
----->8-----
VERSION="14.04.5 LTS, Trusty Tahr"
git version 2.11.0
tar (GNU tar) 1.27.1
----->8-----

git log --oneline --decorate -17
----->8-----
4b9c7077a6 (HEAD) pkg-download: enable hash check for git downloads
9183c9d31e arm-trusted-firmware: exclude from hash when downloading from git
6db2b0ba07 mxs-bootlets: exclude from hash when downloading from git
69f9d1b489 at91bootstrap3: exclude from hash when downloading from git
804053ad18 barebox: exclude from hash check except for latest version
989d9b77f5 uboot: exclude from hash check except for latest version
95d5d580ea linux-headers: rework hash exclusion
524a3f8aed linux: exclude from hash check except for latest version
8c3f8dc348 vboot-utils: add hash
bfe808e92d uhttpd: add hash
885bce3fec ubus: add hash
331d44fae2 squashfs: correct hash
7c4b32dfb7 linux-firmware: correct hash
d8e8a374e6 fmc: correct hash file
1a6e356b9d aer-inject: remove redundant _SITE_METHOD
0501fe2808 download/git: create GNU format tar files
1a83dda003 (upstream/master) package/ghostscript: new package
----->8-----

Best regards,
Ricardo

[Buildroot] [PATCH 00/16] Enable hash checking for git downloads

[Arnout Vandecappelle]

 Hi Ricardo,

 Thanks for testing!

On 21-03-17 13:19, Ricardo Martincoski wrote:
[snip]
> make host-vboot-utils-source
> ----->8-----
>>>> host-vboot-utils bbdd62f9b030db7ad8eef789aaf58a7ff9a25656 Downloading
> Doing full clone
> Cloning into 'vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656'...
> remote: Sending approximately 38.99 MiB ...
> remote: Total 21863 (delta 14206), reused 21863 (delta 14206)
> Receiving objects: 100% (21863/21863), 38.99 MiB | 1.55 MiB/s, done.
> Resolving deltas: 100% (14206/14206), done.
> warning: refname 'bbdd62f9b030db7ad8eef789aaf58a7ff9a25656' is ambiguous.
> [snip git warning]
> ERROR: vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz has wrong sha256 hash:
> ERROR: expected: e119782a374655117e3d9a4e667b0056c76961c4593ba907f860d1310f6fbc2a
> ERROR: got     : d95b64b1f1de4a3ffa5c2e446d7c8e92aa197aee10de24206b2ea2deb5a8b947
> ERROR: Incomplete download, or man-in-the-middle (MITM) attack
> --2017-03-21 09:14:38--  http://sources.buildroot.net/vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz
> Resolving localhost (localhost)... 127.0.0.1
> Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
> Proxy request sent, awaiting response... 200 OK
> Length: 35645166 (34M) [application/x-gzip]
> [snip long lines]
> ERROR: vboot-utils-bbdd62f9b030db7ad8eef789aaf58a7ff9a25656.tar.gz has wrong sha256 hash:
> ERROR: expected: e119782a374655117e3d9a4e667b0056c76961c4593ba907f860d1310f6fbc2a
> ERROR: got     : 2c71c3d04b9397ccb4b18202ca83d507f227e1e39c2bab6c9be2c3859155a52b

 This is worrisome - both your tarball and the one from sources.buildroot.net
differ from the one I have, and are also different between them. Well, the one
on sources.buildroot.net can be explained: that tarball was created at a time
that our git helper was not reproducible yet, so it will have the wrong hash.

 Peter, could you do a refresh of all git-downloaded packages on
sources.buildroot.net? You can get them with

make printvars VARS="$(git grep -l 'git:\|METHOD = git' -- \*.mk | \
		sed 's%.*/\([^/]*\).mk%\1_SOURCE%g' | \
		tr 'a-z-\n' 'A-Z_ ')"


 And that probably also explains why my hash is wrong. I downloaded the
vboot-utils tarball on Jan 7, 2016. However, the git archive creation was
changed a couple of time and has only been stable since commit 4f87aa7db5ee of
Oct 25. So I generated the hash based on the wrong tarball...

 I'll have to recheck my patches that add hashes.

> ERROR: Incomplete download, or man-in-the-middle (MITM) attack
> ----->8-----
> 
> git clean -ffdx
> make arm_juno_defconfig
> make arm-trusted-firmware-source
> ----->8-----
> Doing shallow clone
> Cloning into 'arm-trusted-firmware-v1.2'...
> remote: Counting objects: 645, done.
> remote: Compressing objects: 100% (550/550), done.
> remote: Total 645 (delta 240), reused 288 (delta 46), pack-reused 0
> Receiving objects: 100% (645/645), 1.96 MiB | 827.00 KiB/s, done.
> Resolving deltas: 100% (240/240), done.
> Note: checking out 'd0c104e1e1ad0102f0f4c70997b7ee6e6fbbe273'.
> [snip git warning]
> warning: refname 'v1.2' is ambiguous.
> ERROR: arm-trusted-firmware-v1.2.tar.gz has wrong sha256 hash:
> ERROR: expected: cbdd9b770ec1ab4933fc7f9f520daea5a364bb4dc964820fb017a0cf8c7df556
> ERROR: got     : 0eeba7a89028392a97fd64fc9052a36391af388ff716bd7c884cd50098a2f50c

 Hm, this one is a little bit annoying. arm-trusted-firmware-v1.2.tar.gz
actually does get added to BR_NO_CHECK_HASH_FOR. However, version v1.2 is the
default version and it *does* have a hash. check-hash has the following code:

if [ ${nb_checks} -eq 0 ]; then
    case " ${BR_NO_CHECK_HASH_FOR} " in
    *" ${base} "*)
        # File explicitly has no hash
        exit 0
        ;;
    esac
    printf "ERROR: No hash found for %s\n" "${base}" >&2
    exit 3
fi

i.e. the hash check is only skipped when there is no hash in the hash file.

 What happens here is that the tarball that is generated by github is slightly
different from the tarball that we generate. So using the hash of the github
tarball to check the hash of our tarball is going to fail.

 Yann, I think the solution is to move the BR_NO_CHECK_HASH_FOR out of the
${nb_checks}. Actually move it all the way to the beginning and skip the check
entirely. Do you agree?  The commit log that introduced it (8d2f4e6246) offers
no hint as to why the check isn't skipped completely.

 Note that this could also happen for e.g. U-Boot when downloading from git and
setting the version to 2017.03. It's not very likely, however; AFAICS
arm-trusted-firmware is the only one which has a reasonable chance of ever
hitting this issue.


> ERROR: Incomplete download, or man-in-the-middle (MITM) attack
> --2017-03-21 09:04:41--  http://sources.buildroot.net/arm-trusted-firmware-v1.2.tar.gz
> Resolving localhost (localhost)... 127.0.0.1
> Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
> Proxy request sent, awaiting response... 200 OK
> Length: 1808700 (1,7M) [application/x-gzip]
> Saving to: ?/tmp/git-hash/output/build/.arm-trusted-firmware-v1.2.tar.gz.clvxYW/output?
> [snip long lines]
> arm-trusted-firmware-v1.2.tar.gz: OK (sha256: cbdd9b770ec1ab4933fc7f9f520daea5a364bb4dc964820fb017a0cf8c7df556)
> arm-trusted-firmware-v1.2.tar.gz: OK (md5: fac2c08bd74337fec2e14a98fc9f748f)

 Here it works because the tarball on sources.buildroot.net is the one from the
github helper.

 Regards,
 Arnout

> ----->8-----
> 
> cat /etc/os-release | grep VERSION= ; git --version ; tar --version | grep tar
> ----->8-----
> VERSION="14.04.5 LTS, Trusty Tahr"
> git version 2.11.0
> tar (GNU tar) 1.27.1
> ----->8-----
> 
> git log --oneline --decorate -17
> ----->8-----
> 4b9c7077a6 (HEAD) pkg-download: enable hash check for git downloads
> 9183c9d31e arm-trusted-firmware: exclude from hash when downloading from git
> 6db2b0ba07 mxs-bootlets: exclude from hash when downloading from git
> 69f9d1b489 at91bootstrap3: exclude from hash when downloading from git
> 804053ad18 barebox: exclude from hash check except for latest version
> 989d9b77f5 uboot: exclude from hash check except for latest version
> 95d5d580ea linux-headers: rework hash exclusion
> 524a3f8aed linux: exclude from hash check except for latest version
> 8c3f8dc348 vboot-utils: add hash
> bfe808e92d uhttpd: add hash
> 885bce3fec ubus: add hash
> 331d44fae2 squashfs: correct hash
> 7c4b32dfb7 linux-firmware: correct hash
> d8e8a374e6 fmc: correct hash file
> 1a6e356b9d aer-inject: remove redundant _SITE_METHOD
> 0501fe2808 download/git: create GNU format tar files
> 1a83dda003 (upstream/master) package/ghostscript: new package
> ----->8-----
> 
> Best regards,
> Ricardo
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 00/16] Enable hash checking for git downloads

[Thomas Petazzoni]

Hello,

On Tue, 21 Mar 2017 01:06:56 +0100, Arnout Vandecappelle
(Essensium/Mind) wrote:

> Patch 1 fixes the case when tar is compiled to generated Posix format
> tarballs.

So I've applied this one.

> Patch 2 is just a cleanup. This patch is optional.
> 
> Patch 3-5 fix incorrect hash files, which weren't noticed before
> because hashes weren't checked.
> 
> Patch 6-8 adds a hash for packages that actually have a hash file but
> set the hash type to 'none'. These patches are optional.

And all those ones.

> Patches 9-15 add exclusions for the different packages with CUSTOM_GIT
> configuration.

Still needs to review those one carefully.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 00/16] Enable hash checking for git downloads

[Thomas Petazzoni]

Hello,

On Tue, 21 Mar 2017 01:06:56 +0100, Arnout Vandecappelle
(Essensium/Mind) wrote:

> [PATCH 09/16] linux: exclude from hash check except for latest
> [PATCH 10/16] linux-headers: rework hash exclusion
> [PATCH 11/16] uboot: exclude from hash check except for latest
> [PATCH 12/16] barebox: exclude from hash check except for latest
> [PATCH 13/16] at91bootstrap3: exclude from hash when downloading from
> [PATCH 14/16] mxs-bootlets: exclude from hash when downloading from
> [PATCH 15/16] arm-trusted-firmware: exclude from hash when
> [PATCH 16/16] pkg-download: enable hash check for git downloads

All those patches applied. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com