[Buildroot] [PATCH 0/6 v3] toolchain: better handle hashes (branch yem/dl-hash-toolchains)

[Buildroot] [PATCH 0/6 v3] toolchain: better handle hashes (branch yem/dl-hash-toolchains)

[Yann E. MORIN]

Hello All!

This series makes it so we can download custom external toolchains, and
still have mandatory checks of hashes.

It also adds the long-awaited-for hashes for gcc and binutils.

Finally, it fixes a slightly annoying warning message when downloading
from a repository.


Changes v2 -> v3:
  - add hashes for gcc and binutils
  - fix a few more typoes

Changes v1 -> v2:
  - fix a few typoes


Regards,
Yann E. MORIN.


The following changes since commit ca735d8a0b86b7ad274319774df2bca0abfe4ee9:

  package/ipmiutil: do not build doc (2015-04-06 12:42:54 +0200)

are available in the git repository at:

  git://git.busybox.net/~ymorin/git/buildroot yem/dl-hash-toolchains

for you to fetch changes up to 21c6a4cd9d79b9279ba50426a2d0baaa1bc3f0b0:

  support/download: restore mandatory check of hashes (2015-04-06 14:11:45 +0200)

----------------------------------------------------------------
Yann E. MORIN (6):
      support/download: do not warn about missing hash file for repositories
      package/gcc: add hashes
      package/binutils: add hashes
      support/download: add possibility to not fail on missing hash
      toolchain/external: ignore missing hash for custom downloaded toolchain
      support/download: restore mandatory check of hashes

 package/binutils/binutils.hash                     |  7 +++++++
 package/gcc/gcc-final/gcc-final.hash               |  1 +
 package/gcc/gcc-initial/gcc-initial.hash           |  1 +
 package/gcc/gcc.hash                               |  7 +++++++
 package/pkg-download.mk                            |  4 ++++
 support/download/check-hash                        | 12 ++++++++++--
 toolchain/toolchain-external/toolchain-external.mk |  2 ++
 7 files changed, 32 insertions(+), 2 deletions(-)
 create mode 100644 package/binutils/binutils.hash
 create mode 120000 package/gcc/gcc-final/gcc-final.hash
 create mode 120000 package/gcc/gcc-initial/gcc-initial.hash
 create mode 100644 package/gcc/gcc.hash

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/6 v3] support/download: do not warn about missing hash file for repositories

[Yann E. MORIN]

When downloading from a repository, we explicitly pass no hash file,
because we can't check hashes in that case.

However, we're still printing a message that there is a missign hash
file.

Beside being a bit annoying (since we can't do anything about it), it
may also be wrong, especially for packages for which we support multiple
versions, with some being downloaded via a git clone and others as
tarballs.

Just print no warning when the path to the hash file is empty.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>

---
Changes v1 -> v2;
  - fix typoes in commit log
---
 support/download/check-hash | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/support/download/check-hash b/support/download/check-hash
index 67c1780..d37f1cd 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -31,8 +31,12 @@ h_file="${1}"
 file="${2}"
 base="${3}"
 
+# Bail early if no hash to check
+if [ -z "${h_file}" ]; then
+    exit 0
+fi
 # Does the hash-file exist?
-if [ -z "${h_file}" -o ! -f "${h_file}" ]; then
+if [ ! -f "${h_file}" ]; then
     printf "WARNING: no hash file for %s\n" "${base}" >&2
     exit 0
 fi
-- 
1.9.1

[Buildroot] [PATCH 1/6 v3] support/download: do not warn about missing hash file for repositories

[Arnout Vandecappelle]

On 06/04/15 14:13, Yann E. MORIN wrote:
> When downloading from a repository, we explicitly pass no hash file,
> because we can't check hashes in that case.
> 
> However, we're still printing a message that there is a missign hash
> file.
> 
> Beside being a bit annoying (since we can't do anything about it), it
> may also be wrong, especially for packages for which we support multiple
> versions, with some being downloaded via a git clone and others as
> tarballs.
> 
> Just print no warning when the path to the hash file is empty.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>


 Regards,
 Arnout

> 
> ---
> Changes v1 -> v2;
>   - fix typoes in commit log
> ---
>  support/download/check-hash | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/support/download/check-hash b/support/download/check-hash
> index 67c1780..d37f1cd 100755
> --- a/support/download/check-hash
> +++ b/support/download/check-hash
> @@ -31,8 +31,12 @@ h_file="${1}"
>  file="${2}"
>  base="${3}"
>  
> +# Bail early if no hash to check
> +if [ -z "${h_file}" ]; then
> +    exit 0
> +fi
>  # Does the hash-file exist?
> -if [ -z "${h_file}" -o ! -f "${h_file}" ]; then
> +if [ ! -f "${h_file}" ]; then
>      printf "WARNING: no hash file for %s\n" "${base}" >&2
>      exit 0
>  fi
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 1/6 v3] support/download: do not warn about missing hash file for repositories

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Mon,  6 Apr 2015 14:13:06 +0200, Yann E. MORIN wrote:
> When downloading from a repository, we explicitly pass no hash file,
> because we can't check hashes in that case.
> 
> However, we're still printing a message that there is a missign hash
> file.
> 
> Beside being a bit annoying (since we can't do anything about it), it
> may also be wrong, especially for packages for which we support multiple
> versions, with some being downloaded via a git clone and others as
> tarballs.
> 
> Just print no warning when the path to the hash file is empty.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>

Applied, thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 2/6 v3] package/gcc: add hashes

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
---
 package/gcc/gcc-final/gcc-final.hash     | 1 +
 package/gcc/gcc-initial/gcc-initial.hash | 1 +
 package/gcc/gcc.hash                     | 7 +++++++
 3 files changed, 9 insertions(+)
 create mode 120000 package/gcc/gcc-final/gcc-final.hash
 create mode 120000 package/gcc/gcc-initial/gcc-initial.hash
 create mode 100644 package/gcc/gcc.hash

diff --git a/package/gcc/gcc-final/gcc-final.hash b/package/gcc/gcc-final/gcc-final.hash
new file mode 120000
index 0000000..7ac9361
--- /dev/null
+++ b/package/gcc/gcc-final/gcc-final.hash
@@ -0,0 +1 @@
+../gcc.hash
\ No newline at end of file
diff --git a/package/gcc/gcc-initial/gcc-initial.hash b/package/gcc/gcc-initial/gcc-initial.hash
new file mode 120000
index 0000000..7ac9361
--- /dev/null
+++ b/package/gcc/gcc-initial/gcc-initial.hash
@@ -0,0 +1 @@
+../gcc.hash
\ No newline at end of file
diff --git a/package/gcc/gcc.hash b/package/gcc/gcc.hash
new file mode 100644
index 0000000..d9a04fc
--- /dev/null
+++ b/package/gcc/gcc.hash
@@ -0,0 +1,7 @@
+# Locally computed (upstream only has sigs):
+sha256  eef3f0456db8c3d992cbb51d5d32558190bc14f3bc19383dd93acc27acc6befc  gcc-4.5.4.tar.bz2
+sha256  92e61c6dc3a0a449e62d72a38185fda550168a86702dea07125ebd3ec3996282  gcc-4.7.4.tar.bz2
+sha256  4a80aa23798b8e9b5793494b8c976b39b8d9aa2e53cd5ed5534aff662a7f8695  gcc-4.8.4.tar.bz2
+sha256  2020c98295856aa13fda0f2f3a4794490757fc24bcca918d52cc8b4917b972dd  gcc-4.9.2.tar.bz2
+# No hash for the ARC variant, comes from the github-helper:
+none    xxx                                                               gcc-arc-2014.12.tar.gz
-- 
1.9.1

[Buildroot] [PATCH 3/6 v3] package/binutils: add hashes

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
---
 package/binutils/binutils.hash | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 package/binutils/binutils.hash

diff --git a/package/binutils/binutils.hash b/package/binutils/binutils.hash
new file mode 100644
index 0000000..4f99514
--- /dev/null
+++ b/package/binutils/binutils.hash
@@ -0,0 +1,7 @@
+# Locally computed:
+sha256 6c7af8ed1c8cf9b4b9d6e6fe09a3e1d3d479fe63984ba8b9b26bf356b6313ca9  binutils-2.22.tar.bz2
+sha256 fe914e56fed7a9ec2eb45274b1f2e14b0d8b4f41906a5194eac6883cfe5c1097  binutils-2.23.2.tar.bz2
+sha256 e5e8c5be9664e7f7f96e0d09919110ab5ad597794f5b1809871177a0f0f14137  binutils-2.24.tar.bz2
+sha256 22defc65cfa3ef2a3395faaea75d6331c6e62ea5dfacfed3e2ec17b08c882923  binutils-2.25.tar.bz2
+# No hash for the ARC variant, comes from the github-helper:
+none   xxx                                                               binutils-arc-2014.12.tar.gz
-- 
1.9.1

[Buildroot] [PATCH 4/6 v3] support/download: add possibility to not fail on missing hash

[Yann E. MORIN]

In very constrained cases, it might be needed to not fail if a hash is
missing. This is notably the case for custom external toolchains to be
downloaded, because we do have a .hash file for external toolchains,
but we oviously can not have hashes for all existing cutom toolchains
(he, "custom"!).

So, add a way to avoid failing in that case.

We use a magic value depending on the file to be downloaded, instead of
yes/no, to avoid any accidental use. We also do not document it, because
we do not want users to use it, and even more, to rely on it, since we
still have hopes to get rid of that in the future.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>

---
Changes v1 -> v2:
  - fix typoes in commit log
---
 package/pkg-download.mk     | 4 ++++
 support/download/check-hash | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index e274712..1e080e5 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -60,6 +60,10 @@ domainseparator = $(if $(1),$(1),/)
 # github(user,package,version): returns site of GitHub repository
 github = https://github.com/$(1)/$(2)/archive/$(3)
 
+# Compute magic value to ignore a missing hash
+# $1: filename, without path components
+ignore-missing-hash-magic = $(shell sha1sum <<<"$(1)" |cut -d ' ' -f 1)
+
 ################################################################################
 # The DOWNLOAD_* helpers are in charge of getting a working copy
 # of the source repository for their corresponding SCM,
diff --git a/support/download/check-hash b/support/download/check-hash
index d37f1cd..a091c61 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -99,6 +99,10 @@ while read t h f; do
 done <"${h_file}"
 
 if [ ${nb_checks} -eq 0 ]; then
+    if [ "${BR_MISSING_HASH_OK}" = "$(sha1sum <<<"${base}" |cut -d ' ' -f 1)" ]; then
+        printf "WARNING: ignored missing hash for %s\n" "${base}" >&2
+        exit 0
+    fi
     printf "ERROR: No hash found for %s\n" "${base}" >&2
     exit 0
 fi
-- 
1.9.1

[Buildroot] [PATCH 4/6 v3] support/download: add possibility to not fail on missing hash

[Arnout Vandecappelle]

On 06/04/15 14:13, Yann E. MORIN wrote:
> In very constrained cases, it might be needed to not fail if a hash is
> missing. This is notably the case for custom external toolchains to be
> downloaded, because we do have a .hash file for external toolchains,
> but we oviously can not have hashes for all existing cutom toolchains
> (he, "custom"!).
> 
> So, add a way to avoid failing in that case.
> 
> We use a magic value depending on the file to be downloaded, instead of
> yes/no, to avoid any accidental use. We also do not document it, because
> we do not want users to use it, and even more, to rely on it, since we
> still have hopes to get rid of that in the future.

 I don't think that complexity is really warranted. Something like
BR_MISSING_HASH=OK should be enough.

 I do agree with not documenting it, however.

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
> 
> ---
> Changes v1 -> v2:
>   - fix typoes in commit log
> ---
>  package/pkg-download.mk     | 4 ++++
>  support/download/check-hash | 4 ++++
>  2 files changed, 8 insertions(+)
> 
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index e274712..1e080e5 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -60,6 +60,10 @@ domainseparator = $(if $(1),$(1),/)
>  # github(user,package,version): returns site of GitHub repository
>  github = https://github.com/$(1)/$(2)/archive/$(3)
>  
> +# Compute magic value to ignore a missing hash
> +# $1: filename, without path components
> +ignore-missing-hash-magic = $(shell sha1sum <<<"$(1)" |cut -d ' ' -f 1)
> +
>  ################################################################################
>  # The DOWNLOAD_* helpers are in charge of getting a working copy
>  # of the source repository for their corresponding SCM,
> diff --git a/support/download/check-hash b/support/download/check-hash
> index d37f1cd..a091c61 100755
> --- a/support/download/check-hash
> +++ b/support/download/check-hash
> @@ -99,6 +99,10 @@ while read t h f; do
>  done <"${h_file}"
>  
>  if [ ${nb_checks} -eq 0 ]; then
> +    if [ "${BR_MISSING_HASH_OK}" = "$(sha1sum <<<"${base}" |cut -d ' ' -f 1)" ]; then
> +        printf "WARNING: ignored missing hash for %s\n" "${base}" >&2

 There shouldn't be a warning in this case either, right? Neither end users nor
buildroot developers can do anything about it, so what would be the point...

 I'd also make the branching more explicit:

if [ ${nb_checks} -eq 0 ]; then
    if [ "${BR_MISSING_HASH}" = "OK" ]; then
        exit 0
    else
        printf "ERROR: No hash found for %s\n" "${base}" >&2
        exit 0
    fi
fi


 Regards,
 Arnout

> +        exit 0
> +    fi
>      printf "ERROR: No hash found for %s\n" "${base}" >&2
>      exit 0
>  fi
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 5/6 v3] toolchain/external: ignore missing hash for custom downloaded toolchain

[Yann E. MORIN]

We will *always* be missing a hash file for custom external toolchains
that are downloaded.

So, just ignore that failure.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>

---
Changes v1 -> v2:
  - fix typoes in title
---
 toolchain/toolchain-external/toolchain-external.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
index c0429bb..3659511 100644
--- a/toolchain/toolchain-external/toolchain-external.mk
+++ b/toolchain/toolchain-external/toolchain-external.mk
@@ -400,6 +400,8 @@ else
 # Custom toolchain
 TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
+# Magic value to tell the download helper to avoid failling on missing hash
+export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))
 endif
 
 # In fact, we don't need to download the toolchain, since it is already
-- 
1.9.1

[Buildroot] [PATCH 5/6 v3] toolchain/external: ignore missing hash for custom downloaded toolchain

[Arnout Vandecappelle]

On 06/04/15 14:13, Yann E. MORIN wrote:
> We will *always* be missing a hash file for custom external toolchains
> that are downloaded.
> 
> So, just ignore that failure.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
> 
> ---
> Changes v1 -> v2:
>   - fix typoes in title
> ---
>  toolchain/toolchain-external/toolchain-external.mk | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
> index c0429bb..3659511 100644
> --- a/toolchain/toolchain-external/toolchain-external.mk
> +++ b/toolchain/toolchain-external/toolchain-external.mk
> @@ -400,6 +400,8 @@ else
>  # Custom toolchain
>  TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
>  TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
> +# Magic value to tell the download helper to avoid failling on missing hash
> +export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))

 Urgh - this disables the hash check for all packages, not just the toolchain.
So NACK for exporting it.

 But I see that it's not so simple to get it into the environment of the
download step...


 Perhaps instead we should add a global variable with the files that should not
be hash-checked:

NOHASH_FILES += $(TOOLCHAIN_EXTERNAL_SOURCE)

and in pkg-download.mk

hasharg = $(if $(filter-out $(TOOLCHAIN_EXTERNAL_SOURCE),$(1)),
	-H $(PKGDIR)/$($(PKG)_RAWNAME).hash)

define DOWNLOAD_WGET
        $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
                -o $(DL_DIR)/$(2) \
                $(call hasharg,$(2)) \
                $(QUIET) \
                -- \
                '$(call qstrip,$(1))'
endef

(all completely untested, obviously :-)


 Regards,
 Arnout

>  endif
>  
>  # In fact, we don't need to download the toolchain, since it is already
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 5/6 v3] toolchain/external: ignore missing hash for custom downloaded toolchain

[Yann E. MORIN]

Arnout, All,

On 2015-04-06 23:03 +0200, Arnout Vandecappelle spake thusly:
> On 06/04/15 14:13, Yann E. MORIN wrote:
> > We will *always* be missing a hash file for custom external toolchains
> > that are downloaded.
> > 
> > So, just ignore that failure.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
> > 
> > ---
> > Changes v1 -> v2:
> >   - fix typoes in title
> > ---
> >  toolchain/toolchain-external/toolchain-external.mk | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
> > index c0429bb..3659511 100644
> > --- a/toolchain/toolchain-external/toolchain-external.mk
> > +++ b/toolchain/toolchain-external/toolchain-external.mk
> > @@ -400,6 +400,8 @@ else
> >  # Custom toolchain
> >  TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
> >  TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
> > +# Magic value to tell the download helper to avoid failling on missing hash
> > +export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))
> 
>  Urgh - this disables the hash check for all packages, not just the toolchain.

No, that disable the check only for that one file, because we're not
chcking whether it is set, but whether it is set to the correct value
(i.e. the sha1 of the filename).

> So NACK for exporting it.

OK, this is indeed fishy.

>  But I see that it's not so simple to get it into the environment of the
> download step...
> 
> 
>  Perhaps instead we should add a global variable with the files that should not
> be hash-checked:
> 
> NOHASH_FILES += $(TOOLCHAIN_EXTERNAL_SOURCE)
> 
> and in pkg-download.mk
> 
> hasharg = $(if $(filter-out $(TOOLCHAIN_EXTERNAL_SOURCE),$(1)),

You probably meant s/TOOLCHAIN_EXTERNAL_SOURCE/NOHASH_FILES/ ?

> 	-H $(PKGDIR)/$($(PKG)_RAWNAME).hash)
> 
> define DOWNLOAD_WGET
>         $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
>                 -o $(DL_DIR)/$(2) \
>                 $(call hasharg,$(2)) \
>                 $(QUIET) \
>                 -- \
>                 '$(call qstrip,$(1))'
> endef
> 
> (all completely untested, obviously :-)

Well, that sounds a bit better than my proposal.

Still, I prefer:

  - we offload the check in the dl-wrapper so we can do the check in a
    single place, rather than re-add extra code in each download macros
    (the dl-wrapper was added because the Makefile macros were too
    complex to handle);

  - we pass obscur values (like a sha1), rather than the filenames, to
    make it even less easy to use.

I'll respin a series taking into account your comments. Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 5/6 v3] toolchain/external: ignore missing hash for custom downloaded toolchain

[Arnout Vandecappelle]

On 06/04/15 23:20, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2015-04-06 23:03 +0200, Arnout Vandecappelle spake thusly:
>> On 06/04/15 14:13, Yann E. MORIN wrote:
>>> We will *always* be missing a hash file for custom external toolchains
>>> that are downloaded.
>>>
>>> So, just ignore that failure.
>>>
>>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>>> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
>>> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
>>>
>>> ---
>>> Changes v1 -> v2:
>>>   - fix typoes in title
>>> ---
>>>  toolchain/toolchain-external/toolchain-external.mk | 2 ++
>>>  1 file changed, 2 insertions(+)
>>>
>>> diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
>>> index c0429bb..3659511 100644
>>> --- a/toolchain/toolchain-external/toolchain-external.mk
>>> +++ b/toolchain/toolchain-external/toolchain-external.mk
>>> @@ -400,6 +400,8 @@ else
>>>  # Custom toolchain
>>>  TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
>>>  TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
>>> +# Magic value to tell the download helper to avoid failling on missing hash
>>> +export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))
>>
>>  Urgh - this disables the hash check for all packages, not just the toolchain.
> 
> No, that disable the check only for that one file, because we're not
> chcking whether it is set, but whether it is set to the correct value
> (i.e. the sha1 of the filename).

 Ah of course, that's probably why you introduced that.

 But it does mean that only a single file can ever be bypassed...

> 
>> So NACK for exporting it.
> 
> OK, this is indeed fishy.
> 
>>  But I see that it's not so simple to get it into the environment of the
>> download step...
>>
>>
>>  Perhaps instead we should add a global variable with the files that should not
>> be hash-checked:
>>
>> NOHASH_FILES += $(TOOLCHAIN_EXTERNAL_SOURCE)
>>
>> and in pkg-download.mk
>>
>> hasharg = $(if $(filter-out $(TOOLCHAIN_EXTERNAL_SOURCE),$(1)),
> 
> You probably meant s/TOOLCHAIN_EXTERNAL_SOURCE/NOHASH_FILES/ ?

 Untested and all :-)

> 
>> 	-H $(PKGDIR)/$($(PKG)_RAWNAME).hash)
>>
>> define DOWNLOAD_WGET
>>         $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
>>                 -o $(DL_DIR)/$(2) \
>>                 $(call hasharg,$(2)) \
>>                 $(QUIET) \
>>                 -- \
>>                 '$(call qstrip,$(1))'
>> endef
>>
>> (all completely untested, obviously :-)
> 
> Well, that sounds a bit better than my proposal.
> 
> Still, I prefer:
> 
>   - we offload the check in the dl-wrapper so we can do the check in a
>     single place, rather than re-add extra code in each download macros
>     (the dl-wrapper was added because the Makefile macros were too
>     complex to handle);

 Ack that, though I don't immediately see a way to do that.

> 
>   - we pass obscur values (like a sha1), rather than the filenames, to
>     make it even less easy to use.

 I really don't see the point of that... If it's not documented, you have to
look at the source code to find out which variable to set. And when you look at
the source code, you immediately see how this obscure value is calculated so it
makes no real difference.


 Regards,
 Arnout

> 
> I'll respin a series taking into account your comments. Thanks! :-)
> 
> Regards,
> Yann E. MORIN.
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 5/6 v3] toolchain/external: ignore missing hash for custom downloaded toolchain

[Yann E. MORIN]

Arnout, All,

On 2015-04-07 01:24 +0200, Arnout Vandecappelle spake thusly:
> On 06/04/15 23:20, Yann E. MORIN wrote:
> > On 2015-04-06 23:03 +0200, Arnout Vandecappelle spake thusly:
> >> On 06/04/15 14:13, Yann E. MORIN wrote:
[--SNIP--]
> >>> +# Magic value to tell the download helper to avoid failling on missing hash
> >>> +export BR_MISSING_HASH_OK := $(call ignore-missing-hash-magic,$(TOOLCHAIN_EXTERNAL_SOURCE))
> >>  Urgh - this disables the hash check for all packages, not just the toolchain.
> > 
> > No, that disable the check only for that one file, because we're not
> > chcking whether it is set, but whether it is set to the correct value
> > (i.e. the sha1 of the filename).
> 
>  Ah of course, that's probably why you introduced that.
> 
>  But it does mean that only a single file can ever be bypassed...

Right.

[--SNIP--]
> >> hasharg = $(if $(filter-out $(TOOLCHAIN_EXTERNAL_SOURCE),$(1)),
> > You probably meant s/TOOLCHAIN_EXTERNAL_SOURCE/NOHASH_FILES/ ?
>  Untested and all :-)

Hehe! :-)

> >> 	-H $(PKGDIR)/$($(PKG)_RAWNAME).hash)
> >>
> >> define DOWNLOAD_WGET
> >>         $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
> >>                 -o $(DL_DIR)/$(2) \
> >>                 $(call hasharg,$(2)) \
> >>                 $(QUIET) \
> >>                 -- \
> >>                 '$(call qstrip,$(1))'
> >> endef
> >>
> >> (all completely untested, obviously :-)
> > 
> > Well, that sounds a bit better than my proposal.
> > 
> > Still, I prefer:
> > 
> >   - we offload the check in the dl-wrapper so we can do the check in a
> >     single place, rather than re-add extra code in each download macros
> >     (the dl-wrapper was added because the Makefile macros were too
> >     complex to handle);
>  Ack that, though I don't immediately see a way to do that.

Don;t worry, I'll find a way... ;-]

> >   - we pass obscur values (like a sha1), rather than the filenames, to
> >     make it even less easy to use.
> 
>  I really don't see the point of that... If it's not documented, you have to
> look at the source code to find out which variable to set. And when you look at
> the source code, you immediately see how this obscure value is calculated so it
> makes no real difference.

OK, makes sense.

I will rework the series accordingly. Thanks for the suggestions!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 6/6 v3] support/download: restore mandatory check of hashes

[Yann E. MORIN]

Now that custom external toolchains to be downloaded proprely instruct
to not fail on a missing hash, restore the mandatory hash check for
everything else.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
---
 support/download/check-hash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/download/check-hash b/support/download/check-hash
index a091c61..96d12a5 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -104,5 +104,5 @@ if [ ${nb_checks} -eq 0 ]; then
         exit 0
     fi
     printf "ERROR: No hash found for %s\n" "${base}" >&2
-    exit 0
+    exit 3
 fi
-- 
1.9.1

[Buildroot] [PATCH 6/6 v3] support/download: restore mandatory check of hashes

[Arnout Vandecappelle]

On 06/04/15 14:13, Yann E. MORIN wrote:
> Now that custom external toolchains to be downloaded proprely instruct
> to not fail on a missing hash, restore the mandatory hash check for
> everything else.

 There are also a couple of uclibc versions without hash.

 Otherwise looks good :-)

 Regards,
 Arnout

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
> ---
>  support/download/check-hash | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/support/download/check-hash b/support/download/check-hash
> index a091c61..96d12a5 100755
> --- a/support/download/check-hash
> +++ b/support/download/check-hash
> @@ -104,5 +104,5 @@ if [ ${nb_checks} -eq 0 ]; then
>          exit 0
>      fi
>      printf "ERROR: No hash found for %s\n" "${base}" >&2
> -    exit 0
> +    exit 3
>  fi
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 6/6 v3] support/download: restore mandatory check of hashes

[Yann E. MORIN]

Arnout, All,

On 2015-04-06 23:05 +0200, Arnout Vandecappelle spake thusly:
> On 06/04/15 14:13, Yann E. MORIN wrote:
> > Now that custom external toolchains to be downloaded proprely instruct
> > to not fail on a missing hash, restore the mandatory hash check for
> > everything else.
> 
>  There are also a couple of uclibc versions without hash.

Then they should be added (unless they are git snapshots or the
likes...) ;-)

I'll see which are missing and add them; thanks for noticing.

Regards,
Yann E. MORIN.

>  Otherwise looks good :-)
> 
>  Regards,
>  Arnout
> 
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
> > ---
> >  support/download/check-hash | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/support/download/check-hash b/support/download/check-hash
> > index a091c61..96d12a5 100755
> > --- a/support/download/check-hash
> > +++ b/support/download/check-hash
> > @@ -104,5 +104,5 @@ if [ ${nb_checks} -eq 0 ]; then
> >          exit 0
> >      fi
> >      printf "ERROR: No hash found for %s\n" "${base}" >&2
> > -    exit 0
> > +    exit 3
> >  fi
> > 
> 
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'