[Buildroot] [PATCH 0/3] support/download: be more aggressive on missing hashes (branch yem/dl-hash)

[Buildroot] [PATCH 0/3] support/download: be more aggressive on missing hashes (branch yem/dl-hash)

[Yann E. MORIN]

Hello All!

This series makes hashes mandatory when a .hash file exists.

Currently, we treat missing hashes as a mere warning. Unfortunately,
that often goes un-noticed by most users, and thus we get a lot of
package bumps that are missing the new hashes corresponding to the new
version.

We now make that a hard error, so users really notice something is
wrong.

Of course, if no .hash file exists, the behaviour is as yet unchanged.

Regards,
Yann E. MORIN.


The following changes since commit 2d05afa42792193fa392c9f5417e8effc73d1e38:

  cups: deprecate package due to security issues (2015-03-16 22:26:30 +0100)

are available in the git repository at:

  git://git.busybox.net/~ymorin/git/buildroot yem/dl-hash

for you to fetch changes up to e443dfcec4c084f823c718aa4c7740eddb3b628e:

  support/download: always fail when there's no hash (2015-03-16 23:33:48 +0100)

----------------------------------------------------------------
Yann E. MORIN (3):
      support/download: return different exit codes for different failures
      support/download: properly catch missing hashes
      support/download: always fail when there's no hash

 support/download/check-hash | 15 +++++++++------
 support/download/dl-wrapper | 11 ++++++++++-
 2 files changed, 19 insertions(+), 7 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/3] support/download: return different exit codes for different failures

[Yann E. MORIN]

Return different exit codes depending on the error that occured:

  0: no error (hash file missing, or all hashes match)
  1: hash file exists, but at least one hash in error
  3: hash file exists, but no hash for file to check

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 support/download/check-hash | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/support/download/check-hash b/support/download/check-hash
index 4c07274..4cc62f3 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -9,6 +9,13 @@ set -e
 #   $3: the final basename of the file, to which it will be ultimately
 #       saved as, to be able to match it to the corresponding hashes
 #       in the .hash file
+#
+# Exits with:
+#   0: the hash file exists and the file to check matches all its hashes,
+#      or the hash file does not exist
+#   1: the hash file exists and the file to check does not match at least
+#      of its hashes
+#   2: the hash file exists and there was no hash to check the fiel against
 
 while getopts :q OPT; do
     case "${OPT}" in
@@ -83,7 +90,7 @@ done <"${h_file}"
 if [ ${nb_checks} -eq 0 ]; then
     if [ -n "${BR2_ENFORCE_CHECK_HASH}" ]; then
         printf "ERROR: No hash found for %s\n" "${base}" >&2
-        exit 1
+        exit 2
     else
         printf "WARNING: No hash found for %s\n" "${base}" >&2
     fi
-- 
1.9.1

[Buildroot] [PATCH 2/3] support/download: properly catch missing hashes

[Yann E. MORIN]

When checking hashes reports no hash for a file, and this is treated as
an error (now: because BR2_ENFORCE_CHECK_HASH is set; later: becasue
that will be the new and only behaviour), exit promptly in error.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 support/download/dl-wrapper | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 3b30840..d93a159 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -48,10 +48,19 @@ main() {
         error "no hash-file specified, use -H\n"
     fi
 
-    # If the output file already exists, do not download it again
+    # If the output file already exists and:
+    # - there's no .hash file: do not download it again and exit promptly
+    # - matches all its hashes: do not download it again and exit promptly
+    # - fails at least one of its hashes: force a re-download
+    # - there's no hash (but a .hash file): consider it a hard error
     if [ -e "${output}" ]; then
         if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
             exit 0
+        elif [ ${?} -eq 2 ]; then
+            # Do not remove the file, otherwise it might get re-downloaded
+            # from a later location (i.e. primary -> upstream -> mirror).
+            # Do not print a message, check-hash already did
+            exit 2
         fi
         rm -f "${output}"
         warn "Re-downloading '%s'...\n" "${output##*/}"
-- 
1.9.1

[Buildroot] [PATCH 3/3] support/download: always fail when there's no hash

[Yann E. MORIN]

At the time we introduced hashes, we did not want to be too harsh in the
beginning, and give people some time to adapt and accept to hashes. So
we so far only whined^Wwarned about a missing hash file.

Some time has passed now, and people are still missing updating hashes
when bumping packages.

Let's make that warning a little bit more annoying...

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 support/download/check-hash | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/support/download/check-hash b/support/download/check-hash
index 4cc62f3..7971c38 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -88,10 +88,6 @@ while read t h f; do
 done <"${h_file}"
 
 if [ ${nb_checks} -eq 0 ]; then
-    if [ -n "${BR2_ENFORCE_CHECK_HASH}" ]; then
-        printf "ERROR: No hash found for %s\n" "${base}" >&2
-        exit 2
-    else
-        printf "WARNING: No hash found for %s\n" "${base}" >&2
-    fi
+    printf "ERROR: No hash found for %s\n" "${base}" >&2
+    exit 2
 fi
-- 
1.9.1

[Buildroot] [PATCH 1/3] support/download: return different exit codes for different failures

[Samuel Martin]

Yann,

Just few typos ;)

On Mon, Mar 16, 2015 at 11:41 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Return different exit codes depending on the error that occured:
>
>   0: no error (hash file missing, or all hashes match)
>   1: hash file exists, but at least one hash in error
>   3: hash file exists, but no hash for file to check
s/3/2/

>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  support/download/check-hash | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/support/download/check-hash b/support/download/check-hash
> index 4c07274..4cc62f3 100755
> --- a/support/download/check-hash
> +++ b/support/download/check-hash
> @@ -9,6 +9,13 @@ set -e
>  #   $3: the final basename of the file, to which it will be ultimately
>  #       saved as, to be able to match it to the corresponding hashes
>  #       in the .hash file
> +#
> +# Exits with:
> +#   0: the hash file exists and the file to check matches all its hashes,
> +#      or the hash file does not exist
> +#   1: the hash file exists and the file to check does not match at least
> +#      of its hashes
> +#   2: the hash file exists and there was no hash to check the fiel against
s/fiel/file/

>
>  while getopts :q OPT; do
>      case "${OPT}" in
> @@ -83,7 +90,7 @@ done <"${h_file}"
>  if [ ${nb_checks} -eq 0 ]; then
>      if [ -n "${BR2_ENFORCE_CHECK_HASH}" ]; then
>          printf "ERROR: No hash found for %s\n" "${base}" >&2
> -        exit 1
> +        exit 2
>      else
>          printf "WARNING: No hash found for %s\n" "${base}" >&2
>      fi
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot


Regards,

-- 
Samuel

[Buildroot] [PATCH 3/3] support/download: always fail when there's no hash

[Samuel Martin]

On Mon, Mar 16, 2015 at 11:41 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> At the time we introduced hashes, we did not want to be too harsh in the
> beginning, and give people some time to adapt and accept to hashes. So
> we so far only whined^Wwarned about a missing hash file.
>
> Some time has passed now, and people are still missing updating hashes
> when bumping packages.
>
> Let's make that warning a little bit more annoying...
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

> ---
>  support/download/check-hash | 8 ++------
>  1 file changed, 2 insertions(+), 6 deletions(-)
>
> diff --git a/support/download/check-hash b/support/download/check-hash
> index 4cc62f3..7971c38 100755
> --- a/support/download/check-hash
> +++ b/support/download/check-hash
> @@ -88,10 +88,6 @@ while read t h f; do
>  done <"${h_file}"
>
>  if [ ${nb_checks} -eq 0 ]; then
> -    if [ -n "${BR2_ENFORCE_CHECK_HASH}" ]; then
> -        printf "ERROR: No hash found for %s\n" "${base}" >&2
> -        exit 2
> -    else
> -        printf "WARNING: No hash found for %s\n" "${base}" >&2
> -    fi
> +    printf "ERROR: No hash found for %s\n" "${base}" >&2
> +    exit 2
>  fi
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot


Regards,

-- 
Samuel

[Buildroot] [PATCH 0/3] support/download: be more aggressive on missing hashes (branch yem/dl-hash)

[Yann E. MORIN]

All,

On 2015-03-16 23:41 +0100, Yann E. MORIN spake thusly:
> This series makes hashes mandatory when a .hash file exists.

Forget this series: it is broken for git downloads, and potentially for
all non-tarball downloads as well...

Sorry...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 3/3] support/download: always fail when there's no hash

[Arnout Vandecappelle]

On 16/03/15 23:41, Yann E. MORIN wrote:
> At the time we introduced hashes, we did not want to be too harsh in the
> beginning, and give people some time to adapt and accept to hashes. So
> we so far only whined^Wwarned about a missing hash file.
> 
> Some time has passed now, and people are still missing updating hashes
> when bumping packages.
> 
> Let's make that warning a little bit more annoying...

 If you ever revive this series (which I think would be a good idea), don't
forget to remove the reference to BR2_ENFORCE_CHECK_HASH from the manual as well.


 Regards,
 Arnout

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  support/download/check-hash | 8 ++------
>  1 file changed, 2 insertions(+), 6 deletions(-)
> 
> diff --git a/support/download/check-hash b/support/download/check-hash
> index 4cc62f3..7971c38 100755
> --- a/support/download/check-hash
> +++ b/support/download/check-hash
> @@ -88,10 +88,6 @@ while read t h f; do
>  done <"${h_file}"
>  
>  if [ ${nb_checks} -eq 0 ]; then
> -    if [ -n "${BR2_ENFORCE_CHECK_HASH}" ]; then
> -        printf "ERROR: No hash found for %s\n" "${base}" >&2
> -        exit 2
> -    else
> -        printf "WARNING: No hash found for %s\n" "${base}" >&2
> -    fi
> +    printf "ERROR: No hash found for %s\n" "${base}" >&2
> +    exit 2
>  fi
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 3/3] support/download: always fail when there's no hash

[Yann E. MORIN]

Arnout, All,

On 2015-03-19 00:15 +0100, Arnout Vandecappelle spake thusly:
> On 16/03/15 23:41, Yann E. MORIN wrote:
> > At the time we introduced hashes, we did not want to be too harsh in the
> > beginning, and give people some time to adapt and accept to hashes. So
> > we so far only whined^Wwarned about a missing hash file.
> > 
> > Some time has passed now, and people are still missing updating hashes
> > when bumping packages.
> > 
> > Let's make that warning a little bit more annoying...
> 
>  If you ever revive this series (which I think would be a good idea), don't
> forget to remove the reference to BR2_ENFORCE_CHECK_HASH from the manual as well.

Already done in v2:
    http://patchwork.ozlabs.org/patch/450979/

;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'