[Buildroot] [PATCH 0/4 v11] Some download-related changes (branch yem/check-downloads)

[Buildroot] [PATCH 0/4 v11] Some download-related changes (branch yem/check-downloads)

[Yann E. MORIN]

Hello All!

This series contains those download-related changes:

  - Patch   1   : don't use DL_DIR as scratchpad for temporary VCS
    checkouts, since DL_DIR is a precious location

  - Patches 2..3: check hashes of downloaded files, to ensure the
    files are genuine (with doc)

  - Patch   4   : add hashes for ca-certificates

Hashes for more packages (eg. openssh, openssh, dropbear...) can be
added at a later step.


Changes v10 -> v11:
  - rebase after slight changes in master  (Peter)

Changes v9 -> v10:
  - use /bin/bash instead of /bin/sh, since we use bashisms  (Peter)
  - drop patches already applied  (Peter)

Changes v8 -> v9:
  - include change by Ryan about svn peg-revision
  - rebase on top of master after the crazy $$ springling ;-)

Changes v7 -> v8:
  - convert scp and localfiles  (Thomas DS, Peter)
  - typoes, expand MITM  (Thomas DS)
  - use a cleaner and saner code path to detect errors in helpers

Changes v6 -> v7:
  - use single-line title in manual  (Samuel)
  - use leftover cruft in bzr helper  (Thomas)

Changes v5 -> v6:
  - ensure we can run multiple parallel downloads from different build
    dirs with the same BR2_DL_DIR  (Thomas)
  - also convert bzr
  - fix for stray failed downloads

Changes v4 -> v5:
  - update ca-certificates hashes after version bump
  - fix detection of comments and empty lines in .hash file
  - rebase on top of master

Changes v3 -> v4:
  - enhance the manual, typoes  (Gustavo, Samuel, Thomas DS)
  - remove spurious test bump on package/fis  (Samuel)

Changes v2 -> v3:
  - avoid partial downloads  (Thomas DS)
  - store hash-type in .hash file  (Gustavo)
  - typoes  (Samuel, Gustavo)
  - add possibility to treat missing hashes as an error

Changes v1 -> v2:
  - don't do any behavioural change when switching to shell scripts,
    just reproduce exactly what was in the Makefile  (Luca, Arnout)
  - support more than one hash algorithm  (Arnout, Gustavo)
  - typoes  (Luca, Baruch)


Regards,
Yann E. MORIN.


The following changes since commit f56f4da887971f299ebea4f1e806a9b2eaf3ce38:

  zyre: remove spurious dependency on filemq (2014-07-03 16:30:36 +0200)

are available in the git repository at:

  git://gitorious.org/buildroot/buildroot.git yem/check-downloads

for you to fetch changes up to 00582a8025a651a3aefbf46849f1dd36fd1cb940:

  package/ca-certificates: add tarball's hashes (2014-07-03 19:27:57 +0200)

----------------------------------------------------------------
Yann E. MORIN (4):
      pkg-infra: don't use DL_DIR as scratchpad for temporary downloads
      pkg-infra: add possiblity to check downloaded files against known hashes
      manual: add documentation about packages' hashes
      package/ca-certificates: add tarball's hashes

 docs/manual/adding-packages-directory.txt    | 66 ++++++++++++++++++++++++
 package/ca-certificates/ca-certificates.hash |  3 ++
 package/pkg-download.mk                      | 20 ++++++--
 support/download/bzr                         | 25 +++++++--
 support/download/check-hash                  | 77 ++++++++++++++++++++++++++++
 support/download/cp                          | 12 ++++-
 support/download/cvs                         | 34 +++++++++---
 support/download/git                         | 33 +++++++++---
 support/download/hg                          | 35 ++++++++++---
 support/download/scp                         | 14 ++++-
 support/download/svn                         | 33 +++++++++---
 support/download/wget                        | 26 +++++++---
 12 files changed, 336 insertions(+), 42 deletions(-)
 create mode 100644 package/ca-certificates/ca-certificates.hash
 create mode 100755 support/download/check-hash

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] pkg-infra: don't use DL_DIR as scratchpad for temporary downloads

[Yann E. MORIN]

DL_DIR can be a very precious place for some users: they use it to store
all the downloaded archives to share across all their Buildroot (and
maybe non-Buildroot) builds.

We do not want to trash this location with our temporary downloads (e.g.
git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).

Turns out that we already have some kind of scratchpad, the BUILD_DIR.
Although it is not really a disposable location, that's the best we have
so far.

Also, we create the temporary tarballs with mktemp using the final tarrball,
as template, since we want the temporary to be on the same filesystem as
the final location, so the 'mv' is just a plain, atomic rename(2), and we
are not left with a half-copied file as the final location.

Using mktemp ensures all temp file names are unique, so it allows for
parallel downloads from different build dirs at the same time, without
cloberring each downloads.

Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared
locations, sometime with little place (eg. tmpfs), and some of the
repositories we clone/checkout can be very big.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
[tested a particular scenario that used to fail: two separate builds
using a shared DL_DIR, ccache enabled, so that they run almost
synchronously. These would download the same file at the same time,
corrupting each other. With the patches in this series, all works
fine.]

---
Change v7 -> v8:
  - use a cleaner and saner code path to detect errors
  - convert the cp and scp helpers, too  (Thomas DS)

Changes v6 -> v7:
  - drop leftovers from a previous attempt in bzr  (Thomas)

Changes v5 -> v6:
  - ensure we can do parallel downloads from multiple build-dirs  (Thomas)

Changes v4 -> v5
  - explain why we create the temp files where we create them  (Arnout)

Changes v3 -> v4:
  - remove spurious bump in package/fis  (Samuel)
---
 support/download/bzr  | 25 ++++++++++++++++++++++---
 support/download/cp   | 12 +++++++++++-
 support/download/cvs  | 34 +++++++++++++++++++++++++++-------
 support/download/git  | 33 ++++++++++++++++++++++++++-------
 support/download/hg   | 35 ++++++++++++++++++++++++++++-------
 support/download/scp  | 14 +++++++++++++-
 support/download/svn  | 33 ++++++++++++++++++++++++++-------
 support/download/wget | 26 ++++++++++++++++++++------
 8 files changed, 173 insertions(+), 39 deletions(-)

diff --git a/support/download/bzr b/support/download/bzr
index f07732e..3f52ee9 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -9,11 +9,30 @@ set -e
 #   $2: bzr revision
 #   $3: output file
 # And this environment:
-#   BZR       : the bzr command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   BZR      : the bzr command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 rev="${2}"
 output="${3}"
 
-${BZR} export "${output}" "${repo}" -r "${rev}"
+tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - the we move to a temp file in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${BZR} export "${tmp_dl}" "${repo}" -r "${rev}"; then
+    if mv "${tmp_dl}" "${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -f "${tmp_dl}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/cp b/support/download/cp
index 3e9f548..8f6bc06 100755
--- a/support/download/cp
+++ b/support/download/cp
@@ -13,4 +13,14 @@ set -e
 source="${1}"
 output="${2}"
 
-${LOCALFILES} "${source}" "${output}"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+ret=1
+if ${LOCALFILES} "${source}" "${tmp_output}"; then
+    mv "${tmp_output}" "${output}"
+    ret=0
+fi
+
+# Cleanup
+rm -f "${tmp_output}"
+exit ${ret}
diff --git a/support/download/cvs b/support/download/cvs
index 60787a6..9aeed79 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -11,8 +11,8 @@ set -e
 #   $4: package's basename (eg. foobar-1.2.3)
 #   $5: output file
 # And this environment:
-#   CVS       : the cvs command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   CVS      : the cvs command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 rev="${2}"
@@ -20,8 +20,28 @@ rawname="${3}"
 basename="${4}"
 output="${5}"
 
-cd "${BR2_DL_DIR}"
-${CVS} -z3 -d":pserver:anonymous@${repo}" \
-       co -d "${basename}" -r ":${rev}" -P "${rawname}"
-tar czf "${output}" "${basename}"
-rm -rf "${basename}"
+repodir="${basename}.tmp-cvs-checkout"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create a temporary tarball in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${CVS} -z3 -d":pserver:anonymous@${repo}" \
+           co -d "${repodir}" -r ":${rev}" -P "${rawname}"; then
+    if tar czf "${tmp_output}" "${repodir}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -rf "${repodir}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/git b/support/download/git
index 5397096..badb491 100755
--- a/support/download/git
+++ b/support/download/git
@@ -10,15 +10,28 @@ set -e
 #   $3: package's basename (eg. foobar-1.2.3)
 #   $4: output file
 # And this environment:
-#   BR2_DL_DIR: path to Buildroot's download dir
-#   GIT       : the git command to call
+#   GIT      : the git command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 cset="${2}"
 basename="${3}"
 output="${4}"
 
-repodir="${BR2_DL_DIR}/${basename}"
+repodir="${basename}.tmp-git-checkout"
+tmp_tar="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create the uncomporessed tarball in tht same trashable location
+# - then we create a temporary compressed tarball in the final location, so
+#   it is on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
 
 if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
     printf "Doing shallow clone\n"
@@ -28,10 +41,16 @@ else
     ${GIT} clone --bare "${repo}" "${repodir}"
 fi
 
+ret=1
 pushd "${repodir}" >/dev/null
-${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}"
-gzip -c "${output}.tmp" >"${output}"
-rm -f "${output}.tmp"
+if ${GIT} archive --prefix="${basename}/" -o "${tmp_tar}" \
+                  --format=tar "${cset}"; then
+    if gzip -c "${tmp_tar}" >"${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
 popd >/dev/null
 
-rm -rf "${repodir}"
+rm -rf "${repodir}" "${tmp_tar}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/hg b/support/download/hg
index d2e69c3..6e9e26b 100755
--- a/support/download/hg
+++ b/support/download/hg
@@ -10,16 +10,37 @@ set -e
 #   $3: package's basename (eg. foobar-1.2.3)
 #   $4: output file
 # And this environment:
-#   HG        : the hg command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   HG       : the hg command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 cset="${2}"
 basename="${3}"
 output="${4}"
 
-cd "${BR2_DL_DIR}"
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
-${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \
-              --rev "${cset}" "${output}"
-rm -rf "${basename}"
+repodir="${basename}.tmp-hg-checkout"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create a temporary tarball in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${HG} clone --noupdate --rev "${cset}" "${repo}" "${repodir}"; then
+    if ${HG} archive --repository "${repodir}" --type tgz \
+                     --prefix "${basename}" --rev "${cset}" \
+                     "${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -rf "${repodir}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/scp b/support/download/scp
index 2997927..d3aad43 100755
--- a/support/download/scp
+++ b/support/download/scp
@@ -12,5 +12,17 @@ set -e
 
 url="${1}"
 output="${2}"
+tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
 
-${SCP} "${url}" "${output}"
+ret=1
+if ${SCP} "${url}" "${tmp_dl}"; then
+    if mv "${tmp_dl}" "${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -f "${tmp_dl}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/svn b/support/download/svn
index 142a8d8..39cbbcb 100755
--- a/support/download/svn
+++ b/support/download/svn
@@ -10,16 +10,35 @@ set -e
 #   $3: package's basename (eg. foobar-1.2.3)
 #   $4: output file
 # And this environment:
-#   SVN       : the svn command to call
-#   BR2_DL_DIR: path to Buildroot's download dir
+#   SVN      : the svn command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 repo="${1}"
 rev="${2}"
 basename="${3}"
 output="${4}"
 
-pushd "${BR2_DL_DIR}" >/dev/null
-${SVN} export "${repo}@${rev}" "${basename}"
-tar czf "${output}" "${basename}"
-rm -rf "${basename}"
-popd >/dev/null
+repodir="${basename}.tmp-svn-checkout"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create a temporary tarball in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${SVN} export "${repo}@${rev}" "${repodir}"; then
+    if tar czf "${tmp_output}" "${repodir}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
+fi
+
+# Cleanup
+rm -rf "${repodir}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/wget b/support/download/wget
index 91ffd10..cbeca3b 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -8,14 +8,28 @@ set -e
 #   $1: URL
 #   $2: output file
 # And this environment:
-#   WGET      : the wget command to call
+#   WGET     : the wget command to call
+#   BUILD_DIR: path to Buildroot's build dir
 
 url="${1}"
 output="${2}"
 
-if ${WGET} -O "${output}.tmp" "${url}"; then
-    mv "${output}.tmp" "${output}"
-else
-    rm -f "${output}.tmp"
-    exit 1
+tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we copy to a temporary tarball in the final location, so it is
+#   on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${WGET} -O "${tmp_dl}" "${url}"; then
+    if mv "${tmp_dl}" "${tmp_output}"; then
+        mv "${tmp_output}" "${output}"
+        ret=0
+    fi
 fi
+
+rm -f "${tmp_dl}" "${tmp_output}"
+exit ${ret}
-- 
1.9.1

[Buildroot] [PATCH 2/4] pkg-infra: add possiblity to check downloaded files against known hashes

[Yann E. MORIN]

Some of the packages that Buildroot might build are sensitive packages,
related to security: openssl, dropbear, ca-certificates...

Some of those packages are downloaded over plain http, because there is
no way to get them over a secure channel, such as https.

In these dark times of pervasive surveillance, the potential for harm that
a tampered-with package could generate, we may want to check the integrity
of those sensitive packages.

So, each package may now provide a list of hashes for all files that needs
to be downloaded, and Buildroot will just fail if any downloaded file does
not match its known hash, in which case it is removed.

Hashes can be any of the md5, sha1 or sha2 variants, and will be checked
even if the file was pre-downloaded.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Baruch Siach <baruch@tkos.co.il>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>

---
Changes v9 -> v10:
  - use bash as shell  (Peter)

Changes v7 -> v8
  - expand MITM to its full-length meaning  (Thomas DS)
  - typo  (Thomas DS)

Changes v4 -> v5:
  - fix detection of comments and empty lines

---
Note: this is not a bullet-proof solution, since Buildroot may itself be
compromised. But since we do sign our releases, then we secure the list of
hashes at the same time. Only random snapshots from the repository may be
at risk of tampering, although this is highly doubtfull, given how git
stores its data.
---
 package/pkg-download.mk     | 20 ++++++++++--
 support/download/check-hash | 76 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 93 insertions(+), 3 deletions(-)
 create mode 100755 support/download/check-hash

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index d3cd0c1..7f208d5 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -58,6 +58,17 @@ domainseparator=$(if $(1),$(1),/)
 # github(user,package,version): returns site of GitHub repository
 github = https://github.com/$(1)/$(2)/archive/$(3)
 
+# Helper for checking a tarball's checksum
+# If the hash does not match, remove the incorrect file
+# $(1): the path to the file with the hashes
+# $(2): the full path to the file to check
+define VERIFY_HASH
+	if ! support/download/check-hash $(1) $(2); then \
+		rm -f $(2); \
+		exit 1; \
+	fi
+endef
+
 ################################################################################
 # The DOWNLOAD_* helpers are in charge of getting a working copy
 # of the source repository for their corresponding SCM,
@@ -148,7 +159,8 @@ endef
 define DOWNLOAD_SCP
 	test -e $(DL_DIR)/$(2) || \
 	$(EXTRA_ENV) support/download/scp '$(call stripurischeme,$(call qstrip,$(1)))' \
-					  $(DL_DIR)/$(2)
+					  $(DL_DIR)/$(2) && \
+	$(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
 endef
 
 define SOURCE_CHECK_SCP
@@ -179,7 +191,8 @@ endef
 
 define DOWNLOAD_WGET
 	test -e $(DL_DIR)/$(2) || \
-	$(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2)
+	$(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2) && \
+	$(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
 endef
 
 define SOURCE_CHECK_WGET
@@ -193,7 +206,8 @@ endef
 define DOWNLOAD_LOCALFILES
 	test -e $(DL_DIR)/$(2) || \
 	$(EXTRA_ENV) support/download/cp $(call stripurischeme,$(call qstrip,$(1))) \
-					 $(DL_DIR)
+					 $(DL_DIR) && \
+	$(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
 endef
 
 define SOURCE_CHECK_LOCALFILES
diff --git a/support/download/check-hash b/support/download/check-hash
new file mode 100755
index 0000000..9ea7c41
--- /dev/null
+++ b/support/download/check-hash
@@ -0,0 +1,76 @@
+#!/bin/bash
+set -e
+
+# Helper to check a file matches its known hash
+# Call it with:
+#   $1: the full path to the file to check
+#   $2: the path of the file containing all the the expected hashes
+
+h_file="${1}"
+file="${2}"
+
+# Does the hash-file exist?
+if [ ! -f "${h_file}" ]; then
+    exit 0
+fi
+
+# Check one hash for a file
+# $1: known hash
+# $2: file (full path)
+check_one_hash() {
+    _h="${1}"
+    _known="${2}"
+    _file="${3}"
+
+    # Note: sha3 is not supported, since there is currently no implementation
+    #       (the NIST has yet to publish the parameters).
+    case "${_h}" in
+        md5|sha1)                       ;;
+        sha224|sha256|sha384|sha512)    ;;
+        *) # Unknown hash, exit with error
+            printf "ERROR: unknown hash '%s' for '%s'\n"  \
+                   "${_h}" "${_file##*/}" >&2
+            exit 1
+            ;;
+    esac
+
+    # Do the hashes match?
+    _hash=$( ${_h}sum "${_file}" |cut -d ' ' -f 1 )
+    if [ "${_hash}" = "${_known}" ]; then
+        printf "%s: OK (%s: %s)\n" "${_file##*/}" "${_h}" "${_hash}"
+        return 0
+    fi
+
+    printf "ERROR: %s has wrong %s hash:\n" "${_file##*/}" "${_h}" >&2
+    printf "ERROR: expected: %s\n" "${_known}" >&2
+    printf "ERROR: got     : %s\n" "${_hash}" >&2
+    printf "ERROR: Incomplete download, or man-in-the-middle (MITM) attack\n" >&2
+
+    exit 1
+}
+
+# Do we know one or more hashes for that file?
+nb_checks=0
+while read t h f; do
+    case "${t}" in
+        ''|'#'*)
+            # Skip comments and empty lines
+            continue
+            ;;
+        *)
+            if [ "${f}" = "${file##*/}" ]; then
+                check_one_hash "${t}" "${h}" "${file}"
+                : $((nb_checks++))
+            fi
+            ;;
+    esac
+done <"${h_file}"
+
+if [ ${nb_checks} -eq 0 ]; then
+    if [ -n "${BR2_ENFORCE_CHECK_HASH}" ]; then
+        printf "ERROR: No hash found for %s\n" "${file}" >&2
+        exit 1
+    else
+        printf "WARNING: No hash found for %s\n" "${file}" >&2
+    fi
+fi
-- 
1.9.1

[Buildroot] [PATCH 3/4] manual: add documentation about packages' hashes

[Yann E. MORIN]

Although md5 is, for legacy reasons, a supported hash type,
it is not documented on purpose, since it is now known to
be weak.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Baruch Siach <baruch@tkos.co.il>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
Changes v6 -> v7:
  - use single-line title  (Samuel)
---
 docs/manual/adding-packages-directory.txt | 66 +++++++++++++++++++++++++++++++
 support/download/check-hash               |  1 +
 2 files changed, 67 insertions(+)

diff --git a/docs/manual/adding-packages-directory.txt b/docs/manual/adding-packages-directory.txt
index 69b012c..c2f0a1a 100644
--- a/docs/manual/adding-packages-directory.txt
+++ b/docs/manual/adding-packages-directory.txt
@@ -357,3 +357,69 @@ different way, using different infrastructures:
 
 Further formatting details: see xref:writing-rules-mk[the writing
 rules].
+
+[[adding-packages-hash]]
+=== The +.hash+ file
+
+Optionally, you can add a third file, named +libfoo.hash+, that contains
+the hashes of the downloaded files for the +libfoo+ package.
+
+The hashes stored in that file are used to validate the integrity of the
+downloaded files.
+
+The format of this file is one line for each file for which to check the
+hash, each line being space-separated, with these three fields:
+
+* the type of hash, one of:
+** +sha1+, +sha224+, +sha256+, +sha384+, +sha512+
+* the hash of the file:
+** for +sha1+, 40 hexadecimal characters
+** for +sha224+, 56 hexadecimal characters
+** for +sha256+, 64 hexadecimal characters
+** for +sha384+, 96 hexadecimal characters
+** for +sha512+, 128 hexadecimal characters
+* the name of the file, without any directory component
+
+Lines starting with a +#+ sign are considered comments, and ignored. Empty
+lines are ignored.
+
+There can be more than one hash for a single file, each on its own line. In
+this case, all hashes must match.
+
+Ideally, the hashes stored in this file should match the hashes published by
+upstream, e.g. on their website, in the e-mail announcement... If upstream
+provides more than one type of hash (say, +sha1+ and +sha512+), then it is
+best to add all those hashes in the +.hash+ file. If upstream does not
+provide any hash, then compute at least one yourself, and mention this in a
+comment line above the hashes.
+
+*Note:* the number of spaces does not matter, so one can use spaces to
+properly align the different fields.
+
+The example below defines a +sha1+ and a +sha256+ published by upstream for
+the main +libfoo-1.2.3.tar.bz2+ tarball, plus two locally-computed hashes,
+a +sha256+ for a downloaded patch, and a +sha1+ for a downloaded binary blob:
+
+----
+# Hashes from: http://www.foosoftware.org/download/libfoo-1.2.3.tar.bz2.{sha1,sha256}:
+sha1   486fb55c3efa71148fe07895fd713ea3a5ae343a                         libfoo-1.2.3.tar.bz2
+sha256 efc8103cc3bcb06bda6a781532d12701eb081ad83e8f90004b39ab81b65d4369 libfoo-1.2.3.tar.bz2
+
+# No upstream hashes for the following:
+sha256 ff52101fb90bbfc3fe9475e425688c660f46216d7e751c4bbdb1dc85cdccacb9 libfoo-fix-blabla.patch
+sha1   2d608f3c318c6b7557d551a5a09314f03452f1a1                         libfoo-data.bin
+----
+
+If the +.hash+ file is present, and it contains one or more hashes for a
+downloaded file, the hash(es) computed by Buildroot (after download) must
+match the hash(es) stored in the +.hash+ file. If one or more hashes do
+not match, Buildroot considers this an error, deletes the downloaded file,
+and aborts.
+
+If the +.hash+ file is present, but it does not contain a hash for a
+downloaded file, no check is done for that file. If you set the
+environment variable +BR2_ENFORCE_CHECK_HASH+ to a non-empty value, and
+there is no hash for a downloaded file, Buildroot considers this an
+error, deletes the downloaded file, and aborts.
+
+If the +.hash+ file is missing, then no check is done at all.
diff --git a/support/download/check-hash b/support/download/check-hash
index 9ea7c41..f1e0c1b 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -22,6 +22,7 @@ check_one_hash() {
     _known="${2}"
     _file="${3}"
 
+    # Note: md5 is supported, but undocumented on purpose.
     # Note: sha3 is not supported, since there is currently no implementation
     #       (the NIST has yet to publish the parameters).
     case "${_h}" in
-- 
1.9.1

[Buildroot] [PATCH 4/4] package/ca-certificates: add tarball's hashes

[Yann E. MORIN]

ca-certificates contains sensitive security-related information,
and we want to ensure the archive that we download has not been
compromised.

Add the sha1 and sha256 hashes from Debian's packaging.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Martin Bark <martin@barkynet.com>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>

---
Changes v4 -> v5:
  - update hashes since we've bumped the version
---
 package/ca-certificates/ca-certificates.hash | 3 +++
 1 file changed, 3 insertions(+)
 create mode 100644 package/ca-certificates/ca-certificates.hash

diff --git a/package/ca-certificates/ca-certificates.hash b/package/ca-certificates/ca-certificates.hash
new file mode 100644
index 0000000..bcd0723
--- /dev/null
+++ b/package/ca-certificates/ca-certificates.hash
@@ -0,0 +1,3 @@
+# hashes from: $(CA_CERTIFICATES_SITE)/ca-certificates_$(CA_CERTIFICATES_VERSION).dsc :
+sha1   ad57a45f0422fafd78a2e8191e5204f2306cc91b                         ca-certificates_20140223.tar.xz
+sha256 815b7cd97200b0d76450bb3e7d9b65997ac494ab6467b17369f65b2ef94bcb0c ca-certificates_20140223.tar.xz
-- 
1.9.1

[Buildroot] [PATCH 1/4] pkg-infra: don't use DL_DIR as scratchpad for temporary downloads

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > DL_DIR can be a very precious place for some users: they use it to store
 > all the downloaded archives to share across all their Buildroot (and
 > maybe non-Buildroot) builds.

 > We do not want to trash this location with our temporary downloads (e.g.
 > git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).

 > Turns out that we already have some kind of scratchpad, the BUILD_DIR.
 > Although it is not really a disposable location, that's the best we have
 > so far.

 > Also, we create the temporary tarballs with mktemp using the final tarrball,

s/tarrball/tarball/

 > ---
 > Change v7 -> v8:
 >   - use a cleaner and saner code path to detect errors
 >   - convert the cp and scp helpers, too  (Thomas DS)

Committed, thanks.

I think it would be good to either delay the creation of tmp_output
until just before it it used and/or cleanup on errors to not end up with
a bunch of odd named temporary files in DL_DIR, but that can be added
later.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/4] pkg-infra: add possiblity to check downloaded files against known hashes

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Some of the packages that Buildroot might build are sensitive packages,
 > related to security: openssl, dropbear, ca-certificates...

 > Some of those packages are downloaded over plain http, because there is
 > no way to get them over a secure channel, such as https.

 > In these dark times of pervasive surveillance, the potential for harm that
 > a tampered-with package could generate, we may want to check the integrity
 > of those sensitive packages.

 > So, each package may now provide a list of hashes for all files that needs
 > to be downloaded, and Buildroot will just fail if any downloaded file does
 > not match its known hash, in which case it is removed.

 > Hashes can be any of the md5, sha1 or sha2 variants, and will be checked
 > even if the file was pre-downloaded.

 > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
 > Cc: Baruch Siach <baruch@tkos.co.il>
 > Cc: Arnout Vandecappelle <arnout@mind.be>
 > Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
 > Reviewed-by: Samuel Martin <s.martin49@gmail.com>
 > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>

 > ---
 > Changes v9 -> v10:
 >   - use bash as shell  (Peter)

 > Changes v7 -> v8
 >   - expand MITM to its full-length meaning  (Thomas DS)
 >   - typo  (Thomas DS)

 > Changes v4 -> v5:
 >   - fix detection of comments and empty lines

 > ---
 > Note: this is not a bullet-proof solution, since Buildroot may itself be
 > compromised. But since we do sign our releases, then we secure the list of
 > hashes at the same time. Only random snapshots from the repository may be
 > at risk of tampering, although this is highly doubtfull, given how git
 > stores its data.
 > ---
 >  package/pkg-download.mk     | 20 ++++++++++--
 >  support/download/check-hash | 76 +++++++++++++++++++++++++++++++++++++++++++++
 >  2 files changed, 93 insertions(+), 3 deletions(-)
 >  create mode 100755 support/download/check-hash

 > diff --git a/package/pkg-download.mk b/package/pkg-download.mk
 > index d3cd0c1..7f208d5 100644
 > --- a/package/pkg-download.mk
 > +++ b/package/pkg-download.mk
 > @@ -58,6 +58,17 @@ domainseparator=$(if $(1),$(1),/)
 >  # github(user,package,version): returns site of GitHub repository
 >  github = https://github.com/$(1)/$(2)/archive/$(3)
 
 > +# Helper for checking a tarball's checksum
 > +# If the hash does not match, remove the incorrect file
 > +# $(1): the path to the file with the hashes
 > +# $(2): the full path to the file to check
 > +define VERIFY_HASH
 > +	if ! support/download/check-hash $(1) $(2); then \
 > +		rm -f $(2); \
 > +		exit 1; \
 > +	fi

I wonder if it would be a worthwhile optimization to do the -f
<whatever>.hash file check here, so we don't need to run the script at
all for packages without hashes, but perhaps that's not measurable.


 > +++ b/support/download/check-hash
 > @@ -0,0 +1,76 @@
 > +#!/bin/bash
 > +set -e
 > +
 > +# Helper to check a file matches its known hash
 > +# Call it with:
 > +#   $1: the full path to the file to check
 > +#   $2: the path of the file containing all the the expected hashes
 > +
 > +h_file="${1}"
 > +file="${2}"
 > +
 > +# Does the hash-file exist?
 > +if [ ! -f "${h_file}" ]; then
 > +    exit 0
 > +fi
 > +
 > +# Check one hash for a file
 > +# $1: known hash
 > +# $2: file (full path)
 > +check_one_hash() {
 > +    _h="${1}"
 > +    _known="${2}"
 > +    _file="${3}"
 > +
 > +    # Note: sha3 is not supported, since there is currently no implementation
 > +    #       (the NIST has yet to publish the parameters).
 > +    case "${_h}" in
 > +        md5|sha1)                       ;;
 > +        sha224|sha256|sha384|sha512)    ;;

All of these come from coreutils and have been available for ages so
they are basically guaranteed to be available on the build machine,
right?

If not, we should probably warn with a sensible error message instead of
what we get now:

sudo mv /usr/bin/sha1sum{,-dontuse}
make ca-certificates-source
support/download/check-hash: line 39: sha1sum: command not found
ERROR: ca-certificates_20140223.tar.xz has wrong sha1 hash:
ERROR: expected: ad57a45f0422fafd78a2e8191e5204f2306cc91b
ERROR: got     : 
ERROR: Incomplete download, or man-in-the-middle (MITM) attack
package/pkg-generic.mk:73: recipe for target '/home/peko/source/buildroot/output/build/ca-certificates-20140223/.stamp_downloaded' failed


But we can fix that up later if needed - Committed, thanks.


-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 3/4] manual: add documentation about packages' hashes

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Although md5 is, for legacy reasons, a supported hash type,
 > it is not documented on purpose, since it is now known to
 > be weak.

 > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
 > Cc: Baruch Siach <baruch@tkos.co.il>
 > Cc: Arnout Vandecappelle <arnout@mind.be>
 > Cc: Samuel Martin <s.martin49@gmail.com>
 > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
 > Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

 > ---
 > Changes v6 -> v7:
 >   - use single-line title  (Samuel)

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 4/4] package/ca-certificates: add tarball's hashes

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > ca-certificates contains sensitive security-related information,
 > and we want to ensure the archive that we download has not been
 > compromised.

 > Add the sha1 and sha256 hashes from Debian's packaging.

 > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
 > Cc: Martin Bark <martin@barkynet.com>
 > Reviewed-by: Samuel Martin <s.martin49@gmail.com>

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 1/4] pkg-infra: don't use DL_DIR as scratchpad for temporary downloads

[Yann E. MORIN]

Peter, All,

On 2014-07-04 23:35 +0200, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> 
>  > DL_DIR can be a very precious place for some users: they use it to store
>  > all the downloaded archives to share across all their Buildroot (and
>  > maybe non-Buildroot) builds.
> 
>  > We do not want to trash this location with our temporary downloads (e.g.
>  > git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).
> 
>  > Turns out that we already have some kind of scratchpad, the BUILD_DIR.
>  > Although it is not really a disposable location, that's the best we have
>  > so far.
> 
>  > Also, we create the temporary tarballs with mktemp using the final tarrball,
> 
> s/tarrball/tarball/
> 
>  > ---
>  > Change v7 -> v8:
>  >   - use a cleaner and saner code path to detect errors
>  >   - convert the cp and scp helpers, too  (Thomas DS)
> 
> Committed, thanks.
> 
> I think it would be good to either delay the creation of tmp_output
> until just before it it used and/or cleanup on errors to not end up with
> a bunch of odd named temporary files in DL_DIR, but that can be added
> later.

As discussed on IRC: "yeah, probably, yes."
I'll cook up a patch.

Thanks.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/4] pkg-infra: add possiblity to check downloaded files against known hashes

[Bernd Kuhls]

Hi Yann,

Am Thu, 03 Jul 2014 21:36:21 +0200 schrieb Yann E. MORIN:

> +    # Note: sha3 is not supported, since there is currently no 
implementation
> +    #       (the NIST has yet to publish the parameters).

sqlite.org started to release sha3 hashes for their tarballs:
https://www.sqlite.org/download.html

Apparently a sha3sum utility is not available so I tried

openssl dgst -sha3-256 dl/sqlite/sqlite-autoconf-3330000.tar.gz

which shows the hash mentioned upstream:

SHA3-256(dl/sqlite/sqlite-autoconf-3330000.tar.gz)= 
6e94e9453cedf8f2023e3923f856741d1e28a2271e9f93d24d95fa48870edaad

Should we add the openssl-way to verify sha3 hashes?

Should our hash files support different sha3 methods?
Openssl support sha3-224, sha3-256, sha3-384 & sha3-512.

Regards, Bernd

[Buildroot] [PATCH 2/4] pkg-infra: add possiblity to check downloaded files against known hashes

[Yann E. MORIN]

Bernd, All,

On 2020-11-05 19:38 +0100, Bernd Kuhls spake thusly:
> Am Thu, 03 Jul 2014 21:36:21 +0200 schrieb Yann E. MORIN:
> sqlite.org started to release sha3 hashes for their tarballs:
> https://www.sqlite.org/download.html
> 
> Apparently a sha3sum utility is not available so I tried
> 
> openssl dgst -sha3-256 dl/sqlite/sqlite-autoconf-3330000.tar.gz
> 
> which shows the hash mentioned upstream:
> 
> SHA3-256(dl/sqlite/sqlite-autoconf-3330000.tar.gz)= 
> 6e94e9453cedf8f2023e3923f856741d1e28a2271e9f93d24d95fa48870edaad
> 
> Should we add the openssl-way to verify sha3 hashes?

Unfortunately, this is only present in very recent versions of openssl.
It was missing in Ubuntu 16.04, while it is present in 18.04 (both still
supported LTS versions)

So, I am not really sure how we can move forward...

If we were to add it, and were to make it mandatory that we be able to
validate them, then it would mean we would have to build our own
host-openssl prior to doing downloads. This is very not nice (see the
existing issue with host-tar, which we are trying to get rid of).

If we were to add it, and only check for them if the host openssl did
support them, and ignore those hashes otherwise, then it would mean some
packages which have only sha3 hashes would not be checked (even if
in-tree we ensure a fallback hash exists, that might not be the case for
external trees, or the situation can slip our attention).

If we don't add it, and upstream does not provide anything else, then we
will have to resort to locally computing our own hashes.

Franckly, my preference would got for the third option: not support sha3,
and add our own hashes. Adding our own hashes is anyway what we already
do for a lot of packages already. sha3 does provide extra resilience,
thanks to its novel design, but sha2 is still far from being considered
broken yet [0].

One thing we may consider adding to reinforce our robustness, is to
store the file size in the hash file, in addition to the hash, e.g.:

    sha256  c35d87f1d0...bbff51fe689  2439463  busybox-1.32.0.tar.bz2

This would protect against size-extension attacks, which afaiu are the
only attacks really considered for now against sha2 [1]...

And we could be backward compatible and recognise 3- or 4-field lines,
to decide whether the size is present of not, and not checking it in the
latter case.

> Should our hash files support different sha3 methods?
> Openssl support sha3-224, sha3-256, sha3-384 & sha3-512.

If we were to add such support, then yes, we would have to add all four
lengths, as we do for sha224, sha256, sha384, and sha512, which all four
are really just sha2 variants.

But unfortunately, I am not convinced we can implement it for now. And
be sure it saddens me... :-(

[0] https://issha2brokenyet.com/  (just for the lols)
[1] https://en.wikipedia.org/wiki/SHA-2#Comparison_of_SHA_functions

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/4] pkg-infra: add possiblity to check downloaded files against known hashes

[Bernd Kuhls]

Am Thu, 05 Nov 2020 22:12:32 +0100 schrieb Yann E. MORIN:

> Bernd, All,
> 
> On 2020-11-05 19:38 +0100, Bernd Kuhls spake thusly:
>> Am Thu, 03 Jul 2014 21:36:21 +0200 schrieb Yann E. MORIN: sqlite.org
>> started to release sha3 hashes for their tarballs:
>> https://www.sqlite.org/download.html
>> 
>> Apparently a sha3sum utility is not available so I tried
>> 
>> openssl dgst -sha3-256 dl/sqlite/sqlite-autoconf-3330000.tar.gz
>> 
>> which shows the hash mentioned upstream:
>> 
>> SHA3-256(dl/sqlite/sqlite-autoconf-3330000.tar.gz)=
>> 6e94e9453cedf8f2023e3923f856741d1e28a2271e9f93d24d95fa48870edaad
>> 
>> Should we add the openssl-way to verify sha3 hashes?
> 
> Unfortunately, this is only present in very recent versions of openssl.
> It was missing in Ubuntu 16.04, while it is present in 18.04 (both still
> supported LTS versions)
> 
> So, I am not really sure how we can move forward...

Hi Yann,

what about host-rhash?
http://patchwork.ozlabs.org/project/buildroot/list/?series=212773

$ output/host/bin/rhash --sha3-256 dl/sqlite/sqlite-
autoconf-3330000.tar.gz 
6e94e9453cedf8f2023e3923f856741d1e28a2271e9f93d24d95fa48870edaad  dl/
sqlite/sqlite-autoconf-3330000.tar.gz

Regards, Bernd

[Buildroot] [PATCH 2/4] pkg-infra: add possiblity to check downloaded files against known hashes

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

 > So, I am not really sure how we can move forward...

 > If we were to add it, and were to make it mandatory that we be able to
 > validate them, then it would mean we would have to build our own
 > host-openssl prior to doing downloads. This is very not nice (see the
 > existing issue with host-tar, which we are trying to get rid of).

Indeed, lets not go there before a significant amount of upstreams start
to only provide sha3 hashes.


 > Franckly, my preference would got for the third option: not support sha3,
 > and add our own hashes. Adding our own hashes is anyway what we already
 > do for a lot of packages already. sha3 does provide extra resilience,
 > thanks to its novel design, but sha2 is still far from being considered
 > broken yet [0].

Agreed!

 > One thing we may consider adding to reinforce our robustness, is to
 > store the file size in the hash file, in addition to the hash, e.g.:

 >     sha256  c35d87f1d0...bbff51fe689  2439463  busybox-1.32.0.tar.bz2

 > This would protect against size-extension attacks, which afaiu are the
 > only attacks really considered for now against sha2 [1]...

 > And we could be backward compatible and recognise 3- or 4-field lines,
 > to decide whether the size is present of not, and not checking it in the
 > latter case.

I wonder if the gain is worth the extra complexity for our users and in
the implementation. Are there are any realistic size extension attacks
against sha256?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/4] pkg-infra: add possiblity to check downloaded files against known hashes

[Yann E. MORIN]

Peter, All,

On 2020-11-07 18:27 +0100, Peter Korsgaard spake thusly:
[--SNIP--]
>  > One thing we may consider adding to reinforce our robustness, is to
>  > store the file size in the hash file, in addition to the hash, e.g.:
[--SNIP--]
>  > This would protect against size-extension attacks, which afaiu are the
>  > only attacks really considered for now against sha2 [1]...
[--SNIP--]
> I wonder if the gain is worth the extra complexity for our users and in
> the implementation.

The implementation is pretty trivial. I have more changes against the
manual than I have against the code...

> Are there are any realistic size extension attacks
> against sha256?

Good question... At least, it looks like it was known from the onset,
during the standardisation process:

    https://www.cryptologie.net/article/417/how-did-length-extension-attacks-made-it-into-sha-2/
    http://www.cs.utsa.edu/~wagner/CS4363/SHS/dfips-180-2-comments1.pdf (comment #3)

And the following seems to imply it is pretty trivial (for a seasoned
cryptographer at least):

    https://www.whitehatsec.com/blog/hash-length-extension-attacks/
    https://blog.skullsecurity.org/2012/everything-you-need-to-know-about-hash-length-extension-attacks

So yes, it looks like length extension attacks (LEA) are easy...

However, now that I've read a bit more, especially that last article, I
doubt we'd be susceptible to such attacks. Indeed, LEA target MACs, that
is signatures. We're not using hashes that way; we just hash files, not
secrets.

So maybe this is not so interesting to add the size to the hash file...

Crypto is hard, damit... ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/4] pkg-infra: add possiblity to check downloaded files against known hashes

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

 >> I wonder if the gain is worth the extra complexity for our users and in
 >> the implementation.

 > The implementation is pretty trivial. I have more changes against the
 > manual than I have against the code...

yes, I was mainly referring to the first part, E.G. our users. Now the
hash lines are an algorithm prefix and then the output of <alg>sum, but
with the suggested change this is no longer the case.

But yes, it isn't a big complication.


 > However, now that I've read a bit more, especially that last article, I
 > doubt we'd be susceptible to such attacks. Indeed, LEA target MACs, that
 > is signatures. We're not using hashes that way; we just hash files, not
 > secrets.

I am not a cryptographer, but I would imagine that creating LEA attacks
against the kind of hashes we have is HARD to do, otherwise a lot of
things would break, and all those upstreams publishing hashes with their
releases would be for nothing.

-- 
Bye, Peter Korsgaard