[Buildroot] [PATCH 0/2] core/download: fix extra downloads for scm-based main downloads

[Buildroot] [PATCH 0/2] core/download: fix extra downloads for scm-based main downloads

[Yann E. MORIN]

Hello All!

As reported by Luca [0], when the main download for a package is from an
SCM (gt, hg...), then extra downloads are also attempted using the same
site method.

This is due to the big rework of the download infra, when we introduced
the git cache feature.

It is to be noted that, before that big rewrite, the error was a silent
failure (and would have thus failed later), while we now detect that
failure at download time (rather than at extract or later).


Regards,
Yann E. MORIN.


The following changes since commit babc94e9dd4a1a04c4a0befdb6d32236a30b8ea8

  mbedtls: security bump to version 2.7.2 (2018-04-25 15:29:57 +0200)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 58a86c2e2f56abb66012e88c26f5e8c1d6737a2d

  core/pkg-infra: don't enforce site-method for extra downloads (2018-04-25 21:28:49 +0200)


----------------------------------------------------------------
Yann E. MORIN (2):
      core/pkg-infra: set no-check-hash only for main download
      core/pkg-infra: don't enforce site-method for extra downloads

 package/pkg-download.mk | 4 +---
 package/pkg-generic.mk  | 7 ++++++-
 2 files changed, 7 insertions(+), 4 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/2] core/pkg-infra: set no-check-hash only for main download

[Yann E. MORIN]

Currently, when the main download if from an SCM, we exclude all
downloads from being hash-checked, on the assumption that we don't
have hashes for downloads from an SCM.

However, the exclusion s computed on the DOWNLOAD macro, which is
called for each download of a package, thus effectively disabling
hash checks for extra downloads, even though those are only ever
download with wget.

What we really wanted to do, in fact, was to exclude just the main
download.

We fix that by appending the main source file to the global list of
excluded files, from within the generic-package infra itself.

Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
 package/pkg-download.mk | 2 --
 package/pkg-generic.mk  | 4 ++++
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 2c4ad3ba2c..2d9b0d4a41 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -92,8 +92,6 @@ endif
 
 define DOWNLOAD
 	$(Q)mkdir -p $($(PKG)_DL_DIR)
-	$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),
-		BR_NO_CHECK_HASH_FOR=$(notdir $(call qstrip,$(1)))) \
 	$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
 		-c '$($(PKG)_DL_VERSION)' \
 		-d '$($(PKG)_DL_DIR)' \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 1c9dd1d734..02a56e3d94 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -481,6 +481,10 @@ ifndef $(2)_PATCH
  endif
 endif
 
+ifneq ($$(filter bzr cvs hg svn,$$($(2)_SITE)),)
+BR_NO_CHECK_HASH_FOR += $$($(2)_SOURCE)
+endif
+
 $(2)_ALL_DOWNLOADS = \
 	$$(foreach p,$$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS),\
 		$$(if $$(findstring ://,$$(p)),$$(p),\
-- 
2.14.1

[Buildroot] [PATCH 2/2] core/pkg-infra: don't enforce site-method for extra downloads

[Yann E. MORIN]

The site method only ever applies to the main download, while extra
downloads are always to be fetches with wget.

However, the site method is prepended to the URL from within the
DOWNLOAD macro (well, a variable evaluated in the DOWNLOAD macro),
which is called for each download of a package, thus effectively
prepending the site method to all downloads, even the extra ones (and
the patches).

We fix that by prepending the site method from within the
generic-package infra, so that it only applies to the main download.

For that, we move the main _SOURCE out of the foreach loop, so that
we can prepend the site-method to it, without impacting the other
downloads.

Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
 package/pkg-download.mk | 2 +-
 package/pkg-generic.mk  | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 2d9b0d4a41..2ca6ccf44d 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -82,7 +82,7 @@ endif
 
 ifeq ($(BR2_PRIMARY_SITE_ONLY),)
 DOWNLOAD_URIS += \
-	-u $($(PKG)_SITE_METHOD)+$(patsubst %/,%,$(dir $(call qstrip,$(1))))
+	-u $(patsubst %/,%,$(dir $(call qstrip,$(1))))
 ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
 DOWNLOAD_URIS += \
 	-u $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 02a56e3d94..1affdb7f71 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -486,7 +486,8 @@ BR_NO_CHECK_HASH_FOR += $$($(2)_SOURCE)
 endif
 
 $(2)_ALL_DOWNLOADS = \
-	$$(foreach p,$$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS),\
+	$$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$($(2)_SOURCE) \
+	$$(foreach p,$$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS),\
 		$$(if $$(findstring ://,$$(p)),$$(p),\
 			$$($(2)_SITE)/$$(p)))
 
-- 
2.14.1

[Buildroot] [PATCH 2/2] core/pkg-infra: don't enforce site-method for extra downloads

[Luca Ceresoli]

Hi Yann,

On 25/04/2018 21:41, Yann E. MORIN wrote:
> The site method only ever applies to the main download, while extra
> downloads are always to be fetches with wget.
> 
> However, the site method is prepended to the URL from within the
> DOWNLOAD macro (well, a variable evaluated in the DOWNLOAD macro),
> which is called for each download of a package, thus effectively
> prepending the site method to all downloads, even the extra ones (and
> the patches).
> 
> We fix that by prepending the site method from within the
> generic-package infra, so that it only applies to the main download.
> 
> For that, we move the main _SOURCE out of the foreach loop, so that
> we can prepend the site-method to it, without impacting the other
> downloads.
> 
> Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>

Thanks for such a prompt fix!

Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

-- 
Luca

[Buildroot] [PATCH 1/2] core/pkg-infra: set no-check-hash only for main download

[Luca Ceresoli]

Hi Yann,

On 25/04/2018 21:41, Yann E. MORIN wrote:
> Currently, when the main download if from an SCM, we exclude all
> downloads from being hash-checked, on the assumption that we don't
> have hashes for downloads from an SCM.
> 
> However, the exclusion s computed on the DOWNLOAD macro, which is
> called for each download of a package, thus effectively disabling
> hash checks for extra downloads, even though those are only ever
> download with wget.
> 
> What we really wanted to do, in fact, was to exclude just the main
> download.
> 
> We fix that by appending the main source file to the global list of
> excluded files, from within the generic-package infra itself.
> 
> Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>

Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

-- 
Luca

[Buildroot] [PATCH 1/2] core/pkg-infra: set no-check-hash only for main download

[Thomas Petazzoni]

Hello,

On Wed, 25 Apr 2018 21:41:52 +0200, Yann E. MORIN wrote:
> Currently, when the main download if from an SCM, we exclude all
> downloads from being hash-checked, on the assumption that we don't
> have hashes for downloads from an SCM.
> 
> However, the exclusion s computed on the DOWNLOAD macro, which is
> called for each download of a package, thus effectively disabling
> hash checks for extra downloads, even though those are only ever
> download with wget.
> 
> What we really wanted to do, in fact, was to exclude just the main
> download.
> 
> We fix that by appending the main source file to the global list of
> excluded files, from within the generic-package infra itself.
> 
> Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
>  package/pkg-download.mk | 2 --
>  package/pkg-generic.mk  | 4 ++++
>  2 files changed, 4 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/2] core/pkg-infra: don't enforce site-method for extra downloads

[Thomas Petazzoni]

Hello,

On Wed, 25 Apr 2018 21:41:53 +0200, Yann E. MORIN wrote:
> The site method only ever applies to the main download, while extra
> downloads are always to be fetches with wget.
> 
> However, the site method is prepended to the URL from within the
> DOWNLOAD macro (well, a variable evaluated in the DOWNLOAD macro),
> which is called for each download of a package, thus effectively
> prepending the site method to all downloads, even the extra ones (and
> the patches).
> 
> We fix that by prepending the site method from within the
> generic-package infra, so that it only applies to the main download.
> 
> For that, we move the main _SOURCE out of the foreach loop, so that
> we can prepend the site-method to it, without impacting the other
> downloads.
> 
> Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
>  package/pkg-download.mk | 2 +-
>  package/pkg-generic.mk  | 3 ++-
>  2 files changed, 3 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com