[Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account

[Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account

[Thomas Petazzoni]

Hello,

The starting point for this series is to fix the following build
failure:

  http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log

The problem is the following:

 - The _SOURCE for gettext-tiny is a .tar.gz

 - In addition to this _SOURCE tarball, we also extract the
   gettext-gnu source code, which is a .tar.xz

 - Extracting .tar.xz may require building host-xzcat, is xzcat is not
   available system-wide

 - The build fails because host-xzcat is not added in the dependencies
   of gettext-tiny. Indeed, while generic-package takes care of adding
   the extract dependencies needed for _SOURCE, it does not know about
   other extract dependencies.

This issue appears in
http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log
because BR2_PER_PACKAGE_DIRECTORIES=y and therefore any missing
dependency is immediately visible. But in theory, it could appear
without per-package support.

This series solves that by introducing an extractor-pkg-dependency
that returns the list of packages that must be added to
<pkg>_EXTRACT_DEPENDENCIES to extract the archive passed as
argument. It is then used in package/pkg-generic.mk to replace the
existing logic to calculate <pkg>_EXTRACT_DEPENDENCIES, and then in
the several places where suitable-extractor was used to manually
extract a tarball.

We also rename extractor-dependency to extractor-system-dependency to
clarify its difference from extractor-pkg-dependency, and we rework
the implementation of extractor-system-dependency.

Best regards,

Thomas Petazzoni

Thomas Petazzoni (12):
  package/pkg-utils.mk: rename extractor-dependency to
    extractor-system-dependency
  package/pkg-utils.mk: add extractor-pkg-dependency macro
  package/pkg-utils.mk: rework implementation of
    extractor-system-dependency
  package/pkg-generic.mk: use extractor-pkg-dependency macro
  arch/arch.mk.xtensa: fix usage of arch-xtensa-overlay-extract
  package/android-tools: add missing extract dependencies
  package/cargo: add missing extract dependencies
  package/gettext-tiny: add missing extract dependencies
  package/murata-cyw-fw: add missing extract dependencies
  package/nvidia-tegra23/nvidia-tegra32-binaries: add missing extract
    dependencies
  package/perl: add missing extract dependencies
  package/rust-bin: add missing extract dependencies

 arch/arch.mk.xtensa                           |  2 ++
 boot/uboot/uboot.mk                           |  1 +
 linux/linux.mk                                |  1 +
 package/android-tools/android-tools.mk        |  2 ++
 package/binutils/binutils.mk                  |  2 ++
 package/cargo/cargo.mk                        |  2 ++
 package/gcc/gcc-final/gcc-final.mk            |  1 +
 package/gcc/gcc-initial/gcc-initial.mk        |  1 +
 package/gdb/gdb.mk                            |  2 ++
 package/gettext-tiny/gettext-tiny.mk          |  1 +
 package/murata-cyw-fw/murata-cyw-fw.mk        |  3 +++
 .../nvidia-tegra23-binaries.mk                |  1 +
 package/perl/perl.mk                          |  1 +
 package/pkg-generic.mk                        | 12 ++--------
 package/pkg-utils.mk                          | 24 ++++++++++++++-----
 package/rust-bin/rust-bin.mk                  |  4 +++-
 support/dependencies/check-host-lzip.mk       |  1 -
 support/dependencies/check-host-xzcat.mk      |  1 -
 18 files changed, 43 insertions(+), 19 deletions(-)

-- 
2.23.0

[Buildroot] [PATCH 01/12] package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency

[Thomas Petazzoni]

The extractor-dependency macro returns which system-provided tools are
needed to be able to extract the archive passed as argument. The
result of this macro is added to DL_TOOLS_DEPENDENCIES so that the
logic in support/dependencies/ verifies that the necessary tools are
provided by the system.

However, we are going to add another macro, extractor-pkg-dependency,
which says which Buildroot packages are needed to extract the archive
passed as argument. Indeed, for those archive types, if the extractor
is not provided system-wide, we build it as a host Buildroot package.

To clarify the distinction between the upcoming
extractor-pkg-dependency and existing extractor-dependency, we rename
the latter to extractor-system-dependency.

We take this opportunity to extend the documentation of this macro.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-generic.mk |  2 +-
 package/pkg-utils.mk   | 14 +++++++++-----
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 6d598cfd9d..5ad8f1d894 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -1098,7 +1098,7 @@ else ifeq ($$($(2)_SITE_METHOD),cvs)
 DL_TOOLS_DEPENDENCIES += cvs
 endif # SITE_METHOD
 
-DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
+DL_TOOLS_DEPENDENCIES += $$(call extractor-system-dependency,$$($(2)_SOURCE))
 
 # Ensure all virtual targets are PHONY. Listed alphabetically.
 .PHONY:	$(1) \
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index d38971caf6..0f916d81d8 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -46,11 +46,15 @@ INFLATE.tar  = cat
 # suitable-extractor(filename): returns extractor based on suffix
 suitable-extractor = $(INFLATE$(suffix $(1)))
 
-# extractor-dependency(filename): returns extractor for 'filename' if the
-# extractor is a dependency. If we build the extractor return nothing.
-# $(firstword) is used here because the extractor can have arguments, like
-# ZCAT="gzip -d -c", and to check for the dependency we only want 'gzip'.
-extractor-dependency = $(firstword $(INFLATE$(filter-out \
+# extractor-system-dependency(filename): returns the name of the tool
+# needed to extract 'filename', and is meant to be used with
+# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
+# provided by the system Buildroot runs on.
+#
+# $(firstword) is used here because the extractor can have arguments,
+# like ZCAT="gzip -d -c", and to check for the dependency we only want
+# 'gzip'.
+extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
 	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
 
 # check-deprecated-variable -- throw an error on deprecated variables
-- 
2.23.0

[Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro

[Thomas Petazzoni]

To extract some archive types, if the extracting tool is not available
on the system where Buildroot runs on, we build it using a Buildroot
host package.

Such dependencies are currently explicitly handled by the
inner-generic-package macro, but in fact we also need to handle them
in all places where the "suitable-extractor" macro is invoked, and
some packages invoke it directly. Otherwise, such packages may be
missing a dependency to the appropriate host Buildroot package
building the extracting tool they need. An example is gettext-tiny,
whose source code is a gzip-compressed tarball, but in addition
manually extracts a xz-compressed tarball.

This extractor-pkg-dependency macro will be used in follow-up commits
to ensure all the packages that use suitable-extractor properly add
the correct dependencies.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-utils.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 0f916d81d8..35a441e0aa 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -46,6 +46,14 @@ INFLATE.tar  = cat
 # suitable-extractor(filename): returns extractor based on suffix
 suitable-extractor = $(INFLATE$(suffix $(1)))
 
+EXTRACTOR_PKG_DEPENDENCY.lzma = $(BR2_XZCAT_HOST_DEPENDENCY)
+EXTRACTOR_PKG_DEPENDENCY.xz   = $(BR2_XZCAT_HOST_DEPENDENCY)
+EXTRACTOR_PKG_DEPENDENCY.lz   = $(BR2_LZIP_HOST_DEPENDENCY)
+
+# extractor-pkg-dependency(filename): returns a Buildroot package
+# dependency needed to extract file based on suffix
+extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
+
 # extractor-system-dependency(filename): returns the name of the tool
 # needed to extract 'filename', and is meant to be used with
 # DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
-- 
2.23.0

[Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency

[Thomas Petazzoni]

Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
we can use them to implement extractor-system-dependency: if for a
given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
variable is empty, then it means we need the corresponding extractor
tool to be provided by the system.

Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
longer used, so we can drop it from support/dependencies/.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-utils.mk                     | 4 ++--
 support/dependencies/check-host-lzip.mk  | 1 -
 support/dependencies/check-host-xzcat.mk | 1 -
 3 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 35a441e0aa..d324934dba 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
 # $(firstword) is used here because the extractor can have arguments,
 # like ZCAT="gzip -d -c", and to check for the dependency we only want
 # 'gzip'.
-extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
-	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
+extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
+	$(firstword $(INFLATE$(suffix $(1)))))
 
 # check-deprecated-variable -- throw an error on deprecated variables
 # example:
diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
index cdd784058c..98a4f42388 100644
--- a/support/dependencies/check-host-lzip.mk
+++ b/support/dependencies/check-host-lzip.mk
@@ -1,5 +1,4 @@
 ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
 BR2_LZIP_HOST_DEPENDENCY = host-lzip
-EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
 LZCAT = $(HOST_DIR)/bin/lzip -d -c
 endif
diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
index e5d72f95dd..9a78a8f38a 100644
--- a/support/dependencies/check-host-xzcat.mk
+++ b/support/dependencies/check-host-xzcat.mk
@@ -3,6 +3,5 @@
 
 ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
 BR2_XZCAT_HOST_DEPENDENCY = host-xz
-EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
 XZCAT = $(HOST_DIR)/bin/xzcat
 endif
-- 
2.23.0

[Buildroot] [PATCH 04/12] package/pkg-generic.mk: use extractor-pkg-dependency macro

[Thomas Petazzoni]

Instead of manually calculating the EXTRACT_DEPENDENCIES value based
on the archive extension, let's use the newly introduced
extractor-pkg-dependency macro.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-generic.mk | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 5ad8f1d894..45a9a133bb 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -644,15 +644,7 @@ $(2)_EXTRACT_DEPENDENCIES += $$(BR2_TAR_HOST_DEPENDENCY)
 endif
 
 ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
-ifneq ($$(filter .xz .lzma,$$(suffix $$($(2)_SOURCE))),)
-$(2)_EXTRACT_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)
-endif
-endif
-
-ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
-ifneq ($$(filter .lz,$$(suffix $$($(2)_SOURCE))),)
-$(2)_EXTRACT_DEPENDENCIES += $$(BR2_LZIP_HOST_DEPENDENCY)
-endif
+$(2)_EXTRACT_DEPENDENCIES += $$(call extractor-pkg-dependency,$$($(2)_SOURCE))
 endif
 
 ifeq ($$(BR2_CCACHE),y)
-- 
2.23.0

[Buildroot] [PATCH 05/12] arch/arch.mk.xtensa: fix usage of arch-xtensa-overlay-extract

[Thomas Petazzoni]

arch-xtensa-overlay-extract was used in gcc, linux, uboot, binutils
and gdb without properly accouting for the package dependencies that
may be needed to extract the overlay tarball.

This commit fixes that by introducing an
ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES variable in arch.mk.xtensa,
and then use it where appropriate in gcc, linux, uboot, binutils and
gdb.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
Note: if this is preferred, we could avoid adding the
ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES variable, and directly call
extractor-pkg-dependency from each of gcc, linux, uboot, binutils and
gdb. Having an intermediate variable felt a bit simpler/cleaner, but I
wouldn't mind changing it.
---
 arch/arch.mk.xtensa                    | 2 ++
 boot/uboot/uboot.mk                    | 1 +
 linux/linux.mk                         | 1 +
 package/binutils/binutils.mk           | 2 ++
 package/gcc/gcc-final/gcc-final.mk     | 1 +
 package/gcc/gcc-initial/gcc-initial.mk | 1 +
 package/gdb/gdb.mk                     | 2 ++
 7 files changed, 10 insertions(+)

diff --git a/arch/arch.mk.xtensa b/arch/arch.mk.xtensa
index fd410f6bfa..c0f83c27d2 100644
--- a/arch/arch.mk.xtensa
+++ b/arch/arch.mk.xtensa
@@ -19,6 +19,8 @@ else
 ARCH_XTENSA_OVERLAY_FILE = $(BR_ARCH_XTENSA_OVERLAY_FILE)
 endif
 
+ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(ARCH_XTENSA_OVERLAY_FILE))
+
 ################################################################################
 # arch-xtensa-overlay-extract -- extract an extensa overlay
 #
diff --git a/boot/uboot/uboot.mk b/boot/uboot/uboot.mk
index 0ebda6588c..eaf8d4a30a 100644
--- a/boot/uboot/uboot.mk
+++ b/boot/uboot/uboot.mk
@@ -185,6 +185,7 @@ ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
 define UBOOT_XTENSA_OVERLAY_EXTRACT
 	$(call arch-xtensa-overlay-extract,$(@D),u-boot)
 endef
+UBOOT_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
 UBOOT_POST_EXTRACT_HOOKS += UBOOT_XTENSA_OVERLAY_EXTRACT
 UBOOT_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
 endif
diff --git a/linux/linux.mk b/linux/linux.mk
index 762f8868b1..325a15f74c 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -120,6 +120,7 @@ ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
 define LINUX_XTENSA_OVERLAY_EXTRACT
 	$(call arch-xtensa-overlay-extract,$(@D),linux)
 endef
+LINUX_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
 LINUX_POST_EXTRACT_HOOKS += LINUX_XTENSA_OVERLAY_EXTRACT
 LINUX_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
 endif
diff --git a/package/binutils/binutils.mk b/package/binutils/binutils.mk
index a19d6940f7..0af83e92d0 100644
--- a/package/binutils/binutils.mk
+++ b/package/binutils/binutils.mk
@@ -120,8 +120,10 @@ ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
 define BINUTILS_XTENSA_OVERLAY_EXTRACT
 	$(call arch-xtensa-overlay-extract,$(@D),binutils)
 endef
+BINUTILS_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
 BINUTILS_POST_EXTRACT_HOOKS += BINUTILS_XTENSA_OVERLAY_EXTRACT
 BINUTILS_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
+HOST_BINUTILS_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
 HOST_BINUTILS_POST_EXTRACT_HOOKS += BINUTILS_XTENSA_OVERLAY_EXTRACT
 HOST_BINUTILS_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
 endif
diff --git a/package/gcc/gcc-final/gcc-final.mk b/package/gcc/gcc-final/gcc-final.mk
index 24d034b720..57a7e6d8a2 100644
--- a/package/gcc/gcc-final/gcc-final.mk
+++ b/package/gcc/gcc-final/gcc-final.mk
@@ -17,6 +17,7 @@ HOST_GCC_FINAL_DEPENDENCIES = \
 HOST_GCC_FINAL_EXCLUDES = $(HOST_GCC_EXCLUDES)
 
 ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
+HOST_GCC_FINAL_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
 HOST_GCC_FINAL_POST_EXTRACT_HOOKS += HOST_GCC_XTENSA_OVERLAY_EXTRACT
 HOST_GCC_FINAL_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
 endif
diff --git a/package/gcc/gcc-initial/gcc-initial.mk b/package/gcc/gcc-initial/gcc-initial.mk
index f5bc6c5b77..36460d3712 100644
--- a/package/gcc/gcc-initial/gcc-initial.mk
+++ b/package/gcc/gcc-initial/gcc-initial.mk
@@ -18,6 +18,7 @@ HOST_GCC_INITIAL_DEPENDENCIES = $(HOST_GCC_COMMON_DEPENDENCIES)
 HOST_GCC_INITIAL_EXCLUDES = $(HOST_GCC_EXCLUDES)
 
 ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
+HOST_GCC_INITIAL_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
 HOST_GCC_INITIAL_POST_EXTRACT_HOOKS += HOST_GCC_XTENSA_OVERLAY_EXTRACT
 HOST_GCC_INITIAL_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
 endif
diff --git a/package/gdb/gdb.mk b/package/gdb/gdb.mk
index 46d745a897..b1bfca1535 100644
--- a/package/gdb/gdb.mk
+++ b/package/gdb/gdb.mk
@@ -48,8 +48,10 @@ ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
 define GDB_XTENSA_OVERLAY_EXTRACT
 	$(call arch-xtensa-overlay-extract,$(@D),gdb)
 endef
+GDB_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
 GDB_POST_EXTRACT_HOOKS += GDB_XTENSA_OVERLAY_EXTRACT
 GDB_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
+HOST_GDB_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
 HOST_GDB_POST_EXTRACT_HOOKS += GDB_XTENSA_OVERLAY_EXTRACT
 HOST_GDB_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
 endif
-- 
2.23.0

[Buildroot] [PATCH 06/12] package/android-tools: add missing extract dependencies

[Thomas Petazzoni]

android-tools uses suitable-extractor to manually extract a tarball,
so it should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/android-tools/android-tools.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/android-tools/android-tools.mk b/package/android-tools/android-tools.mk
index 3d5c76a322..fda58e13bb 100644
--- a/package/android-tools/android-tools.mk
+++ b/package/android-tools/android-tools.mk
@@ -22,7 +22,9 @@ define ANDROID_TOOLS_DEBIAN_EXTRACT
 endef
 
 HOST_ANDROID_TOOLS_POST_EXTRACT_HOOKS += ANDROID_TOOLS_DEBIAN_EXTRACT
+HOST_ANDROID_TOOLS_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(ANDROID_TOOLS_EXTRA_DOWNLOADS))
 ANDROID_TOOLS_POST_EXTRACT_HOOKS += ANDROID_TOOLS_DEBIAN_EXTRACT
+ANDROID_TOOLS_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(ANDROID_TOOLS_EXTRA_DOWNLOADS))
 
 # Apply the Debian patches before applying the Buildroot patches
 define ANDROID_TOOLS_DEBIAN_PATCH
-- 
2.23.0

[Buildroot] [PATCH 07/12] package/cargo: add missing extract dependencies

[Thomas Petazzoni]

cargo uses suitable-extractor to manually extract a tarball, so it
should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/cargo/cargo.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/cargo/cargo.mk b/package/cargo/cargo.mk
index a387281b4c..adb7bc282e 100644
--- a/package/cargo/cargo.mk
+++ b/package/cargo/cargo.mk
@@ -42,6 +42,7 @@ define HOST_CARGO_EXTRACT_DEPS
 endef
 
 HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_DEPS
+HOST_CARGO_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(CARGO_DEPS_SOURCE))
 
 define HOST_CARGO_EXTRACT_INSTALLER
 	@mkdir -p $(@D)/src/rust-installer
@@ -51,6 +52,7 @@ define HOST_CARGO_EXTRACT_INSTALLER
 endef
 
 HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_INSTALLER
+HOST_CARGO_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(CARGO_INSTALLER_SOURCE))
 
 define HOST_CARGO_SETUP_DEPS
 	mkdir -p $(@D)/.cargo
-- 
2.23.0

[Buildroot] [PATCH 08/12] package/gettext-tiny: add missing extract dependencies

[Thomas Petazzoni]

gettext-tiny uses suitable-extractor to manually extract a tarball, so
it should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.

Fixes:

  http://autobuild.buildroot.net/results/83c6d47c06334bef27791a59bdd491b1de124c49/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/gettext-tiny/gettext-tiny.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/gettext-tiny/gettext-tiny.mk b/package/gettext-tiny/gettext-tiny.mk
index 6ee6269e09..403f974b9d 100644
--- a/package/gettext-tiny/gettext-tiny.mk
+++ b/package/gettext-tiny/gettext-tiny.mk
@@ -39,6 +39,7 @@ define HOST_GETTEXT_TINY_EXTRACT_GNU_GETTEXT
 		$(TAR) --strip-components=1 -C $(@D)/gettext-gnu $(TAR_OPTIONS) -
 endef
 HOST_GETTEXT_TINY_POST_EXTRACT_HOOKS += HOST_GETTEXT_TINY_EXTRACT_GNU_GETTEXT
+HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(GETTEXT_GNU_SOURCE))
 
 define HOST_GETTEXT_TINY_COPY_EXTRA_FILES
 	$(foreach f,$(GETTEXT_TINY_EXTRA_GETTEXT_FILES),\
-- 
2.23.0

[Buildroot] [PATCH 09/12] package/murata-cyw-fw: add missing extract dependencies

[Thomas Petazzoni]

murata-cyw-fw uses suitable-extractor to manually extract a tarball,
so it should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/murata-cyw-fw/murata-cyw-fw.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/package/murata-cyw-fw/murata-cyw-fw.mk b/package/murata-cyw-fw/murata-cyw-fw.mk
index b99168b875..ea4f3fcc34 100644
--- a/package/murata-cyw-fw/murata-cyw-fw.mk
+++ b/package/murata-cyw-fw/murata-cyw-fw.mk
@@ -22,6 +22,9 @@ define MURATA_CYW_FW_EXTRACT_NVRAM_PATCH
 	)
 endef
 MURATA_CYW_FW_POST_EXTRACT_HOOKS += MURATA_CYW_FW_EXTRACT_NVRAM_PATCH
+MURATA_CYW_FW_EXTRACT_DEPENDENCIES += \
+	$(foreach tar,$(notdir $(MURATA_CYW_FW_EXTRA_DOWNLOADS)),\
+		$(call extractor-pkg-dependency,$(tar)))
 
 MURATA_CYW_FW_FILES_$(BR2_PACKAGE_MURATA_CYW_FW_CYW43012) += \
 	brcmfmac43012-sdio.bin \
-- 
2.23.0

[Buildroot] [PATCH 10/12] package/nvidia-tegra23/nvidia-tegra32-binaries: add missing extract dependencies

[Thomas Petazzoni]

nvidia-tegra23-binaries uses suitable-extractor to manually extract a
tarball, so it should have the appropriate package dependencies added
to <pkg>_EXTRACT_DEPENDENCIES.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 .../nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk           | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/nvidia-tegra23/nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk b/package/nvidia-tegra23/nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk
index 9b153b0250..1785856847 100644
--- a/package/nvidia-tegra23/nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk
+++ b/package/nvidia-tegra23/nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk
@@ -44,6 +44,7 @@ define NVIDIA_TEGRA23_BINARIES_EXTRACT_FURTHER
 	$(TAR) --strip-components=0 -C $(@D)/nv_tegra/nv_sample_apps/nvgstapps/ $(TAR_OPTIONS) -
 endef
 NVIDIA_TEGRA23_BINARIES_POST_EXTRACT_HOOKS += NVIDIA_TEGRA23_BINARIES_EXTRACT_FURTHER
+NVIDIA_TEGRA23_BINARIES_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,nvidia_drivers.tbz2)
 
 define NVIDIA_TEGRA23_BINARIES_INSTALL_LIBS
 	mkdir -p $(1)/usr/lib
-- 
2.23.0

[Buildroot] [PATCH 11/12] package/perl: add missing extract dependencies

[Thomas Petazzoni]

perl uses suitable-extractor to manually extract a tarball, so it
should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/perl/perl.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/perl/perl.mk b/package/perl/perl.mk
index 0ab81ddb34..3b8b9fb0c4 100644
--- a/package/perl/perl.mk
+++ b/package/perl/perl.mk
@@ -27,6 +27,7 @@ define PERL_CROSS_EXTRACT
 	$(call suitable-extractor,$(PERL_CROSS_SOURCE)) $(PERL_DL_DIR)/$(PERL_CROSS_SOURCE) | \
 	$(TAR) --strip-components=1 -C $(@D) $(TAR_OPTIONS) -
 endef
+PERL_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(PERL_CROSS_SOURCE))
 PERL_POST_EXTRACT_HOOKS += PERL_CROSS_EXTRACT
 
 # Even though perl is not an autotools-package, it uses config.sub and
-- 
2.23.0

[Buildroot] [PATCH 12/12] package/rust-bin: add missing extract dependencies

[Thomas Petazzoni]

rust-bin uses suitable-extractor to manually extract a tarball, so it
should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/rust-bin/rust-bin.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/rust-bin/rust-bin.mk b/package/rust-bin/rust-bin.mk
index a374215e20..526c45a45e 100644
--- a/package/rust-bin/rust-bin.mk
+++ b/package/rust-bin/rust-bin.mk
@@ -32,7 +32,9 @@ define HOST_RUST_BIN_LIBSTD_EXTRACT
 	cd $(@D)/rustc/lib/rustlib/$(RUSTC_HOST_NAME)/lib; \
 		ln -sf ../../../../../std/$(HOST_RUST_BIN_LIBSTD_HOST_PREFIX)/lib/rustlib/$(RUSTC_HOST_NAME)/lib/* .
 endef
-
+HOST_RUST_BIN_EXTRACT_DEPENDENCIES += \
+	$(foreach f,$(HOST_RUST_BIN_EXTRA_DOWNLOADS),\
+		$(call extractor-pkg-dependency,$(f)))
 HOST_RUST_BIN_POST_EXTRACT_HOOKS += HOST_RUST_BIN_LIBSTD_EXTRACT
 
 HOST_RUST_BIN_INSTALL_OPTS = \
-- 
2.23.0

[Buildroot] [PATCH 05/12] arch/arch.mk.xtensa: fix usage of arch-xtensa-overlay-extract

[Max Filippov]

On Tue, Dec 10, 2019 at 2:00 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> arch-xtensa-overlay-extract was used in gcc, linux, uboot, binutils
> and gdb without properly accouting for the package dependencies that
> may be needed to extract the overlay tarball.
>
> This commit fixes that by introducing an
> ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES variable in arch.mk.xtensa,
> and then use it where appropriate in gcc, linux, uboot, binutils and
> gdb.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> Note: if this is preferred, we could avoid adding the
> ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES variable, and directly call
> extractor-pkg-dependency from each of gcc, linux, uboot, binutils and
> gdb. Having an intermediate variable felt a bit simpler/cleaner, but I
> wouldn't mind changing it.
> ---
>  arch/arch.mk.xtensa                    | 2 ++
>  boot/uboot/uboot.mk                    | 1 +
>  linux/linux.mk                         | 1 +
>  package/binutils/binutils.mk           | 2 ++
>  package/gcc/gcc-final/gcc-final.mk     | 1 +
>  package/gcc/gcc-initial/gcc-initial.mk | 1 +
>  package/gdb/gdb.mk                     | 2 ++
>  7 files changed, 10 insertions(+)

Reviewed-by: Max Filippov <jcmvbkbc@gmail.com>

-- 
Thanks.
-- Max

[Buildroot] [PATCH 01/12] package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency

[Romain Naour]

Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> The extractor-dependency macro returns which system-provided tools are
> needed to be able to extract the archive passed as argument. The
> result of this macro is added to DL_TOOLS_DEPENDENCIES so that the
> logic in support/dependencies/ verifies that the necessary tools are
> provided by the system.
> 
> However, we are going to add another macro, extractor-pkg-dependency,
> which says which Buildroot packages are needed to extract the archive
> passed as argument. Indeed, for those archive types, if the extractor
> is not provided system-wide, we build it as a host Buildroot package.
> 
> To clarify the distinction between the upcoming
> extractor-pkg-dependency and existing extractor-dependency, we rename
> the latter to extractor-system-dependency.
> 
> We take this opportunity to extend the documentation of this macro.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Reviewed-by: Romain Naour <romain.naour@smile.fr>

Best regards,
Romain


> ---
>  package/pkg-generic.mk |  2 +-
>  package/pkg-utils.mk   | 14 +++++++++-----
>  2 files changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 6d598cfd9d..5ad8f1d894 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -1098,7 +1098,7 @@ else ifeq ($$($(2)_SITE_METHOD),cvs)
>  DL_TOOLS_DEPENDENCIES += cvs
>  endif # SITE_METHOD
>  
> -DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
> +DL_TOOLS_DEPENDENCIES += $$(call extractor-system-dependency,$$($(2)_SOURCE))
>  
>  # Ensure all virtual targets are PHONY. Listed alphabetically.
>  .PHONY:	$(1) \
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index d38971caf6..0f916d81d8 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -46,11 +46,15 @@ INFLATE.tar  = cat
>  # suitable-extractor(filename): returns extractor based on suffix
>  suitable-extractor = $(INFLATE$(suffix $(1)))
>  
> -# extractor-dependency(filename): returns extractor for 'filename' if the
> -# extractor is a dependency. If we build the extractor return nothing.
> -# $(firstword) is used here because the extractor can have arguments, like
> -# ZCAT="gzip -d -c", and to check for the dependency we only want 'gzip'.
> -extractor-dependency = $(firstword $(INFLATE$(filter-out \
> +# extractor-system-dependency(filename): returns the name of the tool
> +# needed to extract 'filename', and is meant to be used with
> +# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
> +# provided by the system Buildroot runs on.
> +#
> +# $(firstword) is used here because the extractor can have arguments,
> +# like ZCAT="gzip -d -c", and to check for the dependency we only want
> +# 'gzip'.
> +extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
>  	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
>  
>  # check-deprecated-variable -- throw an error on deprecated variables
> 

[Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro

[Romain Naour]

Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> To extract some archive types, if the extracting tool is not available
> on the system where Buildroot runs on, we build it using a Buildroot
> host package.
> 
> Such dependencies are currently explicitly handled by the
> inner-generic-package macro, but in fact we also need to handle them
> in all places where the "suitable-extractor" macro is invoked, and
> some packages invoke it directly. Otherwise, such packages may be
> missing a dependency to the appropriate host Buildroot package
> building the extracting tool they need. An example is gettext-tiny,
> whose source code is a gzip-compressed tarball, but in addition
> manually extracts a xz-compressed tarball.
> 
> This extractor-pkg-dependency macro will be used in follow-up commits
> to ensure all the packages that use suitable-extractor properly add
> the correct dependencies.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-utils.mk | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index 0f916d81d8..35a441e0aa 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -46,6 +46,14 @@ INFLATE.tar  = cat
>  # suitable-extractor(filename): returns extractor based on suffix
>  suitable-extractor = $(INFLATE$(suffix $(1)))
>  

What about gzip ?

EXTRACTOR_PKG_DEPENDENCY.gz   = $(BR2_GZIP_HOST_DEPENDENCY)
EXTRACTOR_PKG_DEPENDENCY.tgz  = $(BR2_GZIP_HOST_DEPENDENCY)

Best regards,
Romain


> +EXTRACTOR_PKG_DEPENDENCY.lzma = $(BR2_XZCAT_HOST_DEPENDENCY)
> +EXTRACTOR_PKG_DEPENDENCY.xz   = $(BR2_XZCAT_HOST_DEPENDENCY)
> +EXTRACTOR_PKG_DEPENDENCY.lz   = $(BR2_LZIP_HOST_DEPENDENCY)
> +
> +# extractor-pkg-dependency(filename): returns a Buildroot package
> +# dependency needed to extract file based on suffix
> +extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
> +
>  # extractor-system-dependency(filename): returns the name of the tool
>  # needed to extract 'filename', and is meant to be used with
>  # DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
> 

[Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency

[Romain Naour]

Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
> we can use them to implement extractor-system-dependency: if for a
> given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
> variable is empty, then it means we need the corresponding extractor
> tool to be provided by the system.
> 
> Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
> longer used, so we can drop it from support/dependencies/.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-utils.mk                     | 4 ++--
>  support/dependencies/check-host-lzip.mk  | 1 -
>  support/dependencies/check-host-xzcat.mk | 1 -
>  3 files changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index 35a441e0aa..d324934dba 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
>  # $(firstword) is used here because the extractor can have arguments,
>  # like ZCAT="gzip -d -c", and to check for the dependency we only want
>  # 'gzip'.
> -extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
> -	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
> +extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\

I would add a "." after EXTRACTOR_PKG_DEPENDENCY because we want to use
"EXTRACTOR_PKG_DEPENDENCY.$(suffix $(1))"

Best regards,
Romain


> +	$(firstword $(INFLATE$(suffix $(1)))))
>  
>  # check-deprecated-variable -- throw an error on deprecated variables
>  # example:
> diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
> index cdd784058c..98a4f42388 100644
> --- a/support/dependencies/check-host-lzip.mk
> +++ b/support/dependencies/check-host-lzip.mk
> @@ -1,5 +1,4 @@
>  ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
>  BR2_LZIP_HOST_DEPENDENCY = host-lzip
> -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
>  LZCAT = $(HOST_DIR)/bin/lzip -d -c
>  endif
> diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
> index e5d72f95dd..9a78a8f38a 100644
> --- a/support/dependencies/check-host-xzcat.mk
> +++ b/support/dependencies/check-host-xzcat.mk
> @@ -3,6 +3,5 @@
>  
>  ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
>  BR2_XZCAT_HOST_DEPENDENCY = host-xz
> -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
>  XZCAT = $(HOST_DIR)/bin/xzcat
>  endif
> 

[Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro

[Thomas Petazzoni]

On Sat, 14 Dec 2019 22:20:21 +0100
Romain Naour <romain.naour@smile.fr> wrote:

> > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > index 0f916d81d8..35a441e0aa 100644
> > --- a/package/pkg-utils.mk
> > +++ b/package/pkg-utils.mk
> > @@ -46,6 +46,14 @@ INFLATE.tar  = cat
> >  # suitable-extractor(filename): returns extractor based on suffix
> >  suitable-extractor = $(INFLATE$(suffix $(1)))
> >    
> 
> What about gzip ?
> 
> EXTRACTOR_PKG_DEPENDENCY.gz   = $(BR2_GZIP_HOST_DEPENDENCY)
> EXTRACTOR_PKG_DEPENDENCY.tgz  = $(BR2_GZIP_HOST_DEPENDENCY)

Because oddly enough, we are today not taking into account  the
dependency on gzip when extracting tarballs.

We do take it into account only in:

ifneq ($$(filter cvs git svn,$$($(2)_SITE_METHOD)),)
$(2)_DOWNLOAD_DEPENDENCIES += \
        $(BR2_GZIP_HOST_DEPENDENCY) \
        $(BR2_TAR_HOST_DEPENDENCY)
endif

i.e, when we fetch from a version control system, and create our own
tarball locally.

But if we download directly from http/ftp a .tar.gz or .tgz, we don't
rely on BR2_GZIP_HOST_DEPENDENCY, and simply assume we can use the one
from the host machine.

This should be fixed, I agree, but it's already broken today I believe.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency

[Thomas Petazzoni]

On Sat, 14 Dec 2019 22:37:14 +0100
Romain Naour <romain.naour@smile.fr> wrote:

> > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > index 35a441e0aa..d324934dba 100644
> > --- a/package/pkg-utils.mk
> > +++ b/package/pkg-utils.mk
> > @@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
> >  # $(firstword) is used here because the extractor can have arguments,
> >  # like ZCAT="gzip -d -c", and to check for the dependency we only want
> >  # 'gzip'.
> > -extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
> > -	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
> > +extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\  
> 
> I would add a "." after EXTRACTOR_PKG_DEPENDENCY because we want to use
> "EXTRACTOR_PKG_DEPENDENCY.$(suffix $(1))"

You're absolutely right. Now I wonder how it could have worked with my
code. I did quite a bit of testing, verifying the values of a number of
internal variables in various situations to make sure it was all
working well.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency

[Yann E. MORIN]

Romain, All,

On 2019-12-14 22:37 +0100, Romain Naour spake thusly:
> Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> > Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
> > we can use them to implement extractor-system-dependency: if for a
> > given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
> > variable is empty, then it means we need the corresponding extractor
> > tool to be provided by the system.
> > 
> > Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
> > longer used, so we can drop it from support/dependencies/.
> > 
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > ---
> >  package/pkg-utils.mk                     | 4 ++--
> >  support/dependencies/check-host-lzip.mk  | 1 -
> >  support/dependencies/check-host-xzcat.mk | 1 -
> >  3 files changed, 2 insertions(+), 4 deletions(-)
> > 
> > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > index 35a441e0aa..d324934dba 100644
> > --- a/package/pkg-utils.mk
> > +++ b/package/pkg-utils.mk
> > @@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
> >  # $(firstword) is used here because the extractor can have arguments,
> >  # like ZCAT="gzip -d -c", and to check for the dependency we only want
> >  # 'gzip'.
> > -extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
> > -	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
> > +extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
> 
> I would add a "." after EXTRACTOR_PKG_DEPENDENCY because we want to use
> "EXTRACTOR_PKG_DEPENDENCY.$(suffix $(1))"

In a Makefile:
    FOO = a.b
    $(info $(suffix $(FOO)))

would yield:
    .b

so we already have the dot.

Regards,
Yann E. MORIN.

> Best regards,
> Romain
> 
> 
> > +	$(firstword $(INFLATE$(suffix $(1)))))
> >  
> >  # check-deprecated-variable -- throw an error on deprecated variables
> >  # example:
> > diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
> > index cdd784058c..98a4f42388 100644
> > --- a/support/dependencies/check-host-lzip.mk
> > +++ b/support/dependencies/check-host-lzip.mk
> > @@ -1,5 +1,4 @@
> >  ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
> >  BR2_LZIP_HOST_DEPENDENCY = host-lzip
> > -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
> >  LZCAT = $(HOST_DIR)/bin/lzip -d -c
> >  endif
> > diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
> > index e5d72f95dd..9a78a8f38a 100644
> > --- a/support/dependencies/check-host-xzcat.mk
> > +++ b/support/dependencies/check-host-xzcat.mk
> > @@ -3,6 +3,5 @@
> >  
> >  ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
> >  BR2_XZCAT_HOST_DEPENDENCY = host-xz
> > -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
> >  XZCAT = $(HOST_DIR)/bin/xzcat
> >  endif
> > 
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency

[Romain Naour]

Yann, Thomas,

Le 14/12/2019 ? 22:45, Yann E. MORIN a ?crit?:
> Romain, All,
> 
> On 2019-12-14 22:37 +0100, Romain Naour spake thusly:
>> Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
>>> Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
>>> we can use them to implement extractor-system-dependency: if for a
>>> given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
>>> variable is empty, then it means we need the corresponding extractor
>>> tool to be provided by the system.
>>>
>>> Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
>>> longer used, so we can drop it from support/dependencies/.
>>>
>>> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>>> ---
>>>  package/pkg-utils.mk                     | 4 ++--
>>>  support/dependencies/check-host-lzip.mk  | 1 -
>>>  support/dependencies/check-host-xzcat.mk | 1 -
>>>  3 files changed, 2 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
>>> index 35a441e0aa..d324934dba 100644
>>> --- a/package/pkg-utils.mk
>>> +++ b/package/pkg-utils.mk
>>> @@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
>>>  # $(firstword) is used here because the extractor can have arguments,
>>>  # like ZCAT="gzip -d -c", and to check for the dependency we only want
>>>  # 'gzip'.
>>> -extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
>>> -	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
>>> +extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
>>
>> I would add a "." after EXTRACTOR_PKG_DEPENDENCY because we want to use
>> "EXTRACTOR_PKG_DEPENDENCY.$(suffix $(1))"
> 
> In a Makefile:
>     FOO = a.b
>     $(info $(suffix $(FOO)))
> 
> would yield:
>     .b
> 
> so we already have the dot.

Thanks, Indeed we already have the dot.
Sorry for the noise.

Best regards,
Romain


> 
> Regards,
> Yann E. MORIN.
> 
>> Best regards,
>> Romain
>>
>>
>>> +	$(firstword $(INFLATE$(suffix $(1)))))
>>>  
>>>  # check-deprecated-variable -- throw an error on deprecated variables
>>>  # example:
>>> diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
>>> index cdd784058c..98a4f42388 100644
>>> --- a/support/dependencies/check-host-lzip.mk
>>> +++ b/support/dependencies/check-host-lzip.mk
>>> @@ -1,5 +1,4 @@
>>>  ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
>>>  BR2_LZIP_HOST_DEPENDENCY = host-lzip
>>> -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
>>>  LZCAT = $(HOST_DIR)/bin/lzip -d -c
>>>  endif
>>> diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
>>> index e5d72f95dd..9a78a8f38a 100644
>>> --- a/support/dependencies/check-host-xzcat.mk
>>> +++ b/support/dependencies/check-host-xzcat.mk
>>> @@ -3,6 +3,5 @@
>>>  
>>>  ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
>>>  BR2_XZCAT_HOST_DEPENDENCY = host-xz
>>> -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
>>>  XZCAT = $(HOST_DIR)/bin/xzcat
>>>  endif
>>>
>>
> 

[Buildroot] [PATCH 04/12] package/pkg-generic.mk: use extractor-pkg-dependency macro

[Romain Naour]

Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> Instead of manually calculating the EXTRACT_DEPENDENCIES value based
> on the archive extension, let's use the newly introduced
> extractor-pkg-dependency macro.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Reviewed-by: Romain Naour <romain.naour@smile.fr>

Best regards,
Romain

> ---
>  package/pkg-generic.mk | 10 +---------
>  1 file changed, 1 insertion(+), 9 deletions(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 5ad8f1d894..45a9a133bb 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -644,15 +644,7 @@ $(2)_EXTRACT_DEPENDENCIES += $$(BR2_TAR_HOST_DEPENDENCY)
>  endif
>  
>  ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
> -ifneq ($$(filter .xz .lzma,$$(suffix $$($(2)_SOURCE))),)
> -$(2)_EXTRACT_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)
> -endif
> -endif
> -
> -ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
> -ifneq ($$(filter .lz,$$(suffix $$($(2)_SOURCE))),)
> -$(2)_EXTRACT_DEPENDENCIES += $$(BR2_LZIP_HOST_DEPENDENCY)
> -endif
> +$(2)_EXTRACT_DEPENDENCIES += $$(call extractor-pkg-dependency,$$($(2)_SOURCE))
>  endif
>  
>  ifeq ($$(BR2_CCACHE),y)
> 

[Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account

[Yann E. MORIN]

Thomas, All,

On 2019-12-10 22:59 +0100, Thomas Petazzoni spake thusly:
> This series solves that by introducing an extractor-pkg-dependency
> that returns the list of packages that must be added to
> <pkg>_EXTRACT_DEPENDENCIES to extract the archive passed as
> argument. It is then used in package/pkg-generic.mk to replace the
> existing logic to calculate <pkg>_EXTRACT_DEPENDENCIES, and then in
> the several places where suitable-extractor was used to manually
> extract a tarball.

Here is a summarry of our little discussion on IRC:

  - the extract dependencies for the main _SOURCE are automatically
    computed by Buildroot, but not for the _EXTRA_DOWNLOADS, which is
    not very systematic;

  - practically 99% of EXTRA_DOWNLOADS that are compressed, will be
    extracted at build time; the cases for (e.g.) an .xz to be stored
    as-is on the target is expected to be very rare;

  - so we should change the logic in this way (basically):

    $(2)_EXTRACT_DEPENDENCIES = $(foreach URI,$($(2)_SOURCE) $($(2)_EXTRA_DOWNLOADS),$(call extractor-pkg-dependency,$(notdir $(URI))))

Regards,
Yann E. MORIN.

> We also rename extractor-dependency to extractor-system-dependency to
> clarify its difference from extractor-pkg-dependency, and we rework
> the implementation of extractor-system-dependency.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro

[Yann E. MORIN]

Thomas, Romain, All,

On 2019-12-14 22:42 +0100, Thomas Petazzoni spake thusly:
> On Sat, 14 Dec 2019 22:20:21 +0100
> Romain Naour <romain.naour@smile.fr> wrote:
> 
> > > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > > index 0f916d81d8..35a441e0aa 100644
> > > --- a/package/pkg-utils.mk
> > > +++ b/package/pkg-utils.mk
> > > @@ -46,6 +46,14 @@ INFLATE.tar  = cat
> > >  # suitable-extractor(filename): returns extractor based on suffix
> > >  suitable-extractor = $(INFLATE$(suffix $(1)))
> > >    
> > 
> > What about gzip ?
> > 
> > EXTRACTOR_PKG_DEPENDENCY.gz   = $(BR2_GZIP_HOST_DEPENDENCY)
> > EXTRACTOR_PKG_DEPENDENCY.tgz  = $(BR2_GZIP_HOST_DEPENDENCY)
> 
> Because oddly enough, we are today not taking into account  the
> dependency on gzip when extracting tarballs.
> 
> We do take it into account only in:
> 
> ifneq ($$(filter cvs git svn,$$($(2)_SITE_METHOD)),)
> $(2)_DOWNLOAD_DEPENDENCIES += \
>         $(BR2_GZIP_HOST_DEPENDENCY) \
>         $(BR2_TAR_HOST_DEPENDENCY)
> endif
> 
> i.e, when we fetch from a version control system, and create our own
> tarball locally.
> 
> But if we download directly from http/ftp a .tar.gz or .tgz, we don't
> rely on BR2_GZIP_HOST_DEPENDENCY, and simply assume we can use the one
> from the host machine.
> 
> This should be fixed, I agree, but it's already broken today I believe.

In fact no. We need to build our own host-gzip only to create archives,
because some host have pigz instead of gzip, and this creates
non-reproducible archives.

However, decompressing with gzip or pigz is exactly the same; we do not
need our own host-gzip, and we can rely on the system-provided gzip,
whether that be gzip of pigz.

So, the current state of affairs is, AFAICS, correct.

Regards,
Yann E. MORIN.

> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'