[Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring

[Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring

[Thomas Petazzoni]

Hello,

Here is a patch series that implements support for vendoring of Cargo
and Go packages.

As you're aware, many Cargo and Go packages describe their
dependencies in a very language-specific way, and expect such
dependencies to be downloaded by their language-specific tools.

This series builds on top of the work from Patrick Havelange
<patrick.havelange@essensium.com> who worked on the Cargo integration,
but extends it to also cover the case of Go, showing that we have a
solution solving both of these language-specific package managers. For
the Go vendoring, I've also used the indication provided by Christian
Stewart <christian@paral.in>.

Overall, the solution consists in a concept of "download post-process
helper" is introduced in the download infrastructure. These are shell
scripts that can be invoked right after the download is done, but
before the tarball is hash-checked and stored in DL_DIR. The idea is
that such "download post-process helpers" can run the
language-specific vendoring logic to fetch the package
dependencies. Thanks to this, the tarball in DL_DIR for a given
package not only contains the package source code itself, but also the
source code of its Cargo or Go dependencies. The tarball hash covers
the entire source code, the complete tarball is cached in DL_DIR, in
the primary site and backup site, and the build can fully be done
offline.

Such "download post-process helpers" are registered by means of the
<pkg>_DOWNLOAD_POST_PROCESS variable by the appropriate package
infrastructure.

This series is also available at:

  https://github.com/tpetazzoni/buildroot/commits/pkg-mgr-v2

More specifically, the series goes like this:

 - PATCH 1: add the concept of download post-processing in the
   download helper support/download/dl-wrapper

 - PATCH 2: add the <pkg>_DOWNLOAD_POST_PROCESS variable to the
   package infrastructure, which gets passed down to
   support/download/dl-wrapper

 - PATCH 3: add the <pkg>_DL_ENV variable to the package
   infrastructure, which allows to pass extra variables in the
   download wrapper environment. This will be used to pass extra
   variables specific to Cargo/Go vendoring.

 - PATCH 4: add a set of two helper shell functions that will be used
   by the post processing scripts

 - PATCH 5: implements the Go vendoring itself

 - PATCH 6: adds package/tinifier, which is one package that requires
   Go vendoring. I don't particularly need this package, it was just
   added because the other Go packages we have in Buildroot today
   don't need vendoring.

 - PATCH 7: introduces a cargo-package infrastructure, for both target
   and host packages

 - PATCH 8: documents the added cargo-package infrastructure

 - PATCH 9 and 10: converts ripgrep and sentry-cli to the
   cargo-package and host-cargo-package infrastructures respecitively

 - PATCH 11: implements the Cargo vendoring itself. Note that we need
   to bump the ripgrep and sentry-cli package at the same time, as
   with the vendoring their tarball contents change and therefore
   their hash changes

 - PATCH 12: rewrite the cargo-package documentation about dependency
   management

Changes since v1:

 - Add -modcacherw when calling Go to get files read/write instead of
   read-only, to make sure they can be deleted by a "make clean". Was
   reported by Ryan Barnett.

 - Change the Cargo infrastructure to "cd" into the build directory
   before calling Cargo, instead of calling Cargo with the full path
   to Cargo.toml as this was causing issues. Was reported by Ryan
   Barnett, investigated and fixed by Patrick Havelange.

 - Call "go mod init" in the go-post-process script if go.mod doesn't
   exist. Was reported by Christian Stewart, who also provided the
   solution.

 - Add a license warning about Go/Cargo packages, as they might have
   vendored dependencies that are not taken into account by Buildroot
   licensing details. Was reported by Ryan Barnett, and Yann E. Morin
   suggested this simple solution.

Best regards,

Thomas


Patrick Havelange (3):
  package/pkg-cargo.mk: introduce the cargo package infrastructure
  docs/manual/cargo: document the cargo-package infrastructure
  package/ripgrep: convert to cargo infrastructure

Thomas Petazzoni (9):
  support/download/dl-wrapper: add concept of download post-processing
  package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable
  package/pkg-download.mk: add <pkg>_DL_ENV variable
  support/download/post-process-helpers: add helper function for post
    process scripts
  support/download/go-post-process: implement Go vendoring support
  package/tinifier: new package
  package/sentry-cli: convert to host-cargo-package infrastructure
  support/download/cargo-post-process, package/pkg-cargo.mk: enable
    vendoring for Cargo packages
  docs/manual/adding-packages-cargo.txt: rewrite explanation about
    dependency management

 DEVELOPERS                            |   1 +
 docs/manual/adding-packages-cargo.txt | 106 ++++++++----------
 package/Config.in                     |   1 +
 package/Makefile.in                   |   1 +
 package/pkg-cargo.mk                  | 149 ++++++++++++++++++++++++++
 package/pkg-download.mk               |   5 +-
 package/pkg-golang.mk                 |  20 ++--
 package/ripgrep/ripgrep.hash          |   2 +-
 package/ripgrep/ripgrep.mk            |  30 +-----
 package/sentry-cli/sentry-cli.hash    |   2 +-
 package/sentry-cli/sentry-cli.mk      |  24 +----
 package/tinifier/Config.in            |  10 ++
 package/tinifier/tinifier.hash        |   3 +
 package/tinifier/tinifier.mk          |  13 +++
 support/download/cargo-post-process   |  38 +++++++
 support/download/dl-wrapper           |   9 +-
 support/download/go-post-process      |  35 ++++++
 support/download/post-process-helpers |  30 ++++++
 18 files changed, 358 insertions(+), 121 deletions(-)
 create mode 100644 package/pkg-cargo.mk
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
 create mode 100755 support/download/cargo-post-process
 create mode 100755 support/download/go-post-process
 create mode 100644 support/download/post-process-helpers

-- 
2.29.2

[Buildroot] [PATCH v2 01/12] support/download/dl-wrapper: add concept of download post-processing

[Thomas Petazzoni]

In order to support package managers such as Cargo (Rust) or Go, we
want to run some custom logic after the main download, but before
packing the tarball and checking the hash.

To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then
support/download/<something>-post-process will be called.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/download/dl-wrapper | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 3315bd410e..2d74554213 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -25,7 +25,7 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() {
         r)  recurse="-r";;
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
+        p)  post_process="${OPTARG}";;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -135,6 +136,12 @@ main() {
             continue
         fi
 
+        if [ -n "${post_process}" ] ; then
+                ${OLDPWD}/support/download/${post_process}-post-process \
+                         -o "${tmpf}" \
+                         -n "${raw_base_name}"
+        fi
+
         # cd back to free the temp-dir, so we can remove it later
         cd "${OLDPWD}"
 
-- 
2.29.2

[Buildroot] [PATCH v2 02/12] package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable

[Thomas Petazzoni]

This will allow packages to register than a download post-processing
is needed. Note that this variable is intentionally not documented: it
is an internal variable meant to be set by package infrastructures,
not directly by packages.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-download.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 951d2fb554..175b9d7dba 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -108,6 +108,7 @@ define DOWNLOAD
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
-- 
2.29.2

[Buildroot] [PATCH v2 03/12] package/pkg-download.mk: add <pkg>_DL_ENV variable

[Thomas Petazzoni]

This variable can be used by package to pass extra environment
variables to the download logic. It will be used for the Go/Cargo
vendoring.

The <pkg>_DL_ENV variable is intentionally not documented: at this
point, it is not meant to be used by packages directly, but only by
package infrastructures.

Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-download.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 175b9d7dba..c914d016e2 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -99,7 +99,9 @@ endif
 
 define DOWNLOAD
 	$(Q)mkdir -p $($(2)_DL_DIR)
-	$(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
+	$(Q)$(EXTRA_ENV) \
+		$($(2)_DL_ENV) \
+		flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
 		-c '$($(2)_DL_VERSION)' \
 		-d '$($(2)_DL_DIR)' \
 		-D '$(DL_DIR)' \
-- 
2.29.2

[Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts

[Thomas Petazzoni]

download post process scripts will often need to unpack the source
code tarball, do some operation, and then repack it. In order to help
with this, post-process-helpers provide an unpack() function and a
repack() function.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/download/post-process-helpers | 30 +++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 support/download/post-process-helpers

diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers
new file mode 100644
index 0000000000..bed8df2577
--- /dev/null
+++ b/support/download/post-process-helpers
@@ -0,0 +1,30 @@
+
+unpack() {
+        dest="$1"
+        tarball="$2"
+
+        mkdir ${dest}
+        tar -C ${dest} --strip-components=1 -xf ${tarball}
+}
+
+repack() {
+        src="$1"
+        tarball="$2"
+
+        # Generate the archive, sort with the C locale so that it is reproducible.
+        find "$(basename ${src})" -not -type d -print0 >files.list
+        LC_ALL=C sort -z <files.list >files.list.sorted
+
+        # let's use a fixed hardcoded date to be reproducible
+        date="2020-02-06 01:02:03 +0000"
+
+        # Create GNU-format tarballs, since that's the format of the tarballs on
+        # sources.buildroot.org and used in the *.hash files
+        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
+            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
+        gzip -6 -n <new.tar >new.tar.gz
+        mv "${tarball}" "${tarball}".old
+        mv new.tar.gz "${tarball}"
+        rm "${tarball}".old
+        rm -rf ${src}
+}
-- 
2.29.2

[Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

[Thomas Petazzoni]

This commit introduces the download post-process script
support/download/go-post-process, and hooks it into the Go package
infrastructure.

The -modcacherw flag is added to ensure that the Go cache is
read/write, and can be deleted properly upon "make clean".

The <pkg>_LICENSE variable of golang packages is expanded with ",
vendored dependencies licenses probably not listed" as currently for
all packages, the licenses of the vendored dependencies are not taken
into account.

The logic to generate go.mod when not available is moved to the
download post-process helper, as it must be done before "go mod
vendor" is executed. Also, "go mod init" is used instead of manually
crafting go.mod. This was suggested by Christian Stewart
<christian@paral.in>. The Go module name is passed down to
go-post-process using the BR_GOMOD environment variable.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-golang.mk            | 20 ++++++++++--------
 support/download/go-post-process | 35 ++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 9 deletions(-)
 create mode 100755 support/download/go-post-process

diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
index d07242310d..00a74a4d18 100644
--- a/package/pkg-golang.mk
+++ b/package/pkg-golang.mk
@@ -42,12 +42,13 @@ define inner-golang-package
 
 $(2)_BUILD_OPTS += \
 	-ldflags "$$($(2)_LDFLAGS)" \
+	-modcacherw \
 	-tags "$$($(2)_TAGS)" \
 	-trimpath \
 	-p $(PARALLEL_JOBS)
 
 # Target packages need the Go compiler on the host.
-$(2)_DEPENDENCIES += host-go
+$(2)_DOWNLOAD_DEPENDENCIES += host-go
 
 $(2)_BUILD_TARGETS ?= .
 
@@ -72,14 +73,15 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
 # If the go.mod file does not exist, one is written with this root path.
 $(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
 
-# Generate a go.mod file if it doesn't exist. Note: Go is configured
-# to use the "vendor" dir and not make network calls.
-define $(2)_GEN_GOMOD
-	if [ ! -f $$(@D)/go.mod ]; then \
-		printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \
-	fi
-endef
-$(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD
+$(2)_DOWNLOAD_POST_PROCESS = go
+$(2)_DL_ENV = \
+	$(HOST_GO_COMMON_ENV) \
+	GOPROXY=direct \
+	BR_GOMOD=$$($(2)_GOMOD)
+
+# Due to vendoring, it is pretty likely that not all licenses are
+# listed in <pkg>_LICENSE.
+$(2)_LICENSE += , vendored dependencies licenses probably not listed
 
 # Build step. Only define it if not already defined by the package .mk
 # file.
diff --git a/support/download/go-post-process b/support/download/go-post-process
new file mode 100755
index 0000000000..4ca3e9a710
--- /dev/null
+++ b/support/download/go-post-process
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+# Parse our options
+while getopts "n:o:" OPT; do
+        case "${OPT}" in
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
+	exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Go vendoring
+pushd ${base_name} > /dev/null
+
+# Generate go.mod if it doesn't exist
+if [ ! -f go.mod ] && [ -n "${BR_GOMOD}" ]; then
+	go mod init ${BR_GOMOD}
+fi
+
+go mod vendor -v -modcacherw
+popd > /dev/null
+
+repack ${base_name} ${output}
-- 
2.29.2

[Buildroot] [PATCH v2 06/12] package/tinifier: new package

[Thomas Petazzoni]

This is a Go package that needs vendor modules to be downloaded at
build time.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 DEVELOPERS                     |  1 +
 package/Config.in              |  1 +
 package/tinifier/Config.in     | 10 ++++++++++
 package/tinifier/tinifier.hash |  3 +++
 package/tinifier/tinifier.mk   | 13 +++++++++++++
 5 files changed, 28 insertions(+)
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index 9406f30cd5..1a7c22b718 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -2590,6 +2590,7 @@ F:	package/rtc-tools/
 F:	package/sam-ba/
 F:	package/scons/
 F:	package/squashfs/
+F:	package/tinifier/
 F:	package/wayland/
 F:	package/weston/
 F:	support/testing/tests/boot/test_syslinux.py
diff --git a/package/Config.in b/package/Config.in
index dc7139a49a..d5d8caef47 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -298,6 +298,7 @@ comment "Graphic applications"
 	source "package/rrdtool/Config.in"
 	source "package/stellarium/Config.in"
 	source "package/tesseract-ocr/Config.in"
+	source "package/tinifier/Config.in"
 
 comment "Graphic libraries"
 	source "package/cegui/Config.in"
diff --git a/package/tinifier/Config.in b/package/tinifier/Config.in
new file mode 100644
index 0000000000..fbadfe6bd9
--- /dev/null
+++ b/package/tinifier/Config.in
@@ -0,0 +1,10 @@
+config BR2_PACKAGE_TINIFIER
+	bool "tinifier"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	help
+	  CLI tool for images compressing
+
+	  This tool uses tinypng.com API endpoint for compressing your
+	  local jpg/png images (it supports parallel jobs).
+
+	  https://github.com/tarampampam/tinifier
diff --git a/package/tinifier/tinifier.hash b/package/tinifier/tinifier.hash
new file mode 100644
index 0000000000..146700817b
--- /dev/null
+++ b/package/tinifier/tinifier.hash
@@ -0,0 +1,3 @@
+# Locally calculated
+sha256  707a1d9e55aab8c83b65bb10f3ec8c3bc094d77cd576b266f821d9f39133ac3c  tinifier-2.1.0.tar.gz
+sha256  791d8fd993ace44d4d83e2f4820a64d5ad3e37412f029afad46d17a9259de2b6  LICENSE
diff --git a/package/tinifier/tinifier.mk b/package/tinifier/tinifier.mk
new file mode 100644
index 0000000000..b47d265a8e
--- /dev/null
+++ b/package/tinifier/tinifier.mk
@@ -0,0 +1,13 @@
+################################################################################
+#
+# tinifier
+#
+################################################################################
+
+TINIFIER_VERSION = 2.1.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
+TINIFIER_LICENSE = MIT
+TINIFIER_LICENSE_FILES = LICENSE
+TINIFIER_GOMOD = tinifier
+
+$(eval $(golang-package))
-- 
2.29.2

[Buildroot] [PATCH v2 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure

[Thomas Petazzoni]

From: Patrick Havelange <patrick.havelange@essensium.com>

In order to be package agnostic, the install phase is now using cargo
instead of install. TARGET_CONFIGURE_OPTS is now also set when running
cargo in order to support cross compiling C code within cargo.

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
[Thomas: add support for host-cargo-package]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/Makefile.in  |   1 +
 package/pkg-cargo.mk | 140 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 141 insertions(+)
 create mode 100644 package/pkg-cargo.mk

diff --git a/package/Makefile.in b/package/Makefile.in
index 51f5cbce4f..2af123f36d 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -427,3 +427,4 @@ include package/pkg-waf.mk
 include package/pkg-golang.mk
 include package/pkg-meson.mk
 include package/pkg-qmake.mk
+include package/pkg-cargo.mk
diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
new file mode 100644
index 0000000000..058c6756bb
--- /dev/null
+++ b/package/pkg-cargo.mk
@@ -0,0 +1,140 @@
+################################################################################
+# Cargo package infrastructure
+#
+# This file implements an infrastructure that eases development of package
+# .mk files for Cargo packages. It should be used for all packages that use
+# Cargo as their build system.
+#
+# See the Buildroot documentation for details on the usage of this
+# infrastructure
+#
+# In terms of implementation, this Cargo infrastructure requires the .mk file
+# to only specify metadata information about the package: name, version,
+# download URL, etc.
+#
+# We still allow the package .mk file to override what the different steps
+# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined,
+# it is used as the list of commands to perform to build the package,
+# instead of the default Cargo behaviour. The package can also define some
+# post operation hooks.
+#
+################################################################################
+
+################################################################################
+# inner-cargo-package -- defines how the configuration, compilation and
+# installation of a cargo package should be done, implements a few hooks
+# to tune the build process for cargo specifities and calls the generic
+# package infrastructure to generate the necessary make targets
+#
+#  argument 1 is the lowercase package name
+#  argument 2 is the uppercase package name, including a HOST_ prefix
+#             for host packages
+#  argument 3 is the uppercase package name, without the HOST_ prefix
+#             for host packages
+#  argument 4 is the type (target or host)
+################################################################################
+
+define inner-cargo-package
+
+# We need host-rustc to run cargo
+$(2)_DEPENDENCIES += host-rustc
+
+$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+# Note: in all the steps below, we "cd" into the build directory to
+# execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the
+# manifest-path. Indeed while the latter seems to work, it in fact
+# breaks in subtle ways as the way cargo searches for its
+# configuration file is based (among other rules) on the current
+# directory. This means that if cargo is started outside of a package
+# directory, its configuration file will not be taken into account.
+#
+# Also, we pass:
+#  * --offline to prevent cargo from downloading anything
+#  * --locked to force cargo to use the Cargo.lock file, which ensures
+#    that a fixed set of dependency versions is used
+
+#
+# Build step. Only define it if not already defined by the package .mk
+# file.
+#
+ifndef $(2)_BUILD_CMDS
+ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	cd $$(@D) && \
+	$$(TARGET_MAKE_ENV) \
+		$$(TARGET_CONFIGURE_OPTS) \
+		$$($(2)_CARGO_ENV) \
+		cargo build \
+			--offline \
+			--target $$(RUSTC_TARGET_NAME) \
+			$$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
+			--manifest-path Cargo.toml \
+			--locked \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+else # ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	cd $$(@D) && \
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \
+		cargo build \
+			--offline \
+			--release \
+			--manifest-path Cargo.toml \
+			--locked \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+endif # ifeq ($(4),target)
+endif # ifndef $(2)_BUILD_CMDS
+
+#
+# Target installation step. Only define it if not already defined by
+# the package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+	cd $$(@D) && \
+	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
+		cargo install \
+			--target $$(RUSTC_TARGET_NAME) \
+			--offline \
+			--root $$(TARGET_DIR)/usr/ \
+			--bins \
+			--path ./ \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+	cd $$(@D) && \
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \
+		cargo install \
+			--offline \
+			--root $$(HOST_DIR) \
+			--bins \
+			--path ./ \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+# Call the generic package infrastructure to generate the necessary
+# make targets
+$(call inner-generic-package,$(1),$(2),$(3),$(4))
+
+endef
+
+################################################################################
+# cargo-package -- the target generator macro for Cargo packages
+################################################################################
+
+cargo-package = $(call inner-cargo-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)
+host-cargo-package = $(call inner-cargo-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host)
-- 
2.29.2

[Buildroot] [PATCH v2 08/12] docs/manual/cargo: document the cargo-package infrastructure

[Thomas Petazzoni]

From: Patrick Havelange <patrick.havelange@essensium.com>

The Buildroot manual was already providing some details on how to
integrate Cargo packages, and those details now need to be updated
with a proper documentation for the cargo-package infrastructure.

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
[Thomas: numerous updates and extensions.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 docs/manual/adding-packages-cargo.txt | 88 ++++++++++++---------------
 1 file changed, 39 insertions(+), 49 deletions(-)

diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt
index 8fcc80bcc6..c65a32f017 100644
--- a/docs/manual/adding-packages-cargo.txt
+++ b/docs/manual/adding-packages-cargo.txt
@@ -1,7 +1,7 @@
 // -*- mode:doc; -*-
 // vim: set syntax=asciidoc:
 
-=== Integration of Cargo-based packages
+=== Infrastructure for Cargo-based packages
 
 Cargo is the package manager for the Rust programming language. It allows the
 user to build programs or libraries written in Rust, but it also downloads and
@@ -10,7 +10,7 @@ called "crates".
 
 [[cargo-package-tutorial]]
 
-==== Cargo-based package's +Config.in+ file
+==== +cargo-package+ tutorial
 
 The +Config.in+ file of Cargo-based package 'foo' should contain:
 
@@ -25,11 +25,7 @@ The +Config.in+ file of Cargo-based package 'foo' should contain:
 08: 	  http://foosoftware.org/foo/
 ---------------------------
 
-==== Cargo-based package's +.mk+ file
-
-Buildroot does not (yet) provide a dedicated package infrastructure for
-Cargo-based packages. So, we will explain how to write a +.mk+ file for such a
-package. Let's start with an example:
+And the +.mk+ file for this package should contain:
 
 ------------------------------
 01: ################################################################################
@@ -44,52 +40,48 @@ package. Let's start with an example:
 10: FOO_LICENSE = GPL-3.0+
 11: FOO_LICENSE_FILES = COPYING
 12:
-13: FOO_DEPENDENCIES = host-rustc
-14:
-15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-16:
-17: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE)
-18:
-19: FOO_CARGO_OPTS = \
-20: 	$(if $(BR2_ENABLE_DEBUG),,--release) \
-21: 	--target=$(RUSTC_TARGET_NAME) \
-22: 	--manifest-path=$(@D)/Cargo.toml
-23:
-24: define FOO_BUILD_CMDS
-25: 	$(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \
-26: 		cargo build $(FOO_CARGO_OPTS)
-27: endef
-28:
-29: define FOO_INSTALL_TARGET_CMDS
-30: 	$(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \
-31: 		$(TARGET_DIR)/usr/bin/foo
-32: endef
-33:
-34: $(eval $(generic-package))
+13: $(eval $(cargo-package))
 --------------------------------
 
-The Makefile starts with the definition of the standard variables for package
-declaration (lines 7 to 11).
+The Makefile starts with the definition of the standard variables for
+package declaration (lines 7 to 11).
+
+As seen in line 13, it is based on the +cargo-package+
+infrastructure. Cargo will be invoked automatically by this
+infrastructure to build and install the package.
+
+It is still possible to define custom build commands or install
+commands (i.e.  with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS).
+Those will then replace the commands from the cargo infrastructure.
+
+==== +cargo-package+ reference
+
+The main macros for the Cargo package infrastructure are
++cargo-package+ for target packages and +host-cargo-package+ for host
+packages.
+
+Just like the generic infrastructure, the Cargo infrastructure works
+by defining a number of variables before calling the +cargo-package+
+or +host-cargo-package+ macros.
 
-As seen in line 34, it is based on the
-xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines
-the variables required by this particular infrastructure, where Cargo is
-invoked:
+First, all the package metadata information variables that exist in
+the generic infrastructure also exist in the Cargo infrastructure:
++FOO_VERSION+, +FOO_SOURCE+, +FOO_PATCH+, +FOO_SITE+,
++FOO_DEPENDENCIES+, +FOO_LICENSE+, +FOO_LICENSE_FILES+, etc.
 
-* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required
-  to configure the cross-compilation of the package are passed via
-  +FOO_CONF_OPTS+.
+A few additional variables, specific to the Cargo infrastructure, can
+also be defined. Many of them are only useful in very specific cases,
+typical packages will therefore only use a few of them.
 
-* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on
-  the target.
+* +FOO_CARGO_ENV+ can be used to pass additional variables in the
+  environment of +cargo+ invocations. It used at both build and
+  installation time
 
-In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to
-contain +host-cargo+.
+* +FOO_CARGO_BUILD_OPTS+ can be used to pass additional options to
+  +cargo+ at build time.
 
-To sum it up, to add a new Cargo-based package, the Makefile example can be
-copied verbatim then edited to replace all occurences of +FOO+ with the
-uppercase name of the new package and update the values of the standard
-variables.
+* +FOO_CARGO_INSTALL_OPTS+ can be used to pass additional options to
+  +cargo+ at install time.
 
 ==== About Dependencies Management
 
@@ -99,9 +91,7 @@ automatically them. This step can also be performed independently, via the
 +cargo fetch+ command.
 
 Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+. As seen in the package
-Makefile example at line 15, this environment variable is set to
-+$(HOST_DIR)/share/cargo+.
+crates, whose location is given by +$CARGO_HOME+.
 
 This dependency download mechanism is not convenient when performing an offline
 build, as Cargo will fail to fetch the dependencies. In that case, it is advised
-- 
2.29.2

[Buildroot] [PATCH v2 09/12] package/ripgrep: convert to cargo infrastructure

[Thomas Petazzoni]

From: Patrick Havelange <patrick.havelange@essensium.com>

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/ripgrep/ripgrep.mk | 28 +---------------------------
 1 file changed, 1 insertion(+), 27 deletions(-)

diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk
index 9dd8d58de1..97e9e2ce5f 100644
--- a/package/ripgrep/ripgrep.mk
+++ b/package/ripgrep/ripgrep.mk
@@ -9,30 +9,4 @@ RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
 RIPGREP_LICENSE = MIT
 RIPGREP_LICENSE_FILES = LICENSE-MIT
 
-RIPGREP_DEPENDENCIES = host-rustc
-RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-
-RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_BIN_SUBDIR)
-
-RIPGREP_CARGO_OPTS = \
-	--target=$(RUSTC_TARGET_NAME) \
-	--manifest-path=$(@D)/Cargo.toml
-
-ifeq ($(BR2_ENABLE_DEBUG),y)
-RIPGREP_CARGO_BIN_SUBDIR = debug
-else
-RIPGREP_CARGO_OPTS += --release
-RIPGREP_CARGO_BIN_SUBDIR = release
-endif
-
-define RIPGREP_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \
-		cargo build $(RIPGREP_CARGO_OPTS)
-endef
-
-define RIPGREP_INSTALL_TARGET_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \
-		$(TARGET_DIR)/usr/bin/rg
-endef
-
-$(eval $(generic-package))
+$(eval $(cargo-package))
-- 
2.29.2

[Buildroot] [PATCH v2 10/12] package/sentry-cli: convert to host-cargo-package infrastructure

[Thomas Petazzoni]

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/sentry-cli/sentry-cli.mk | 22 ++--------------------
 1 file changed, 2 insertions(+), 20 deletions(-)

diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk
index 7e53f16026..40ca22fdf9 100644
--- a/package/sentry-cli/sentry-cli.mk
+++ b/package/sentry-cli/sentry-cli.mk
@@ -9,24 +9,6 @@ SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
 SENTRY_CLI_LICENSE = BSD-3-clause
 SENTRY_CLI_LICENSE_FILES = LICENSE
 
-HOST_SENTRY_CLI_DEPENDENCIES = host-rustc host-zlib
+HOST_SENTRY_CLI_DEPENDENCIES = host-zlib
 
-HOST_SENTRY_CLI_CARGO_ENV = \
-	CARGO_HOME=$(HOST_DIR)/share/cargo \
-	RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))"
-
-HOST_SENTRY_CLI_CARGO_OPTS = \
-	--release \
-	--manifest-path=$(@D)/Cargo.toml
-
-define HOST_SENTRY_CLI_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(HOST_SENTRY_CLI_CARGO_ENV) \
-		cargo build $(HOST_SENTRY_CLI_CARGO_OPTS)
-endef
-
-define HOST_SENTRY_CLI_INSTALL_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/target/release/sentry-cli \
-		$(HOST_DIR)/bin/sentry-cli
-endef
-
-$(eval $(host-generic-package))
+$(eval $(host-cargo-package))
-- 
2.29.2

[Buildroot] [PATCH v2 11/12] support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages

[Thomas Petazzoni]

This commit adds support/download/cargo-post-process to perform the
vendoring on Cargo packages, and enables it in package/pkg-cargo.mk.

Since it changes the contents of the tarballs for ripgrep and
sentry-cli, it changes their hashes. To not have a different hash for
the same version of ripgrep and sentry-cli, we bump their versions. It
has to be done in the same commit as the Cargo vendoring to make the
series bisectable.

The <pkg>_LICENSE variable of cargo packages is expanded with ",
vendored dependencies licenses probably not listed" as currently for
all packages, the licenses of the vendored dependencies are not taken
into account.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-cargo.mk                | 13 ++++++++--
 package/ripgrep/ripgrep.hash        |  2 +-
 package/ripgrep/ripgrep.mk          |  2 +-
 package/sentry-cli/sentry-cli.hash  |  2 +-
 package/sentry-cli/sentry-cli.mk    |  2 +-
 support/download/cargo-post-process | 38 +++++++++++++++++++++++++++++
 6 files changed, 53 insertions(+), 6 deletions(-)
 create mode 100755 support/download/cargo-post-process

diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
index 058c6756bb..371b8dd424 100644
--- a/package/pkg-cargo.mk
+++ b/package/pkg-cargo.mk
@@ -37,10 +37,17 @@
 define inner-cargo-package
 
 # We need host-rustc to run cargo
-$(2)_DEPENDENCIES += host-rustc
+$(2)_DOWNLOAD_DEPENDENCIES += host-rustc
 
 $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
 
+$(2)_DOWNLOAD_POST_PROCESS = cargo
+$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+# Due to vendoring, it is pretty likely that not all licenses are
+# listed in <pkg>_LICENSE.
+$(2)_LICENSE += , vendored dependencies licenses probably not listed
+
 # Note: in all the steps below, we "cd" into the build directory to
 # execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the
 # manifest-path. Indeed while the latter seems to work, it in fact
@@ -50,7 +57,9 @@ $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
 # directory, its configuration file will not be taken into account.
 #
 # Also, we pass:
-#  * --offline to prevent cargo from downloading anything
+#  * --offline to prevent cargo from downloading anything: all
+#    dependencies should have been built by the download post
+#    process logic
 #  * --locked to force cargo to use the Cargo.lock file, which ensures
 #    that a fixed set of dependency versions is used
 
diff --git a/package/ripgrep/ripgrep.hash b/package/ripgrep/ripgrep.hash
index 0841c0185c..81ac905d3d 100644
--- a/package/ripgrep/ripgrep.hash
+++ b/package/ripgrep/ripgrep.hash
@@ -1,3 +1,3 @@
 # Locally calculated
-sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz
+sha256 80d7f07325f4d485c5e4baf061b0376b45a497ee7a1c540d7894057bb9ac3a59 ripgrep-12.1.1.tar.gz
 sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT
diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk
index 97e9e2ce5f..d21f88a6e8 100644
--- a/package/ripgrep/ripgrep.mk
+++ b/package/ripgrep/ripgrep.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-RIPGREP_VERSION = 0.8.1
+RIPGREP_VERSION = 12.1.1
 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
 RIPGREP_LICENSE = MIT
 RIPGREP_LICENSE_FILES = LICENSE-MIT
diff --git a/package/sentry-cli/sentry-cli.hash b/package/sentry-cli/sentry-cli.hash
index 3b0733a276..63b0c812bc 100644
--- a/package/sentry-cli/sentry-cli.hash
+++ b/package/sentry-cli/sentry-cli.hash
@@ -1,3 +1,3 @@
 # locally calculated
-sha256  5d0f7acf6a139d1c1716b9a8ff76c8bfaab09104ba663c957bb9a5dba2ffbabd  sentry-cli-1.57.0.tar.gz
+sha256  a657dd1a46e3de044deff4c311b7276c5a9409582e707e6e6a78b0cf712591c4  sentry-cli-1.59.0.tar.gz
 sha256  9503def7b54ceb6e3cd182fd59bc05d3a30d7eae481e65aaba4b495133c83c14  LICENSE
diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk
index 40ca22fdf9..58f5f1e325 100644
--- a/package/sentry-cli/sentry-cli.mk
+++ b/package/sentry-cli/sentry-cli.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-SENTRY_CLI_VERSION = 1.57.0
+SENTRY_CLI_VERSION = 1.59.0
 SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
 SENTRY_CLI_LICENSE = BSD-3-clause
 SENTRY_CLI_LICENSE_FILES = LICENSE
diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process
new file mode 100755
index 0000000000..5081476385
--- /dev/null
+++ b/support/download/cargo-post-process
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+while getopts "n:o:" OPT; do
+        case "${OPT}" in
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then
+        exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Cargo vendoring
+pushd ${base_name} > /dev/null
+cargo vendor --locked VENDOR
+echo $?
+# Create the local .cargo/config with vendor info
+mkdir -p .cargo/
+cat <<EOF >.cargo/config
+[source.crates-io]
+replace-with = "vendored-sources"
+
+[source.vendored-sources]
+directory = "VENDOR"
+EOF
+popd > /dev/null
+
+repack ${base_name} ${output}
-- 
2.29.2

[Buildroot] [PATCH v2 12/12] docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management

[Thomas Petazzoni]

Now that we have vendoring support for Cargo packages, let's rewrite
the dependency management section in a more accurate way.

We drop the part about the local cache of the registry, because
+CARGO_HOME+ in Buildroot points to $(HOST_DIR)/share/cargo, which is
not shared between builds nor preserved accross builds, so its effect
as a cache is limited.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 docs/manual/adding-packages-cargo.txt | 22 ++++++++++------------
 1 file changed, 10 insertions(+), 12 deletions(-)

diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt
index c65a32f017..fb3e7d0780 100644
--- a/docs/manual/adding-packages-cargo.txt
+++ b/docs/manual/adding-packages-cargo.txt
@@ -85,15 +85,13 @@ typical packages will therefore only use a few of them.
 
 ==== About Dependencies Management
 
-A crate can depend on other libraries from crates.io or git repositories, listed
-in its Cargo.toml file. Before starting a build, Cargo usually downloads
-automatically them. This step can also be performed independently, via the
-+cargo fetch+ command.
-
-Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+.
-
-This dependency download mechanism is not convenient when performing an offline
-build, as Cargo will fail to fetch the dependencies. In that case, it is advised
-to generate a tarball of the dependencies using the +cargo vendor+ and add it to
-+FOO_EXTRA_DOWNLOADS+.
+A crate can depend on other libraries from crates.io or git
+repositories, listed in its Cargo.toml file. Buildroot automatically
+takes care of downloading such dependencies as part of the download
+step of packages that use the +cargo-package+ infrastructure. Such
+dependencies are then kept together with the package source code in
+the tarball cached in Buildroot's +DL_DIR+, and therefore the hash of
+the package's tarball includes such dependencies.
+
+This mechanism ensures that any change in the dependencies will be
+detected, and allows the build to be performed completely offline.
-- 
2.29.2

[Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts

[Yann E. MORIN]

Thomas, All,

On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
> download post process scripts will often need to unpack the source
> code tarball, do some operation, and then repack it. In order to help
> with this, post-process-helpers provide an unpack() function and a
> repack() function.

You forgot to explain in the commit log why we need a hard-coded, fixed
date.

However, I find it ugly that we resort to such an arbitrary constant. At
the very least, I'd like we take a memorable value, like 1970-01-01
00:00:00 +0000.

However, I am afraid that tar would refuse to extract files to that
date, whining about an "implausibly old time stamp" when extracting such
files...

But see below for a proposal...

> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/download/post-process-helpers | 30 +++++++++++++++++++++++++++
>  1 file changed, 30 insertions(+)
>  create mode 100644 support/download/post-process-helpers
> 
> diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers
> new file mode 100644
> index 0000000000..bed8df2577
> --- /dev/null
> +++ b/support/download/post-process-helpers
> @@ -0,0 +1,30 @@
> +
> +unpack() {
> +        dest="$1"
> +        tarball="$2"

This is a shell script, variables are global. But since we run it with
bash, we can declare such variable as local:

    local dest="${1}"
    local tarball="${2}"

> +        mkdir ${dest}
> +        tar -C ${dest} --strip-components=1 -xf ${tarball}

    local one_file

    one_file="$(find "${dest}" -type f |head -n1)"
    touch -r "${one_file}" "${dest}.timestamp"

> +}
> +
> +repack() {
> +        src="$1"
> +        tarball="$2"

Ditto: make them 'local' (this can be fixed when applying).

Also, I suppose that the 'dest' parameter of unpack is expected to be the
'src' parameter of repack, right?

> +        # Generate the archive, sort with the C locale so that it is reproducible.
> +        find "$(basename ${src})" -not -type d -print0 >files.list
> +        LC_ALL=C sort -z <files.list >files.list.sorted
> +
> +        # let's use a fixed hardcoded date to be reproducible
> +        date="2020-02-06 01:02:03 +0000"

Assuming that unpack's dest and repack's src are pointing to the same
directory:

        date="$(stat -c "${src}.timestamp")"

> +        # Create GNU-format tarballs, since that's the format of the tarballs on
> +        # sources.buildroot.org and used in the *.hash files
> +        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
> +            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted

This will have to be changed according to my pending series to change to
the PAX format, whish is even more reproducible.

> +        gzip -6 -n <new.tar >new.tar.gz
> +        mv "${tarball}" "${tarball}".old
> +        mv new.tar.gz "${tarball}"
> +        rm "${tarball}".old
> +        rm -rf ${src}

Quote all expanded variable, even "${src}". And also rm "${src}.timestamp",
now.

Regards,
Yann E. MORIN.

> +}
> -- 
> 2.29.2
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts

[Yann E. MORIN]

Thomas, All,

A few mre tweaks I forgot about...

On 2020-12-19 18:39 +0100, Yann E. MORIN spake thusly:
> On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
[--SNIP--]
> > +        mkdir ${dest}

Using 'mkdir -p' would ensure that parent direcotries are created as
needed.

> 
>     one_file="$(find "${dest}" -type f |head -n1)"

Of course, the output should be sorted in the C locale, to ensure the
same file is always used as reference (and hence a better name for that
file):

    local ref_file
    ref_file="$(find "${dest}" -type f |LC_ALL=C sort |head -n1)"
    touch -r "${ref_file}" "${dest}.timestamp"

[--SNIP--]
> > +        # let's use a fixed hardcoded date to be reproducible
> > +        date="2020-02-06 01:02:03 +0000"
> Assuming that unpack's dest and repack's src are pointing to the same
> directory:
>         date="$(stat -c "${src}.timestamp")"

Meh.. I forgot the actual format:

    date="$(stat -c '%y' "${src}.timestamp")"

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts

[Yann E. MORIN]

Thomas, All,

On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
> download post process scripts will often need to unpack the source
> code tarball, do some operation, and then repack it. In order to help
> with this, post-process-helpers provide an unpack() function and a
> repack() function.

Sorry to come back yet once more on that one, but I may have found quite
an issue. Basically, your code does this:

    unpack() {
        tar -C ${dest} --strip-components=1 -xf ${tarball}
    }

    repack() {
        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
        gzip -6 -n <new.tar >new.tar.gz
    }

So, it takes a tarball in whatever copression scheme and extract it, but
it will alwyas create a gzip-compressed tarball.

So, how would we hanfle a case where the upstream tarball is, say, a
.tar.bz2 ? For example;

    FOO_SOURCE = foo=1.2.3.tar.bz2
    FOO_SITE = http://foo.example.net/download
    FOO_SITE_METOD = wget
    $(eval $(go-package))

Besides the fact that the packager would see in DL_DIR a .tar.gz tarball
different from the .tar.bz2 he coded, the most problematic issue wil be
that we would have to list the .tar.gz inthe .hash file. This will be
very confusing to packagers and reviewers...

Furthermore, there is another discrepancy in the post-processing
scripts. For example, the go post-processor does, basically:

    if tarball already vendored:
        exit leaving the tarball as-is

    unpack
    vendor
    repack as .tar.gz

So some go tarballs will be left as they are upstream, while others will
be re-encoded. This is going to be quite confusing...

I think the post-processing script should always re-encode the tarball,
so that at least we have a consistent behaviour across the board. This
is pretty trivial to come up with, at the exepense of a bit of overhead
for those tarballs that are already vendored, but the majority of
packages will probably be git clones that need to be vendored, so it is
a small price to pay.

The big problem that's left, is how we handle the discrepancy between
the tarball listed in _SOURCE and the one listed in the .hash file...

I have to admit I do not have an easy solution for that for now...

Especially since with the pending series about changing the way we
assemble tarballs from git and svn: the tar format changes, and the
compression changes...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v2 03/12] package/pkg-download.mk: add <pkg>_DL_ENV variable

[Yann E. MORIN]

Thomas, All,

On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
> This variable can be used by package to pass extra environment
> variables to the download logic. It will be used for the Go/Cargo
> vendoring.
> 
> The <pkg>_DL_ENV variable is intentionally not documented: at this
> point, it is not meant to be used by packages directly, but only by
> package infrastructures.
> 
> Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Although it's not much off your chest on this series: applied to
master, with a little tweak, see below...

> ---
>  package/pkg-download.mk | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 175b9d7dba..c914d016e2 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -99,7 +99,9 @@ endif
>  
>  define DOWNLOAD
>  	$(Q)mkdir -p $($(2)_DL_DIR)
> -	$(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
> +	$(Q)$(EXTRA_ENV) \
> +		$($(2)_DL_ENV) \
> +		flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \

I find this code to be ugly to look at. But I could not come up with
something that is nice to look at. So I moved the two _ENV variables
expansions to the same line, and it looks less-ugly now...

Thanks.

Regards,
Yann E. MORIN.

>  		-c '$($(2)_DL_VERSION)' \
>  		-d '$($(2)_DL_DIR)' \
>  		-D '$(DL_DIR)' \
> -- 
> 2.29.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart]

Thomas, all,


On Sat, Dec 19, 2020 at 7:35 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> The logic to generate go.mod when not available is moved to the
> download post-process helper, as it must be done before "go mod
> vendor" is executed. Also, "go mod init" is used instead of manually
> crafting go.mod. This was suggested by Christian Stewart
> <christian@paral.in>. The Go module name is passed down to
> go-post-process using the BR_GOMOD environment variable.

> +go mod vendor -v -modcacherw
> +popd > /dev/null
> +
> +repack ${base_name} ${output}

This works fine in most cases, however, this causes inconsistent hashes.

On my system, the hash is correct after re-pack. But on 2 of my users
machines, the hash ends up different.

So something specific to the host machine must be influencing the hash
of the repacked archive.

It's either something in "go mod vendor" that's not deterministic, or
maybe host Go is being used instead of the one compiled by Buildroot,
or perhaps it's not possible to always get deterministic hashes this
way.

Can we possibly switch to hashing the download /before/ re-packing it,
given that Go verifies the hashes in go.sum automatically as part of
the "go mod vendor" step?

Go mod vendor will fail if any of the dependencies are different, the
checksums are included in go.sum

Thanks & best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

[Thomas Petazzoni]

Hello Christian,

On Thu, 29 Jul 2021 13:17:09 -0700
Christian Stewart <christian@paral.in> wrote:

> This works fine in most cases, however, this causes inconsistent hashes.
> 
> On my system, the hash is correct after re-pack. But on 2 of my users
> machines, the hash ends up different.
> 
> So something specific to the host machine must be influencing the hash
> of the repacked archive.
> 
> It's either something in "go mod vendor" that's not deterministic, or
> maybe host Go is being used instead of the one compiled by Buildroot,
> or perhaps it's not possible to always get deterministic hashes this
> way.

Meh :-/

> Can we possibly switch to hashing the download /before/ re-packing it,
> given that Go verifies the hashes in go.sum automatically as part of
> the "go mod vendor" step?

This is going to be difficult, because the whole principle of this
patch series is to integrate the vendoring within the download step,
i.e *before* the hash is verified. For example, the hash is verified
even if you're just extracting, and not downloading.

One possibility that we had discussed in the past, but that Yann didn't
like was to create one tarball for the upstream code, and another for
the vendored dependencies.

Based on the difficulties, I would really like to understand a little
bit why the archive is not deterministic. Could you compare the
contents of the download stuff between your different machines, and the
contents of the tarball, and see what changes?

> Go mod vendor will fail if any of the dependencies are different, the
> checksums are included in go.sum

Is the presence of go.sum mandatory?

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

[Vincent Fazio]

Christian, all,

On 7/29/21 3:50 PM, Thomas Petazzoni wrote:
> Hello Christian,
>
> On Thu, 29 Jul 2021 13:17:09 -0700
> Christian Stewart <christian@paral.in> wrote:
>
>> This works fine in most cases, however, this causes inconsistent hashes.
>>
>> On my system, the hash is correct after re-pack. But on 2 of my users
>> machines, the hash ends up different.
>>
>> So something specific to the host machine must be influencing the hash
>> of the repacked archive.
>>
>> It's either something in "go mod vendor" that's not deterministic, or
>> maybe host Go is being used instead of the one compiled by Buildroot,
>> or perhaps it's not possible to always get deterministic hashes this
>> way.
> Meh :-/
>
I think Yann already calls out what may be going on in this patch 
https://patchwork.ozlabs.org/project/buildroot/patch/20201219153525.1361175-5-thomas.petazzoni@bootlin.com/

The tarball format will need to follow the same or similar format we use 
for other source packs by using the PAX format and trimming some 
metadata being included in the archive. GNU format tarballs are not 
guaranteed to be consistent across versions.

See: 
https://patchwork.ozlabs.org/project/buildroot/patch/bcb281853f0da8cd970446f4afed093b317dcc82.1609239666.git.yann.morin.1998@free.fr/


-- 
Vincent Fazio
Embedded Software Engineer - ATS
Extreme Engineering Solutions, Inc
http://www.xes-inc.com

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

[Yann E. MORIN]

Christian, All,

On 2021-07-31 21:59 -0700, Christian Stewart spake thusly:
> Vincent said:
> >I think Yann already calls out what may be going on in this patch
> >https://patchwork.ozlabs.org/project/buildroot/patch/20201219153525.1361175-5-thomas.petazzoni@bootlin.com/
> 
> I think this is probably correct, the .tar.gz needs to have
> deterministic formatting.

Can you try usign the tar helper in support/download/helpers instead?

> In this particular case the user was trying to build on Kali 2.
> 
> Doesn't buildroot compile host-gzip and host-tar for this? I'm
> wondering why it's not the same between host machines.

Since 2020.02, we are no longer building host-tar, unless for very old
versions (i.e. before 1.27), thanks to some tar trickery vy Vincent.
See:

    cbe95b1a455b support/download: add helper to generate a reproducible archive

We also only build host-gzip if the host gzip is pigz (or missing).

> On Thu, Jul 29, 2021 at 1:50 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> 
> > > This works fine in most cases, however, this causes inconsistent hashes.
> > Based on the difficulties, I would really like to understand a little
> > bit why the archive is not deterministic. Could you compare the
> > contents of the download stuff between your different machines, and the
> > contents of the tarball, and see what changes?
> The two versions are attached. The actual contents of the two are
> identical. But for some reason the formatting of
> embiggen-disk-bad.tar.gz is different:
> 
>   00000030: 8232 63de 4282 92dd acff fb79 6740 eae2
> 
> Starting at "2dd acff" - in "embiggen-disk-good" it's
> 
> 00000030: 8232 63de 4282 96dd acff fb79 6740 eae6

So, I zcat both archives, and the two tar are identical!

    $ sha1sum *.tar
    dbb8338ab4acaff1e232c67d3b46602d61114b53  embiggen-disk-bad.tar
    dbb8338ab4acaff1e232c67d3b46602d61114b53  embiggen-disk-good.tar

So I also hashed the archives you provided, and they too are identical:

    $ sha1sum *.tar.gz
    bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-bad.tar.gz
    bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-good.tar.gz

Did you mess up when sending the archives?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

[Yann E. MORIN]

Christian, All,

On 2021-08-01 00:24 -0700, Christian Stewart spake thusly:
> On Sun, Aug 1, 2021 at 12:08 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > So I also hashed the archives you provided, and they too are identical:
> >     $ sha1sum *.tar.gz
> >     bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-bad.tar.gz
> >     bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-good.tar.gz
> > Did you mess up when sending the archives?
> Not sure how that got mixed up, attached is the correct
> embiggen-disk-bad.tar.gz:

So, what are the differences?

First, looking at the compressed archive is useless. We really need to
look at the uncompressed archives to see what is going on.

    $ gunzip embiggen-disk-good.tar.gz embiggen-disk-bad.tar.gz

    $ hxedump -Cv embiggen-disk-good.tar >embiggen-disk-good.tar.hex
    $ hxedump -Cv embiggen-disk-bad.tar >embiggen-disk-bad.tar.hex

    $ diff -du embiggen-disk-good.tar.hex embiggen-disk-bad.tar.hex >embiggen-disk.diff

    $ diffstat <embiggen-disk.diff
     embiggen-disk-bad.tar.hex |   52    26    26     0 +++++++++++++-------------
     1 file changed, 26 insertions(+), 26 deletions(-)

So, there aren't so many differences, in the end. What's about them,
then? Let's see the first hunk (the others are very similar):

    @@ -4,10 +4,10 @@
     00000030  31 61 32 33 30 38 2f 43  4f 4e 54 52 49 42 55 54 |1a2308/CONTRIBUT|
     00000040  49 4e 47 2e 6d 64 00 00  00 00 00 00 00 00 00 00 |ING.md..........|
     00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
    -00000060  00 00 00 00 30 30 30 30  36 34 34 00 30 30 30 30 |....0000644.0000|
    +00000060  00 00 00 00 30 30 30 30  36 36 34 00 30 30 30 30 |....0000664.0000|
     00000070  30 30 30 00 30 30 30 30  30 30 30 00 30 30 30 30 |000.0000000.0000|
     00000080  30 30 30 31 33 34 37 00  31 33 36 31 36 36 36 32 |0001347.13616662|
    -00000090  36 31 33 00 30 32 30 34  30 33 00 20 30 00 00 00 |613.020403. 0...|
    +00000090  36 31 33 00 30 32 30 34  30 35 00 20 30 00 00 00 |613.020405. 0...|
     000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
     000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
     000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|

So, the first obvious delta: the access mode. In the 'good' archive, it
is 0644 (rw-r--r--), while in the 'bad' archive, it is 0664 (rw-rw-r--).

But Buildroot is supposed to enforce the mode, see Makefile:

    70: REQ_UMASK = 0022
    ...
    77: ifneq ($(shell umask):$(CURDIR):$(O),$(REQ_UMASK):$(CANONICAL_CURDIR):$(CANONICAL_O))
    ...
    83: _all:
    84:     @umask $(REQ_UMASK) && \
    85:         $(MAKE) -C $(CANONICAL_CURDIR) --no-print-directory \
    86:             $(MAKECMDGOALS) $(EXTRAMAKEARGS)

So, how comes that the two have different modes?

The second change is a bit more curious. As per the GNU tar manual [0]:
offset 0x94 (148) is 'char chksum[8]'.

So we should expect an 8-byte field (still from [0]):

    The chksum field represents the simple sum of all bytes in the
    header block. Each 8-bit byte in the header is added to an unsigned
    integer, initialized to zero [...]

OK, so because the mode is different, the delta from the mode is carried
over to the chksum.

If we can find why the modes are different in your two cases, we can
probably solve the issue.

QED.

Still, I would prefer that we do use the tarball helper, because then we
would have a single location where we could solve this kind of
problems...

[0] https://www.gnu.org/software/tar/manual/html_node/Standard.html

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart via buildroot]

Hi Yann, all,

Some users are still running into the issue that on their system the
"go mod vendor" process is producing different .tar.gz hashes than on
most other systems. Yann previously checked on this and found that the
access modes are wrong in the "bad" archive:

On Sun, Aug 1, 2021 at 2:14 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> So, what are the differences?
>
> First, looking at the compressed archive is useless. We really need to
> look at the uncompressed archives to see what is going on.
>
>     $ gunzip embiggen-disk-good.tar.gz embiggen-disk-bad.tar.gz

[snip]

> So, the first obvious delta: the access mode. In the 'good' archive, it
> is 0644 (rw-r--r--), while in the 'bad' archive, it is 0664 (rw-rw-r--).
>
> But Buildroot is supposed to enforce the mode, see Makefile:
>
>     70: REQ_UMASK = 0022
>     ...
>     77: ifneq ($(shell umask):$(CURDIR):$(O),$(REQ_UMASK):$(CANONICAL_CURDIR):$(CANONICAL_O))
>     ...
>     83: _all:
>     84:     @umask $(REQ_UMASK) && \
>     85:         $(MAKE) -C $(CANONICAL_CURDIR) --no-print-directory \
>     86:             $(MAKECMDGOALS) $(EXTRAMAKEARGS)

[snip]

> If we can find why the modes are different in your two cases, we can
> probably solve the issue.

Does anyone know a suitable fix for this? I recall there might also
have been some other places where this source of lack of determinism
when making .tar.gz was causing issues.

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart via buildroot]

Hi all,


On Sat, Sep 18, 2021 at 11:20 PM Christian Stewart <christian@paral.in> wrote:
> Some users are still running into the issue that on their system the
> "go mod vendor" process is producing different .tar.gz hashes than on
> most other systems. Yann previously checked on this and found that the
> access modes are wrong in the "bad" archive:

> > So, the first obvious delta: the access mode. In the 'good' archive, it
> > is 0644 (rw-r--r--), while in the 'bad' archive, it is 0664 (rw-rw-r--).
> >
> > But Buildroot is supposed to enforce the mode, see Makefile:
> >
> >     70: REQ_UMASK = 0022
> >     ...
> >     77: ifneq ($(shell umask):$(CURDIR):$(O),$(REQ_UMASK):$(CANONICAL_CURDIR):$(CANONICAL_O))
> >     ...
> >     83: _all:
> >     84:     @umask $(REQ_UMASK) && \
> >     85:         $(MAKE) -C $(CANONICAL_CURDIR) --no-print-directory \
> >     86:             $(MAKECMDGOALS) $(EXTRAMAKEARGS)
>
> [snip]
>
> > If we can find why the modes are different in your two cases, we can
> > probably solve the issue.
>

I have adjusted the code from the "support for Go vendoring" patch,
which I now see was never merged into mainline (yet),

The following now uses the "mk_tar_gz" helper, as Yann suggests to do:

diff --git a/support/download/post-process-helpers
b/support/download/post-process-helpers
old mode 100644
new mode 100755
index bed8df2577..0517abcc24
--- a/support/download/post-process-helpers
+++ b/support/download/post-process-helpers
@@ -1,3 +1,5 @@
+# Source the mk_tar_gz helper.
+. "${0%/*}/helpers"

 unpack() {
         dest="$1"
@@ -8,23 +10,23 @@ unpack() {
 }

 repack() {
-        src="$1"
+        base_name="$1"
         tarball="$2"
-
-        # Generate the archive, sort with the C locale so that it is
reproducible.
-        find "$(basename ${src})" -not -type d -print0 >files.list
-        LC_ALL=C sort -z <files.list >files.list.sorted
+        wdir="$(pwd)"

         # let's use a fixed hardcoded date to be reproducible
         date="2020-02-06 01:02:03 +0000"

-        # Create GNU-format tarballs, since that's the format of the
tarballs on
-        # sources.buildroot.org and used in the *.hash files
-        tar cf new.tar --null --verbatim-files-from --numeric-owner
--format=gnu \
-            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
-        gzip -6 -n <new.tar >new.tar.gz
+        # use the helper to create a reproducible archive
+        # note: the current working dir contains base_name.
+        mk_tar_gz "${base_name}" "${base_name}" "${date}" "${wdir}/new.tar.gz"
+
+        # replace the old tarball
         mv "${tarball}" "${tarball}".old
         mv new.tar.gz "${tarball}"
         rm "${tarball}".old
         rm -rf ${src}
 }
+
+# Keep this line and the following as last lines in this file.
+# vim: ft=bash

I think this might fix the issue, if so, shall I re-submit the go
vendoring series with the adjustments?

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot