[Buildroot] [PATCH next 00/12] Support for Cargo and Go vendoring

[Buildroot] [PATCH next 00/12] Support for Cargo and Go vendoring

[Thomas Petazzoni]

Hello,

Here is a patch series that implements support for vendoring of Cargo
and Go packages.

As you're aware, many Cargo and Go packages describe their
dependencies in a very language-specific way, and expect such
dependencies to be downloaded by their language-specific tools.

This series builds on top of the work from Patrick Havelange
<patrick.havelange@essensium.com> who worked on the Cargo integration,
but extends it to also cover the case of Go, showing that we have a
solution solving both of these language-specific package managers. For
the Go vendoring, I've also used the indication provided by Christian
Stewart <christian@paral.in>.

Overall, the solution consists in a concept of "download post-process
helper" is introduced in the download infrastructure. These are shell
scripts that can be invoked right after the download is done, but
before the tarball is hash-checked and stored in DL_DIR. The idea is
that such "download post-process helpers" can run the
language-specific vendoring logic to fetch the package
dependencies. Thanks to this, the tarball in DL_DIR for a given
package not only contains the package source code itself, but also the
source code of its Cargo or Go dependencies. The tarball hash covers
the entire source code, the complete tarball is cached in DL_DIR, in
the primary site and backup site, and the build can fully be done
offline.

Such "download post-process helpers" are registered by means of the
<pkg>_DOWNLOAD_POST_PROCESS variable by the appropriate package
infrastructure.

This series is also available at:

  https://github.com/tpetazzoni/buildroot/commits/pkg-mgr

More specifically, the series goes like this:

 - PATCH 1: add the concept of download post-processing in the
   download helper support/download/dl-wrapper

 - PATCH 2: add the <pkg>_DOWNLOAD_POST_PROCESS variable to the
   package infrastructure, which gets passed down to
   support/download/dl-wrapper

 - PATCH 3: add the <pkg>_DL_ENV variable to the package
   infrastructure, which allows to pass extra variables in the
   download wrapper environment. This will be used to pass extra
   variables specific to Cargo/Go vendoring.

 - PATCH 4: add a set of two helper shell functions that will be used
   by the post processing scripts

 - PATCH 5: implements the Go vendoring itself

 - PATCH 6: adds package/tinifier, which is one package that requires
   Go vendoring. I don't particularly need this package, it was just
   added because the other Go packages we have in Buildroot today
   don't need vendoring.

 - PATCH 7: introduces a cargo-package infrastructure, for both target
   and host packages

 - PATCH 8: documents the added cargo-package infrastructure

 - PATCH 9 and 10: converts ripgrep and sentry-cli to the
   cargo-package and host-cargo-package infrastructures respecitively

 - PATCH 11: implements the Cargo vendoring itself. Note that we need
   to bump the ripgrep and sentry-cli package at the same time, as
   with the vendoring their tarball contents change and therefore
   their hash changes

 - PATCH 12: rewrite the cargo-package documentation about dependency
   management

Best regards,

Thomas

Patrick Havelange (3):
  package/pkg-cargo.mk: introduce the cargo package infrastructure
  docs/manual/cargo: document the cargo-package infrastructure
  package/ripgrep: convert to cargo infrastructure

Thomas Petazzoni (9):
  support/download/dl-wrapper: add concept of download post-processing
  package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable
  package/pkg-download.mk: add <pkg>_DL_ENV variable
  support/download/post-process-helpers: add helper function for post
    process scripts
  support/download/go-post-process: implement Go vendoring support
  package/tinifier: new package
  package/sentry-cli: convert to host-cargo-package infrastructure
  support/download/cargo-post-process, package/pkg-cargo.mk: enable
    vendoring for Cargo packages
  docs/manual/adding-packages-cargo.txt: rewrite explanation about
    dependency management

 DEVELOPERS                            |   1 +
 docs/manual/adding-packages-cargo.txt | 106 ++++++++++------------
 package/Config.in                     |   1 +
 package/Makefile.in                   |   1 +
 package/pkg-cargo.mk                  | 126 ++++++++++++++++++++++++++
 package/pkg-download.mk               |   5 +-
 package/pkg-golang.mk                 |   7 +-
 package/ripgrep/ripgrep.hash          |   2 +-
 package/ripgrep/ripgrep.mk            |  30 +-----
 package/sentry-cli/sentry-cli.hash    |   2 +-
 package/sentry-cli/sentry-cli.mk      |  24 +----
 package/tinifier/Config.in            |  10 ++
 package/tinifier/tinifier.hash        |   3 +
 package/tinifier/tinifier.mk          |  13 +++
 support/download/cargo-post-process   |  38 ++++++++
 support/download/dl-wrapper           |   9 +-
 support/download/go-post-process      |  29 ++++++
 support/download/post-process-helpers |  30 ++++++
 18 files changed, 324 insertions(+), 113 deletions(-)
 create mode 100644 package/pkg-cargo.mk
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
 create mode 100755 support/download/cargo-post-process
 create mode 100755 support/download/go-post-process
 create mode 100644 support/download/post-process-helpers

-- 
2.28.0

[Buildroot] [PATCH next 01/12] support/download/dl-wrapper: add concept of download post-processing

[Thomas Petazzoni]

In order to support package managers such as Cargo (Rust) or Go, we
want to run some custom logic after the main download, but before
packing the tarball and checking the hash.

To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then
support/download/<something>-post-process will be called.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/download/dl-wrapper | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 3315bd410e..2d74554213 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -25,7 +25,7 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() {
         r)  recurse="-r";;
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
+        p)  post_process="${OPTARG}";;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -135,6 +136,12 @@ main() {
             continue
         fi
 
+        if [ -n "${post_process}" ] ; then
+                ${OLDPWD}/support/download/${post_process}-post-process \
+                         -o "${tmpf}" \
+                         -n "${raw_base_name}"
+        fi
+
         # cd back to free the temp-dir, so we can remove it later
         cd "${OLDPWD}"
 
-- 
2.28.0

[Buildroot] [PATCH next 02/12] package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable

[Thomas Petazzoni]

This will allow packages to register than a download post-processing
is needed. Note that this variable is intentionally not documented: it
is an internal variable meant to be set by package infrastructures,
not directly by packages.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-download.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 951d2fb554..175b9d7dba 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -108,6 +108,7 @@ define DOWNLOAD
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
-- 
2.28.0

[Buildroot] [PATCH next 03/12] package/pkg-download.mk: add <pkg>_DL_ENV variable

[Thomas Petazzoni]

This variable can be used by package to pass extra environment
variables to the download logic. It will be used for the Go/Cargo
vendoring.

The <pkg>_DL_ENV variable is intentionally not documented: at this
point, it is not meant to be used by packages directly, but only by
package infrastructures.

Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-download.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 175b9d7dba..c914d016e2 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -99,7 +99,9 @@ endif
 
 define DOWNLOAD
 	$(Q)mkdir -p $($(2)_DL_DIR)
-	$(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
+	$(Q)$(EXTRA_ENV) \
+		$($(2)_DL_ENV) \
+		flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
 		-c '$($(2)_DL_VERSION)' \
 		-d '$($(2)_DL_DIR)' \
 		-D '$(DL_DIR)' \
-- 
2.28.0

[Buildroot] [PATCH next 04/12] support/download/post-process-helpers: add helper function for post process scripts

[Thomas Petazzoni]

download post process scripts will often need to unpack the source
code tarball, do some operation, and then repack it. In order to help
with this, post-process-helpers provide an unpack() function and a
repack() function.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/download/post-process-helpers | 30 +++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 support/download/post-process-helpers

diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers
new file mode 100644
index 0000000000..bed8df2577
--- /dev/null
+++ b/support/download/post-process-helpers
@@ -0,0 +1,30 @@
+
+unpack() {
+        dest="$1"
+        tarball="$2"
+
+        mkdir ${dest}
+        tar -C ${dest} --strip-components=1 -xf ${tarball}
+}
+
+repack() {
+        src="$1"
+        tarball="$2"
+
+        # Generate the archive, sort with the C locale so that it is reproducible.
+        find "$(basename ${src})" -not -type d -print0 >files.list
+        LC_ALL=C sort -z <files.list >files.list.sorted
+
+        # let's use a fixed hardcoded date to be reproducible
+        date="2020-02-06 01:02:03 +0000"
+
+        # Create GNU-format tarballs, since that's the format of the tarballs on
+        # sources.buildroot.org and used in the *.hash files
+        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
+            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
+        gzip -6 -n <new.tar >new.tar.gz
+        mv "${tarball}" "${tarball}".old
+        mv new.tar.gz "${tarball}"
+        rm "${tarball}".old
+        rm -rf ${src}
+}
-- 
2.28.0

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Thomas Petazzoni]

This commit introduces the download post-process script
support/download/go-post-process, and hooks it into the Go package
infrastructure.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-golang.mk            |  7 ++++++-
 support/download/go-post-process | 29 +++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100755 support/download/go-post-process

diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
index 3813e1c406..150445bf17 100644
--- a/package/pkg-golang.mk
+++ b/package/pkg-golang.mk
@@ -47,7 +47,7 @@ $(2)_BUILD_OPTS += \
 	-p $(PARALLEL_JOBS)
 
 # Target packages need the Go compiler on the host.
-$(2)_DEPENDENCIES += host-go
+$(2)_DOWNLOAD_DEPENDENCIES += host-go
 
 $(2)_BUILD_TARGETS ?= .
 
@@ -72,6 +72,11 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
 # If the go.mod file does not exist, one is written with this root path.
 $(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
 
+$(2)_DOWNLOAD_POST_PROCESS = go
+$(2)_DL_ENV = \
+	$(HOST_GO_COMMON_ENV) \
+	GOPROXY=direct
+
 # Generate a go.mod file if it doesn't exist. Note: Go is configured
 # to use the "vendor" dir and not make network calls.
 define $(2)_GEN_GOMOD
diff --git a/support/download/go-post-process b/support/download/go-post-process
new file mode 100755
index 0000000000..01073aee8b
--- /dev/null
+++ b/support/download/go-post-process
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+# Parse our options
+while getopts "n:o:" OPT; do
+        case "${OPT}" in
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
+	exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Go vendoring
+pushd ${base_name} > /dev/null
+go mod vendor -v
+popd > /dev/null
+
+repack ${base_name} ${output}
-- 
2.28.0

[Buildroot] [PATCH next 06/12] package/tinifier: new package

[Thomas Petazzoni]

This is a Go package that needs vendor modules to be downloaded at
build time.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 DEVELOPERS                     |  1 +
 package/Config.in              |  1 +
 package/tinifier/Config.in     | 10 ++++++++++
 package/tinifier/tinifier.hash |  3 +++
 package/tinifier/tinifier.mk   | 13 +++++++++++++
 5 files changed, 28 insertions(+)
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index d0c9c16423..eb97cfeac0 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -2590,6 +2590,7 @@ F:	package/rtc-tools/
 F:	package/sam-ba/
 F:	package/scons/
 F:	package/squashfs/
+F:	package/tinifier/
 F:	package/wayland/
 F:	package/weston/
 F:	support/testing/tests/boot/test_syslinux.py
diff --git a/package/Config.in b/package/Config.in
index d32a271113..a93959ada8 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -298,6 +298,7 @@ comment "Graphic applications"
 	source "package/rrdtool/Config.in"
 	source "package/stellarium/Config.in"
 	source "package/tesseract-ocr/Config.in"
+	source "package/tinifier/Config.in"
 
 comment "Graphic libraries"
 	source "package/cegui/Config.in"
diff --git a/package/tinifier/Config.in b/package/tinifier/Config.in
new file mode 100644
index 0000000000..fbadfe6bd9
--- /dev/null
+++ b/package/tinifier/Config.in
@@ -0,0 +1,10 @@
+config BR2_PACKAGE_TINIFIER
+	bool "tinifier"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	help
+	  CLI tool for images compressing
+
+	  This tool uses tinypng.com API endpoint for compressing your
+	  local jpg/png images (it supports parallel jobs).
+
+	  https://github.com/tarampampam/tinifier
diff --git a/package/tinifier/tinifier.hash b/package/tinifier/tinifier.hash
new file mode 100644
index 0000000000..146700817b
--- /dev/null
+++ b/package/tinifier/tinifier.hash
@@ -0,0 +1,3 @@
+# Locally calculated
+sha256  707a1d9e55aab8c83b65bb10f3ec8c3bc094d77cd576b266f821d9f39133ac3c  tinifier-2.1.0.tar.gz
+sha256  791d8fd993ace44d4d83e2f4820a64d5ad3e37412f029afad46d17a9259de2b6  LICENSE
diff --git a/package/tinifier/tinifier.mk b/package/tinifier/tinifier.mk
new file mode 100644
index 0000000000..b47d265a8e
--- /dev/null
+++ b/package/tinifier/tinifier.mk
@@ -0,0 +1,13 @@
+################################################################################
+#
+# tinifier
+#
+################################################################################
+
+TINIFIER_VERSION = 2.1.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
+TINIFIER_LICENSE = MIT
+TINIFIER_LICENSE_FILES = LICENSE
+TINIFIER_GOMOD = tinifier
+
+$(eval $(golang-package))
-- 
2.28.0

[Buildroot] [PATCH next 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure

[Thomas Petazzoni]

From: Patrick Havelange <patrick.havelange@essensium.com>

In order to be package agnostic, the install phase is now using cargo
instead of install. TARGET_CONFIGURE_OPTS is now also set when running
cargo in order to support cross compiling C code within cargo.

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
[Thomas: add support for host-cargo-package]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/Makefile.in  |   1 +
 package/pkg-cargo.mk | 123 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+)
 create mode 100644 package/pkg-cargo.mk

diff --git a/package/Makefile.in b/package/Makefile.in
index 51f5cbce4f..2af123f36d 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -427,3 +427,4 @@ include package/pkg-waf.mk
 include package/pkg-golang.mk
 include package/pkg-meson.mk
 include package/pkg-qmake.mk
+include package/pkg-cargo.mk
diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
new file mode 100644
index 0000000000..52c237aa3c
--- /dev/null
+++ b/package/pkg-cargo.mk
@@ -0,0 +1,123 @@
+################################################################################
+# Cargo package infrastructure
+#
+# This file implements an infrastructure that eases development of package
+# .mk files for Cargo packages. It should be used for all packages that use
+# Cargo as their build system.
+#
+# See the Buildroot documentation for details on the usage of this
+# infrastructure
+#
+# In terms of implementation, this Cargo infrastructure requires the .mk file
+# to only specify metadata information about the package: name, version,
+# download URL, etc.
+#
+# We still allow the package .mk file to override what the different steps
+# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined,
+# it is used as the list of commands to perform to build the package,
+# instead of the default Cargo behaviour. The package can also define some
+# post operation hooks.
+#
+################################################################################
+
+################################################################################
+# inner-cargo-package -- defines how the configuration, compilation and
+# installation of a cargo package should be done, implements a few hooks
+# to tune the build process for cargo specifities and calls the generic
+# package infrastructure to generate the necessary make targets
+#
+#  argument 1 is the lowercase package name
+#  argument 2 is the uppercase package name, including a HOST_ prefix
+#             for host packages
+#  argument 3 is the uppercase package name, without the HOST_ prefix
+#             for host packages
+#  argument 4 is the type (target or host)
+################################################################################
+
+define inner-cargo-package
+
+# We need host-rustc to run cargo
+$(2)_DEPENDENCIES += host-rustc
+
+$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+#
+# Build step. Only define it if not already defined by the package .mk
+# file.
+#
+ifndef $(2)_BUILD_CMDS
+ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	$$(TARGET_MAKE_ENV) \
+		$$(TARGET_CONFIGURE_OPTS) \
+		$$($(2)_CARGO_ENV) \
+		cargo build \
+			--offline \
+			--target $$(RUSTC_TARGET_NAME) \
+			$$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
+			--manifest-path $$(@D)/Cargo.toml \
+			--locked \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+else # ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \
+		cargo build \
+			--offline \
+			--release \
+			--manifest-path $$(@D)/Cargo.toml \
+			--locked \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+endif # ifeq ($(4),target)
+endif # ifndef $(2)_BUILD_CMDS
+
+#
+# Target installation step. Only define it if not already defined by
+# the package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
+		cargo install \
+			--target $$(RUSTC_TARGET_NAME) \
+			--offline \
+			--root $$(TARGET_DIR)/usr/ \
+			--bins \
+			--path $$(@D) \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \
+		cargo install \
+			--offline \
+			--root $$(HOST_DIR) \
+			--bins \
+			--path $$(@D) \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+# Call the generic package infrastructure to generate the necessary
+# make targets
+$(call inner-generic-package,$(1),$(2),$(3),$(4))
+
+endef
+
+################################################################################
+# cargo-package -- the target generator macro for Cargo packages
+################################################################################
+
+cargo-package = $(call inner-cargo-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)
+host-cargo-package = $(call inner-cargo-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host)
-- 
2.28.0

[Buildroot] [PATCH next 08/12] docs/manual/cargo: document the cargo-package infrastructure

[Thomas Petazzoni]

From: Patrick Havelange <patrick.havelange@essensium.com>

The Buildroot manual was already providing some details on how to
integrate Cargo packages, and those details now need to be updated
with a proper documentation for the cargo-package infrastructure.

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
[Thomas: numerous updates and extensions.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 docs/manual/adding-packages-cargo.txt | 88 ++++++++++++---------------
 1 file changed, 39 insertions(+), 49 deletions(-)

diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt
index 8fcc80bcc6..c65a32f017 100644
--- a/docs/manual/adding-packages-cargo.txt
+++ b/docs/manual/adding-packages-cargo.txt
@@ -1,7 +1,7 @@
 // -*- mode:doc; -*-
 // vim: set syntax=asciidoc:
 
-=== Integration of Cargo-based packages
+=== Infrastructure for Cargo-based packages
 
 Cargo is the package manager for the Rust programming language. It allows the
 user to build programs or libraries written in Rust, but it also downloads and
@@ -10,7 +10,7 @@ called "crates".
 
 [[cargo-package-tutorial]]
 
-==== Cargo-based package's +Config.in+ file
+==== +cargo-package+ tutorial
 
 The +Config.in+ file of Cargo-based package 'foo' should contain:
 
@@ -25,11 +25,7 @@ The +Config.in+ file of Cargo-based package 'foo' should contain:
 08: 	  http://foosoftware.org/foo/
 ---------------------------
 
-==== Cargo-based package's +.mk+ file
-
-Buildroot does not (yet) provide a dedicated package infrastructure for
-Cargo-based packages. So, we will explain how to write a +.mk+ file for such a
-package. Let's start with an example:
+And the +.mk+ file for this package should contain:
 
 ------------------------------
 01: ################################################################################
@@ -44,52 +40,48 @@ package. Let's start with an example:
 10: FOO_LICENSE = GPL-3.0+
 11: FOO_LICENSE_FILES = COPYING
 12:
-13: FOO_DEPENDENCIES = host-rustc
-14:
-15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-16:
-17: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE)
-18:
-19: FOO_CARGO_OPTS = \
-20: 	$(if $(BR2_ENABLE_DEBUG),,--release) \
-21: 	--target=$(RUSTC_TARGET_NAME) \
-22: 	--manifest-path=$(@D)/Cargo.toml
-23:
-24: define FOO_BUILD_CMDS
-25: 	$(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \
-26: 		cargo build $(FOO_CARGO_OPTS)
-27: endef
-28:
-29: define FOO_INSTALL_TARGET_CMDS
-30: 	$(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \
-31: 		$(TARGET_DIR)/usr/bin/foo
-32: endef
-33:
-34: $(eval $(generic-package))
+13: $(eval $(cargo-package))
 --------------------------------
 
-The Makefile starts with the definition of the standard variables for package
-declaration (lines 7 to 11).
+The Makefile starts with the definition of the standard variables for
+package declaration (lines 7 to 11).
+
+As seen in line 13, it is based on the +cargo-package+
+infrastructure. Cargo will be invoked automatically by this
+infrastructure to build and install the package.
+
+It is still possible to define custom build commands or install
+commands (i.e.  with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS).
+Those will then replace the commands from the cargo infrastructure.
+
+==== +cargo-package+ reference
+
+The main macros for the Cargo package infrastructure are
++cargo-package+ for target packages and +host-cargo-package+ for host
+packages.
+
+Just like the generic infrastructure, the Cargo infrastructure works
+by defining a number of variables before calling the +cargo-package+
+or +host-cargo-package+ macros.
 
-As seen in line 34, it is based on the
-xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines
-the variables required by this particular infrastructure, where Cargo is
-invoked:
+First, all the package metadata information variables that exist in
+the generic infrastructure also exist in the Cargo infrastructure:
++FOO_VERSION+, +FOO_SOURCE+, +FOO_PATCH+, +FOO_SITE+,
++FOO_DEPENDENCIES+, +FOO_LICENSE+, +FOO_LICENSE_FILES+, etc.
 
-* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required
-  to configure the cross-compilation of the package are passed via
-  +FOO_CONF_OPTS+.
+A few additional variables, specific to the Cargo infrastructure, can
+also be defined. Many of them are only useful in very specific cases,
+typical packages will therefore only use a few of them.
 
-* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on
-  the target.
+* +FOO_CARGO_ENV+ can be used to pass additional variables in the
+  environment of +cargo+ invocations. It used at both build and
+  installation time
 
-In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to
-contain +host-cargo+.
+* +FOO_CARGO_BUILD_OPTS+ can be used to pass additional options to
+  +cargo+ at build time.
 
-To sum it up, to add a new Cargo-based package, the Makefile example can be
-copied verbatim then edited to replace all occurences of +FOO+ with the
-uppercase name of the new package and update the values of the standard
-variables.
+* +FOO_CARGO_INSTALL_OPTS+ can be used to pass additional options to
+  +cargo+ at install time.
 
 ==== About Dependencies Management
 
@@ -99,9 +91,7 @@ automatically them. This step can also be performed independently, via the
 +cargo fetch+ command.
 
 Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+. As seen in the package
-Makefile example at line 15, this environment variable is set to
-+$(HOST_DIR)/share/cargo+.
+crates, whose location is given by +$CARGO_HOME+.
 
 This dependency download mechanism is not convenient when performing an offline
 build, as Cargo will fail to fetch the dependencies. In that case, it is advised
-- 
2.28.0

[Buildroot] [PATCH next 09/12] package/ripgrep: convert to cargo infrastructure

[Thomas Petazzoni]

From: Patrick Havelange <patrick.havelange@essensium.com>

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/ripgrep/ripgrep.mk | 28 +---------------------------
 1 file changed, 1 insertion(+), 27 deletions(-)

diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk
index 9dd8d58de1..97e9e2ce5f 100644
--- a/package/ripgrep/ripgrep.mk
+++ b/package/ripgrep/ripgrep.mk
@@ -9,30 +9,4 @@ RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
 RIPGREP_LICENSE = MIT
 RIPGREP_LICENSE_FILES = LICENSE-MIT
 
-RIPGREP_DEPENDENCIES = host-rustc
-RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-
-RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_BIN_SUBDIR)
-
-RIPGREP_CARGO_OPTS = \
-	--target=$(RUSTC_TARGET_NAME) \
-	--manifest-path=$(@D)/Cargo.toml
-
-ifeq ($(BR2_ENABLE_DEBUG),y)
-RIPGREP_CARGO_BIN_SUBDIR = debug
-else
-RIPGREP_CARGO_OPTS += --release
-RIPGREP_CARGO_BIN_SUBDIR = release
-endif
-
-define RIPGREP_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \
-		cargo build $(RIPGREP_CARGO_OPTS)
-endef
-
-define RIPGREP_INSTALL_TARGET_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \
-		$(TARGET_DIR)/usr/bin/rg
-endef
-
-$(eval $(generic-package))
+$(eval $(cargo-package))
-- 
2.28.0

[Buildroot] [PATCH next 10/12] package/sentry-cli: convert to host-cargo-package infrastructure

[Thomas Petazzoni]

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/sentry-cli/sentry-cli.mk | 22 ++--------------------
 1 file changed, 2 insertions(+), 20 deletions(-)

diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk
index 7e53f16026..40ca22fdf9 100644
--- a/package/sentry-cli/sentry-cli.mk
+++ b/package/sentry-cli/sentry-cli.mk
@@ -9,24 +9,6 @@ SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
 SENTRY_CLI_LICENSE = BSD-3-clause
 SENTRY_CLI_LICENSE_FILES = LICENSE
 
-HOST_SENTRY_CLI_DEPENDENCIES = host-rustc host-zlib
+HOST_SENTRY_CLI_DEPENDENCIES = host-zlib
 
-HOST_SENTRY_CLI_CARGO_ENV = \
-	CARGO_HOME=$(HOST_DIR)/share/cargo \
-	RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))"
-
-HOST_SENTRY_CLI_CARGO_OPTS = \
-	--release \
-	--manifest-path=$(@D)/Cargo.toml
-
-define HOST_SENTRY_CLI_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(HOST_SENTRY_CLI_CARGO_ENV) \
-		cargo build $(HOST_SENTRY_CLI_CARGO_OPTS)
-endef
-
-define HOST_SENTRY_CLI_INSTALL_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/target/release/sentry-cli \
-		$(HOST_DIR)/bin/sentry-cli
-endef
-
-$(eval $(host-generic-package))
+$(eval $(host-cargo-package))
-- 
2.28.0

[Buildroot] [PATCH next 11/12] support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages

[Thomas Petazzoni]

This commit adds support/download/cargo-post-process to perform the
vendoring on Cargo packages, and enables it in package/pkg-cargo.mk.

Since it changes the contents of the tarballs for ripgrep and
sentry-cli, it changes their hashes. To not have a different hash for
the same version of ripgrep and sentry-cli, we bump their versions. It
has to be done in the same commit as the Cargo vendoring to make the
series bisectable.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-cargo.mk                |  5 +++-
 package/ripgrep/ripgrep.hash        |  2 +-
 package/ripgrep/ripgrep.mk          |  2 +-
 package/sentry-cli/sentry-cli.hash  |  2 +-
 package/sentry-cli/sentry-cli.mk    |  2 +-
 support/download/cargo-post-process | 38 +++++++++++++++++++++++++++++
 6 files changed, 46 insertions(+), 5 deletions(-)
 create mode 100755 support/download/cargo-post-process

diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
index 52c237aa3c..6653f64594 100644
--- a/package/pkg-cargo.mk
+++ b/package/pkg-cargo.mk
@@ -37,10 +37,13 @@
 define inner-cargo-package
 
 # We need host-rustc to run cargo
-$(2)_DEPENDENCIES += host-rustc
+$(2)_DOWNLOAD_DEPENDENCIES += host-rustc
 
 $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
 
+$(2)_DOWNLOAD_POST_PROCESS = cargo
+$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo
+
 #
 # Build step. Only define it if not already defined by the package .mk
 # file.
diff --git a/package/ripgrep/ripgrep.hash b/package/ripgrep/ripgrep.hash
index 0841c0185c..81ac905d3d 100644
--- a/package/ripgrep/ripgrep.hash
+++ b/package/ripgrep/ripgrep.hash
@@ -1,3 +1,3 @@
 # Locally calculated
-sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz
+sha256 80d7f07325f4d485c5e4baf061b0376b45a497ee7a1c540d7894057bb9ac3a59 ripgrep-12.1.1.tar.gz
 sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT
diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk
index 97e9e2ce5f..d21f88a6e8 100644
--- a/package/ripgrep/ripgrep.mk
+++ b/package/ripgrep/ripgrep.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-RIPGREP_VERSION = 0.8.1
+RIPGREP_VERSION = 12.1.1
 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
 RIPGREP_LICENSE = MIT
 RIPGREP_LICENSE_FILES = LICENSE-MIT
diff --git a/package/sentry-cli/sentry-cli.hash b/package/sentry-cli/sentry-cli.hash
index 3b0733a276..63b0c812bc 100644
--- a/package/sentry-cli/sentry-cli.hash
+++ b/package/sentry-cli/sentry-cli.hash
@@ -1,3 +1,3 @@
 # locally calculated
-sha256  5d0f7acf6a139d1c1716b9a8ff76c8bfaab09104ba663c957bb9a5dba2ffbabd  sentry-cli-1.57.0.tar.gz
+sha256  a657dd1a46e3de044deff4c311b7276c5a9409582e707e6e6a78b0cf712591c4  sentry-cli-1.59.0.tar.gz
 sha256  9503def7b54ceb6e3cd182fd59bc05d3a30d7eae481e65aaba4b495133c83c14  LICENSE
diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk
index 40ca22fdf9..58f5f1e325 100644
--- a/package/sentry-cli/sentry-cli.mk
+++ b/package/sentry-cli/sentry-cli.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-SENTRY_CLI_VERSION = 1.57.0
+SENTRY_CLI_VERSION = 1.59.0
 SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
 SENTRY_CLI_LICENSE = BSD-3-clause
 SENTRY_CLI_LICENSE_FILES = LICENSE
diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process
new file mode 100755
index 0000000000..5081476385
--- /dev/null
+++ b/support/download/cargo-post-process
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+while getopts "n:o:" OPT; do
+        case "${OPT}" in
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then
+        exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Cargo vendoring
+pushd ${base_name} > /dev/null
+cargo vendor --locked VENDOR
+echo $?
+# Create the local .cargo/config with vendor info
+mkdir -p .cargo/
+cat <<EOF >.cargo/config
+[source.crates-io]
+replace-with = "vendored-sources"
+
+[source.vendored-sources]
+directory = "VENDOR"
+EOF
+popd > /dev/null
+
+repack ${base_name} ${output}
-- 
2.28.0

[Buildroot] [PATCH next 12/12] docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management

[Thomas Petazzoni]

Now that we have vendoring support for Cargo packages, let's rewrite
the dependency management section in a more accurate way.

We drop the part about the local cache of the registry, because
+CARGO_HOME+ in Buildroot points to $(HOST_DIR)/share/cargo, which is
not shared between builds nor preserved accross builds, so its effect
as a cache is limited.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 docs/manual/adding-packages-cargo.txt | 22 ++++++++++------------
 1 file changed, 10 insertions(+), 12 deletions(-)

diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt
index c65a32f017..fb3e7d0780 100644
--- a/docs/manual/adding-packages-cargo.txt
+++ b/docs/manual/adding-packages-cargo.txt
@@ -85,15 +85,13 @@ typical packages will therefore only use a few of them.
 
 ==== About Dependencies Management
 
-A crate can depend on other libraries from crates.io or git repositories, listed
-in its Cargo.toml file. Before starting a build, Cargo usually downloads
-automatically them. This step can also be performed independently, via the
-+cargo fetch+ command.
-
-Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+.
-
-This dependency download mechanism is not convenient when performing an offline
-build, as Cargo will fail to fetch the dependencies. In that case, it is advised
-to generate a tarball of the dependencies using the +cargo vendor+ and add it to
-+FOO_EXTRA_DOWNLOADS+.
+A crate can depend on other libraries from crates.io or git
+repositories, listed in its Cargo.toml file. Buildroot automatically
+takes care of downloading such dependencies as part of the download
+step of packages that use the +cargo-package+ infrastructure. Such
+dependencies are then kept together with the package source code in
+the tarball cached in Buildroot's +DL_DIR+, and therefore the hash of
+the package's tarball includes such dependencies.
+
+This mechanism ensures that any change in the dependencies will be
+detected, and allows the build to be performed completely offline.
-- 
2.28.0

[Buildroot] [PATCH next 06/12] package/tinifier: new package

[Ryan Barnett]

Thomas,

I've taken a preliminary look at your pkg-mgr patch series and will be
providing additional feedback over the next few days as I have a
chance to test it further.

On Thu, Nov 19, 2020 at 3:37 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> This is a Go package that needs vendor modules to be downloaded at
> build time.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  DEVELOPERS                     |  1 +
>  package/Config.in              |  1 +
>  package/tinifier/Config.in     | 10 ++++++++++
>  package/tinifier/tinifier.hash |  3 +++
>  package/tinifier/tinifier.mk   | 13 +++++++++++++
>  5 files changed, 28 insertions(+)
>  create mode 100644 package/tinifier/Config.in
>  create mode 100644 package/tinifier/tinifier.hash
>  create mode 100644 package/tinifier/tinifier.mk

[...]

> diff --git a/package/tinifier/tinifier.mk b/package/tinifier/tinifier.mk
> new file mode 100644
> index 0000000000..b47d265a8e
> --- /dev/null
> +++ b/package/tinifier/tinifier.mk
> @@ -0,0 +1,13 @@
> +################################################################################
> +#
> +# tinifier
> +#
> +################################################################################
> +
> +TINIFIER_VERSION = 2.1.0
> +TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
> +TINIFIER_LICENSE = MIT
> +TINIFIER_LICENSE_FILES = LICENSE

I took a look at the legal-info side in regards to downloading
packages with the post-processing support. This has been discussed
previously on the patch "[v3,09/10] package/ripgrep: add legal-info
for dependencies":

https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/

When I ran 'make legal-info' for the tinifier package all that is
mentioned in the 'manifest.csv' file for the package is:

   "tinifier","2.1.0","MIT","LICENSE","tinifier-2.1.0.tar.gz","https://github.com/tarampampam/tinifier/archive/v2.1.0","skeleton-init-common
[unknown] skeleton-init-none [unknown] toolchain-external-bootlin
[unknown]"

This doesn't give any indication or warnings that dependencies were
downloaded or that other open source license could be needed by
including this package. As user of buildroot who may not have any
knowledge in regards to 'go' or 'cargo'. When they see tinifier row in
the manifest.csv file, they could just think that the tinifier package
would only have/introduce the MIT license to their product. Which is
not the case because it downloads the following vendor packages:

  bou.ke/monkey v1.0.2
  github.com/dustin/go-humanize v1.0.0
  github.com/jessevdk/go-flags v1.4.1-0.20181221193153-c0795c8afcf4
  github.com/json-iterator/go v1.1.10
  github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d
  github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
  github.com/modern-go/reflect2 v1.0.1 // indirect
  github.com/olekukonko/tablewriter v0.0.4
  github.com/schollz/progressbar/v3 v3.3.3
  github.com/stretchr/testify v1.6.1

I understand add/showing these license and how to exactly handle this
are future additions.  However, I re-read the section in regards to
the "legal-info" in the buildroot manual I came across this:

  Moreover, due to technical limitations, Buildroot does not produce some
  material that you will or may need, such as the toolchain source code for
  some of the external toolchains and the Buildroot source code itself.
  When you run +make legal-info+, Buildroot produces warnings in the +README+
  file to inform you of relevant material that could not be saved.

So would it be possible to put a warning into the 'legal-info/README'
file that not all the dependency licenses could be downloaded/added to
manifest.csv file?

Maybe a hint could be given to go take a look at the tinifier/go.mod
under the requires section to figure out the licenses. I think this
could be a good temporary solution under a more "dynamic legal-info"
infrastructure could be introduced. Would there be a way to detect if
there are any vendor packages downloaded and then add a warning in the
'legal-info/README' file easily?

Thanks,
-Ryan Barnett

> +TINIFIER_GOMOD = tinifier
> +
> +$(eval $(golang-package))
> --
> 2.28.0

[Buildroot] [PATCH next 06/12] package/tinifier: new package

[Yann E. MORIN]

Ryan, All,

On 2020-11-21 10:37 -0600, Ryan Barnett spake thusly:
> On Thu, Nov 19, 2020 at 3:37 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> > This is a Go package that needs vendor modules to be downloaded at
> > build time.
[--SNIP--]
> > diff --git a/package/tinifier/tinifier.mk b/package/tinifier/tinifier.mk
> > new file mode 100644
> > index 0000000000..b47d265a8e
> > --- /dev/null
> > +++ b/package/tinifier/tinifier.mk
> > @@ -0,0 +1,13 @@
> > +################################################################################
> > +#
> > +# tinifier
> > +#
> > +################################################################################
> > +
> > +TINIFIER_VERSION = 2.1.0
> > +TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
> > +TINIFIER_LICENSE = MIT
> > +TINIFIER_LICENSE_FILES = LICENSE
> 
> I took a look at the legal-info side in regards to downloading
> packages with the post-processing support. This has been discussed
> previously on the patch "[v3,09/10] package/ripgrep: add legal-info
> for dependencies":
> 
> https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/

legal-info is also something Thomas and I discussed and IRC when he
posted his series.

We know it is not perfect, but this can be extended in a followup
series.

> When I ran 'make legal-info' for the tinifier package all that is
> mentioned in the 'manifest.csv' file for the package is:
> 
>    "tinifier","2.1.0","MIT","LICENSE","tinifier-2.1.0.tar.gz","https://github.com/tarampampam/tinifier/archive/v2.1.0","skeleton-init-common
> [unknown] skeleton-init-none [unknown] toolchain-external-bootlin
> [unknown]"
> 
> This doesn't give any indication or warnings that dependencies were
> downloaded or that other open source license could be needed by
> including this package.

To simplify the series, my position as a first step would be to extend
the FOO_LICENSE list in the infra, with just a very short notice,
something like:

    FOO_LICENSE += , vendored licenses not listed

> As user of buildroot who may not have any
> knowledge in regards to 'go' or 'cargo'. When they see tinifier row in
> the manifest.csv file, they could just think that the tinifier package
> would only have/introduce the MIT license to their product. Which is
> not the case because it downloads the following vendor packages:
> 
>   bou.ke/monkey v1.0.2
>   github.com/dustin/go-humanize v1.0.0
>   github.com/jessevdk/go-flags v1.4.1-0.20181221193153-c0795c8afcf4
>   github.com/json-iterator/go v1.1.10
>   github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d
>   github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
>   github.com/modern-go/reflect2 v1.0.1 // indirect
>   github.com/olekukonko/tablewriter v0.0.4
>   github.com/schollz/progressbar/v3 v3.3.3
>   github.com/stretchr/testify v1.6.1
> 
> I understand add/showing these license and how to exactly handle this
> are future additions.

My idea is that the go/cargo/... infras would be responsible for
providing "some kind of" post-legal-info hooks, so they can extend the
licenses list and license files list as well.

But really, I would like to make that a next step, so that the technical
side of the support for package managers can get in sooner rather than
later.

If we can not in the end come up woth a satifying licensing report for
those (or for some of those) package managers, we would at least have
support for building them.

FTR, Thomas and I already adressed that issue quite a while ago, and we
concluded that it was not so obvious as one may initially think (I'd
have to dig my IRC logs to find the explanations...)

>  However, I re-read the section in regards to
> the "legal-info" in the buildroot manual I came across this:
> 
>   Moreover, due to technical limitations, Buildroot does not produce some
>   material that you will or may need, such as the toolchain source code for
>   some of the external toolchains and the Buildroot source code itself.
>   When you run +make legal-info+, Buildroot produces warnings in the +README+
>   file to inform you of relevant material that could not be saved.
> 
> So would it be possible to put a warning into the 'legal-info/README'
> file that not all the dependency licenses could be downloaded/added to
> manifest.csv file?

With the added blurb I suggest above, I think we would be pretty much
covered, no?

Regards,
Yann E. MORIN.

> Maybe a hint could be given to go take a look at the tinifier/go.mod
> under the requires section to figure out the licenses. I think this
> could be a good temporary solution under a more "dynamic legal-info"
> infrastructure could be introduced. Would there be a way to detect if
> there are any vendor packages downloaded and then add a warning in the
> 'legal-info/README' file easily?
> 
> Thanks,
> -Ryan Barnett
> 
> > +TINIFIER_GOMOD = tinifier
> > +
> > +$(eval $(golang-package))
> > --
> > 2.28.0
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH next 06/12] package/tinifier: new package

[Christian Stewart]

Yann,


On Sat, Nov 21, 2020 at 10:04 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > I took a look at the legal-info side in regards to downloading
> > packages with the post-processing support. This has been discussed
> > previously on the patch "[v3,09/10] package/ripgrep: add legal-info
> > for dependencies":
> >
> > https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/
>
> legal-info is also something Thomas and I discussed and IRC when he
> posted his series.
>
> We know it is not perfect, but this can be extended in a followup
> series.

> > I understand add/showing these license and how to exactly handle this
> > are future additions.
>
> My idea is that the go/cargo/... infras would be responsible for
> providing "some kind of" post-legal-info hooks, so they can extend the
> licenses list and license files list as well.
>
> But really, I would like to make that a next step, so that the technical
> side of the support for package managers can get in sooner rather than
> later.
>
> If we can not in the end come up woth a satifying licensing report for
> those (or for some of those) package managers, we would at least have
> support for building them.
>
> FTR, Thomas and I already adressed that issue quite a while ago, and we
> concluded that it was not so obvious as one may initially think (I'd
> have to dig my IRC logs to find the explanations...)

If you're open to building + running host Go code as part of this
process, then it is straightforward to extract the module dependency
graph and the Licenses.

If we're doing this bash + makefile purism, then this is definitely not easy.

Best,
Christian

[Buildroot] [PATCH next 06/12] package/tinifier: new package

[Sam Voss]

Christian, Yann,

On Sun, Nov 22, 2020 at 12:09 AM Christian Stewart <christian@paral.in> wrote:
>
> Yann,
>
>
> On Sat, Nov 21, 2020 at 10:04 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > > I took a look at the legal-info side in regards to downloading
> > > packages with the post-processing support. This has been discussed
> > > previously on the patch "[v3,09/10] package/ripgrep: add legal-info
> > > for dependencies":
> > >
> > > https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/
> >
> > legal-info is also something Thomas and I discussed and IRC when he
> > posted his series.
> >
> > We know it is not perfect, but this can be extended in a followup
> > series.
>
> > > I understand add/showing these license and how to exactly handle this
> > > are future additions.
> >
> > My idea is that the go/cargo/... infras would be responsible for
> > providing "some kind of" post-legal-info hooks, so they can extend the
> > licenses list and license files list as well.
> >
> > But really, I would like to make that a next step, so that the technical
> > side of the support for package managers can get in sooner rather than
> > later.
> >
> > If we can not in the end come up woth a satifying licensing report for
> > those (or for some of those) package managers, we would at least have
> > support for building them.
> >
> > FTR, Thomas and I already adressed that issue quite a while ago, and we
> > concluded that it was not so obvious as one may initially think (I'd
> > have to dig my IRC logs to find the explanations...)
>
> If you're open to building + running host Go code as part of this
> process, then it is straightforward to extract the module dependency
> graph and the Licenses.
>
> If we're doing this bash + makefile purism, then this is definitely not easy.

Same with cargo - unfortunately though for cargo it isn't baked in and
you need to install a plugin to do it. Not the best...but possible.

Sam

[Buildroot] [PATCH next 06/12] package/tinifier: new package

[Ryan Barnett]

Yann, All,

On Sat, Nov 21, 2020 at 12:04 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Ryan, All,
>
> On 2020-11-21 10:37 -0600, Ryan Barnett spake thusly:
> > On Thu, Nov 19, 2020 at 3:37 PM Thomas Petazzoni
> > <thomas.petazzoni@bootlin.com> wrote:
> > > This is a Go package that needs vendor modules to be downloaded at
> > > build time.
> [--SNIP--]
> > > diff --git a/package/tinifier/tinifier.mk b/package/tinifier/tinifier.mk
> > > new file mode 100644
> > > index 0000000000..b47d265a8e
> > > --- /dev/null
> > > +++ b/package/tinifier/tinifier.mk
> > > @@ -0,0 +1,13 @@
> > > +################################################################################
> > > +#
> > > +# tinifier
> > > +#
> > > +################################################################################
> > > +
> > > +TINIFIER_VERSION = 2.1.0
> > > +TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
> > > +TINIFIER_LICENSE = MIT
> > > +TINIFIER_LICENSE_FILES = LICENSE
> >
> > I took a look at the legal-info side in regards to downloading
> > packages with the post-processing support. This has been discussed
> > previously on the patch "[v3,09/10] package/ripgrep: add legal-info
> > for dependencies":
> >
> > https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/
>
> legal-info is also something Thomas and I discussed and IRC when he
> posted his series.
>
> We know it is not perfect, but this can be extended in a followup
> series.
>
> > When I ran 'make legal-info' for the tinifier package all that is
> > mentioned in the 'manifest.csv' file for the package is:
> >
> >    "tinifier","2.1.0","MIT","LICENSE","tinifier-2.1.0.tar.gz","https://github.com/tarampampam/tinifier/archive/v2.1.0","skeleton-init-common
> > [unknown] skeleton-init-none [unknown] toolchain-external-bootlin
> > [unknown]"
> >
> > This doesn't give any indication or warnings that dependencies were
> > downloaded or that other open source license could be needed by
> > including this package.
>
> To simplify the series, my position as a first step would be to extend
> the FOO_LICENSE list in the infra, with just a very short notice,
> something like:
>
>     FOO_LICENSE += , vendored licenses not listed

Adding this within the go/cargo infrastructure works for since it
informs a user that the package has licenses not listed.

Would a note in the buildroot manual be useful under the legal-info
section mention the go/cargo license limitations be useful as well?

> > As user of buildroot who may not have any
> > knowledge in regards to 'go' or 'cargo'. When they see tinifier row in
> > the manifest.csv file, they could just think that the tinifier package
> > would only have/introduce the MIT license to their product. Which is
> > not the case because it downloads the following vendor packages:
> >
> >   bou.ke/monkey v1.0.2
> >   github.com/dustin/go-humanize v1.0.0
> >   github.com/jessevdk/go-flags v1.4.1-0.20181221193153-c0795c8afcf4
> >   github.com/json-iterator/go v1.1.10
> >   github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d
> >   github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
> >   github.com/modern-go/reflect2 v1.0.1 // indirect
> >   github.com/olekukonko/tablewriter v0.0.4
> >   github.com/schollz/progressbar/v3 v3.3.3
> >   github.com/stretchr/testify v1.6.1
> >
> > I understand add/showing these license and how to exactly handle this
> > are future additions.
>
> My idea is that the go/cargo/... infras would be responsible for
> providing "some kind of" post-legal-info hooks, so they can extend the
> licenses list and license files list as well.
>
> But really, I would like to make that a next step, so that the technical
> side of the support for package managers can get in sooner rather than
> later.
>
> If we can not in the end come up woth a satifying licensing report for
> those (or for some of those) package managers, we would at least have
> support for building them.
>
> FTR, Thomas and I already adressed that issue quite a while ago, and we
> concluded that it was not so obvious as one may initially think (I'd
> have to dig my IRC logs to find the explanations...)

The patchworks link I provided in my original feedback email contains
the IRC convo between you and Thomas.

> >  However, I re-read the section in regards to
> > the "legal-info" in the buildroot manual I came across this:
> >
> >   Moreover, due to technical limitations, Buildroot does not produce some
> >   material that you will or may need, such as the toolchain source code for
> >   some of the external toolchains and the Buildroot source code itself.
> >   When you run +make legal-info+, Buildroot produces warnings in the +README+
> >   file to inform you of relevant material that could not be saved.
> >
> > So would it be possible to put a warning into the 'legal-info/README'
> > file that not all the dependency licenses could be downloaded/added to
> > manifest.csv file?
>
> With the added blurb I suggest above, I think we would be pretty much
> covered, no?

Yes the suggested blurb in the go/cargo infrastructure is good with me.

Thanks,
-Ryan

[Buildroot] [PATCH next 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure

[Ryan Barnett]

Thomas,

I've been testing out your pkg-mgr manager series branch over the last
few days. I've run into an issue with the cargo package infrastructure
in trying to build a package after doing a 'make clean'. The error is
follows:

>>> ripgrep 12.1.1 Extracting
gzip -d -c /home/ryan/.br-dl/ripgrep/ripgrep-12.1.1.tar.gz |
/home/ryan/projects/br/br-pkg-mgr/output/host/bin/tar
--strip-components=1 -C
/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1   -xf -
>>> ripgrep 12.1.1 Patching
>>> ripgrep 12.1.1 Configuring
>>> ripgrep 12.1.1 Building
PATH="/home/ryan/projects/br/br-pkg-mgr/output/host/bin:/home/ryan/projects/br/br-pkg-mgr/output/host/sbin:/home/ryan/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
PATH="/home/ryan/projects/br/br-pkg-mgr/output/host/bin:/home/ryan/projects/br/br-pkg-mgr/output/host/sbin:/home/ryan/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
AR="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-ar"
AS="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-as"
LD="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-ld"
NM="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-nm"
CC="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-gcc"
GCC="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-gcc"
CPP="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-cpp"
CXX="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-g++"
FC="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-gfortran"
F77="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-gfortran"
RANLIB="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-ranlib"
READELF="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-readelf"
STRIP="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-strip"
OBJCOPY="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-objcopy"
OBJDUMP="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-objdump"
AR_FOR_BUILD="/usr/bin/ar" AS_FOR_BUILD="/usr/bin/as"
CC_FOR_BUILD="/usr/bin/gcc" GCC_FOR_BUILD="/usr/bin/gcc"
CXX_FOR_BUILD="/usr/bin/g++" LD_FOR_BUILD="/usr/bin/ld"
CPPFLAGS_FOR_BUILD="-I/home/ryan/projects/br/br-pkg-mgr/output/host/include"
CFLAGS_FOR_BUILD="-O2
-I/home/ryan/projects/br/br-pkg-mgr/output/host/include"
CXXFLAGS_FOR_BUILD="-O2
-I/home/ryan/projects/br/br-pkg-mgr/output/host/include"
LDFLAGS_FOR_BUILD="-L/home/ryan/projects/br/br-pkg-mgr/output/host/lib
-Wl,-rpath,/home/ryan/projects/br/br-pkg-mgr/output/host/lib"
FCFLAGS_FOR_BUILD=""
DEFAULT_ASSEMBLER="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-as"
DEFAULT_LINKER="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-ld"
CPPFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE
-D_FILE_OFFSET_BITS=64" CFLAGS="-D_LARGEFILE_SOURCE
-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -Os  "
CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE
-D_FILE_OFFSET_BITS=64  -Os  " LDFLAGS="" FCFLAGS=" -Os " FFLAGS=" -Os
" PKG_CONFIG="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/pkg-config"
STAGING_DIR="/home/ryan/projects/br/br-pkg-mgr/output/host/x86_64-buildroot-linux-gnu/sysroot"
INTLTOOL_PERL=/usr/bin/perl
CARGO_HOME=/home/ryan/projects/br/br-pkg-mgr/output/host/share/cargo
cargo build --offline --target x86_64-unknown-linux-gnu --release
--manifest-path
/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/Cargo.toml
--locked
error: no matching package named `aho-corasick` found
location searched: registry `https://github.com/rust-lang/crates.io-index`
required by package `globset v0.4.5
(/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/crates/globset)`
As a reminder, you're using offline mode (--offline) which can
sometimes cause surprising resolution failures, if this error is too
confusing you may wish to retry without the offline flag.
make[1]: *** [package/pkg-generic.mk:250:
/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/.stamp_built]
Error 101
make: *** [Makefile:84: _all] Error 2

I get this error by running the following commands:

make ripgrep
make clean ripgrep

The defconfig I used is:

BR2_x86_64=y
BR2_x86_corei7=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_X86_64_CORE_I7_GLIBC_STABLE=y
BR2_INIT_NONE=y
# BR2_TARGET_ENABLE_ROOT_LOGIN is not set
# BR2_TARGET_GENERIC_GETTY is not set
# BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW is not set
# BR2_PACKAGE_BUSYBOX is not set

I have BR2_DL_DIR configured as follows:

declare -x BR2_DL_DIR="~/.br-dl"

I can work around the issue by running the following commands:

rm -rf ~/.br-dl/ripgrep/
make clean ripgrep

So it appears that after the rippackage has been downloaded there is
an issue with doing a clean build. I'm not familiar with cargo/rust
builds and how the offline component works but it appears that the
introduction of this cargo infrastructure and the downloading of
vendor dependencies is not currently reproducible.

Let me know if you need any additional information to reproduce the problem.

Thanks,
-Ryan


On Thu, Nov 19, 2020 at 3:37 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> From: Patrick Havelange <patrick.havelange@essensium.com>
>
> In order to be package agnostic, the install phase is now using cargo
> instead of install. TARGET_CONFIGURE_OPTS is now also set when running
> cargo in order to support cross compiling C code within cargo.
>
> Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
> [Thomas: add support for host-cargo-package]
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/Makefile.in  |   1 +
>  package/pkg-cargo.mk | 123 +++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 124 insertions(+)
>  create mode 100644 package/pkg-cargo.mk
>
> diff --git a/package/Makefile.in b/package/Makefile.in
> index 51f5cbce4f..2af123f36d 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -427,3 +427,4 @@ include package/pkg-waf.mk
>  include package/pkg-golang.mk
>  include package/pkg-meson.mk
>  include package/pkg-qmake.mk
> +include package/pkg-cargo.mk
> diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
> new file mode 100644
> index 0000000000..52c237aa3c
> --- /dev/null
> +++ b/package/pkg-cargo.mk
> @@ -0,0 +1,123 @@
> +################################################################################
> +# Cargo package infrastructure
> +#
> +# This file implements an infrastructure that eases development of package
> +# .mk files for Cargo packages. It should be used for all packages that use
> +# Cargo as their build system.
> +#
> +# See the Buildroot documentation for details on the usage of this
> +# infrastructure
> +#
> +# In terms of implementation, this Cargo infrastructure requires the .mk file
> +# to only specify metadata information about the package: name, version,
> +# download URL, etc.
> +#
> +# We still allow the package .mk file to override what the different steps
> +# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined,
> +# it is used as the list of commands to perform to build the package,
> +# instead of the default Cargo behaviour. The package can also define some
> +# post operation hooks.
> +#
> +################################################################################
> +
> +################################################################################
> +# inner-cargo-package -- defines how the configuration, compilation and
> +# installation of a cargo package should be done, implements a few hooks
> +# to tune the build process for cargo specifities and calls the generic
> +# package infrastructure to generate the necessary make targets
> +#
> +#  argument 1 is the lowercase package name
> +#  argument 2 is the uppercase package name, including a HOST_ prefix
> +#             for host packages
> +#  argument 3 is the uppercase package name, without the HOST_ prefix
> +#             for host packages
> +#  argument 4 is the type (target or host)
> +################################################################################
> +
> +define inner-cargo-package
> +
> +# We need host-rustc to run cargo
> +$(2)_DEPENDENCIES += host-rustc
> +
> +$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
> +
> +#
> +# Build step. Only define it if not already defined by the package .mk
> +# file.
> +#
> +ifndef $(2)_BUILD_CMDS
> +ifeq ($(4),target)
> +define $(2)_BUILD_CMDS
> +       $$(TARGET_MAKE_ENV) \
> +               $$(TARGET_CONFIGURE_OPTS) \
> +               $$($(2)_CARGO_ENV) \
> +               cargo build \
> +                       --offline \
> +                       --target $$(RUSTC_TARGET_NAME) \
> +                       $$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
> +                       --manifest-path $$(@D)/Cargo.toml \
> +                       --locked \
> +                       $$($(2)_CARGO_BUILD_OPTS)
> +endef
> +else # ifeq ($(4),target)
> +define $(2)_BUILD_CMDS
> +       $$(HOST_MAKE_ENV) \
> +               RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
> +               $$($(2)_CARGO_ENV) \
> +               cargo build \
> +                       --offline \
> +                       --release \
> +                       --manifest-path $$(@D)/Cargo.toml \
> +                       --locked \
> +                       $$($(2)_CARGO_BUILD_OPTS)
> +endef
> +endif # ifeq ($(4),target)
> +endif # ifndef $(2)_BUILD_CMDS
> +
> +#
> +# Target installation step. Only define it if not already defined by
> +# the package .mk file.
> +#
> +ifndef $(2)_INSTALL_TARGET_CMDS
> +define $(2)_INSTALL_TARGET_CMDS
> +       $$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
> +               cargo install \
> +                       --target $$(RUSTC_TARGET_NAME) \
> +                       --offline \
> +                       --root $$(TARGET_DIR)/usr/ \
> +                       --bins \
> +                       --path $$(@D) \
> +                       --force \
> +                       --locked \
> +                       $$($(2)_CARGO_INSTALL_OPTS)
> +endef
> +endif
> +
> +ifndef $(2)_INSTALL_CMDS
> +define $(2)_INSTALL_CMDS
> +       $$(HOST_MAKE_ENV) \
> +               RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
> +               $$($(2)_CARGO_ENV) \
> +               cargo install \
> +                       --offline \
> +                       --root $$(HOST_DIR) \
> +                       --bins \
> +                       --path $$(@D) \
> +                       --force \
> +                       --locked \
> +                       $$($(2)_CARGO_INSTALL_OPTS)
> +endef
> +endif
> +
> +# Call the generic package infrastructure to generate the necessary
> +# make targets
> +$(call inner-generic-package,$(1),$(2),$(3),$(4))
> +
> +endef
> +
> +################################################################################
> +# cargo-package -- the target generator macro for Cargo packages
> +################################################################################
> +
> +cargo-package = $(call inner-cargo-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)
> +host-cargo-package = $(call inner-cargo-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host)
> --
> 2.28.0
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart]

Hi Thomas,

On Thu, Nov 19, 2020 at 1:37 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> This commit introduces the download post-process script
> support/download/go-post-process, and hooks it into the Go package
> infrastructure.

> +++ b/support/download/go-post-process
> @@ -0,0 +1,29 @@
> +#!/usr/bin/env bash
> +
> +set -e
> +
> +. $(dirname $0)/post-process-helpers
> +
> +# Parse our options
> +while getopts "n:o:" OPT; do
> +        case "${OPT}" in
> +        o)  output="${OPTARG}";;
> +        n)  base_name="${OPTARG}";;
> +        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
> +        \?) error "unknown option '%s'\n" "${OPTARG}";;
> +        esac
> +done
> +
> +# Already vendored tarball, nothing to do
> +if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
> +       exit 0
> +fi
> +
> +unpack ${base_name} ${output}
> +
> +# Do the Go vendoring
> +pushd ${base_name} > /dev/null
> +go mod vendor -v
> +popd > /dev/null
> +
> +repack ${base_name} ${output}

This looks good. There's only one problem - if the package does not
come with a go.mod file (which is the case with
nvidia-container-toolkit).

In this case we need to run the $(2)_GEN_GOMOD script before running
the download post-process.

Today, this is written as so and creates a go.mod if it doesn't already exist:

# Generate a go.mod file if it doesn't exist. Note: Go is configured
# to use the "vendor" dir and not make network calls.
define $(2)_GEN_GOMOD
if [ ! -f $$(@D)/go.mod ]; then \
  printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \
fi
endef
$(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD

Otherwise it looks like it works well, with this one tweak.

Best regards,
Christian Stewart

[Buildroot] [PATCH next 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure

[Thomas Petazzoni]

Hello Ryan,

On Wed, 25 Nov 2020 13:46:38 -0600
Ryan Barnett <ryanbarnett3@gmail.com> wrote:

> error: no matching package named `aho-corasick` found
> location searched: registry `https://github.com/rust-lang/crates.io-index`
> required by package `globset v0.4.5
> (/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/crates/globset)`
> As a reminder, you're using offline mode (--offline) which can
> sometimes cause surprising resolution failures, if this error is too
> confusing you may wish to retry without the offline flag.
> make[1]: *** [package/pkg-generic.mk:250:
> /home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/.stamp_built]
> Error 101
> make: *** [Makefile:84: _all] Error 2
> 
> I get this error by running the following commands:
> 
> make ripgrep
> make clean ripgrep

[...]

> So it appears that after the rippackage has been downloaded there is
> an issue with doing a clean build. I'm not familiar with cargo/rust
> builds and how the offline component works but it appears that the
> introduction of this cargo infrastructure and the downloading of
> vendor dependencies is not currently reproducible.

Thanks for the report, I'll try to have a look as time permits. I'm
also not that familiar with the Cargo stuff, but we'll see if I can
find something.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Thomas Petazzoni]

Hello Christian,

Thanks for the feedback!

On Wed, 25 Nov 2020 12:28:31 -0800
Christian Stewart <christian@paral.in> wrote:

> This looks good. There's only one problem - if the package does not
> come with a go.mod file (which is the case with
> nvidia-container-toolkit).
> 
> In this case we need to run the $(2)_GEN_GOMOD script before running
> the download post-process.
> 
> Today, this is written as so and creates a go.mod if it doesn't already exist:
> 
> # Generate a go.mod file if it doesn't exist. Note: Go is configured
> # to use the "vendor" dir and not make network calls.
> define $(2)_GEN_GOMOD
> if [ ! -f $$(@D)/go.mod ]; then \
>   printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \
> fi
> endef
> $(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD

Ah, good point. Do you have an example of such a package (publicly
available)? That would help testing a solution.

I guess the best solution is to modify the POST_PATCH_HOOKS to be done
by the download post-process script ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart]

Hi Thomas,


On Wed, Nov 25, 2020 at 12:28 PM Christian Stewart <christian@paral.in> wrote:
> On Thu, Nov 19, 2020 at 1:37 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> >
> > This commit introduces the download post-process script
> > support/download/go-post-process, and hooks it into the Go package
> > infrastructure.
>
> This looks good. There's only one problem - if the package does not
> come with a go.mod file (which is the case with
> nvidia-container-toolkit).
>
> In this case we need to run the $(2)_GEN_GOMOD script before running
> the download post-process.

I was able to solve this with the following patch:

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index c914d016e2..29462ebaa4 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -112,6 +112,7 @@ define DOWNLOAD
  -o '$($(2)_DL_DIR)/$(notdir $(1))' \
  $(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
  $(if $($(2)_GIT_SUBMODULES),-r) \
+ $(if $($(2)_GOMOD),-g '$($(2)_GOMOD)') \
  $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
  $(QUIET) \
  -- \
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 2d74554213..2fc530f24f 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -17,7 +17,7 @@
 # We want to catch any unexpected failure, and exit immediately.
 set -e

-export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
+export BR_BACKEND_DL_GETOPTS=":hc:d:g:o:n:N:H:ru:qf:e"

 main() {
     local OPT OPTARG
@@ -25,11 +25,12 @@ main() {
     local -a uris

     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
+    while getopts ":c:d:D:g:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
         D)  old_dl_dir="${OPTARG}";;
+        g)  gomod_init="${OPTARG}";;
         o)  output="${OPTARG}";;
         n)  raw_base_name="${OPTARG}";;
         N)  base_name="${OPTARG}";;
@@ -138,6 +139,7 @@ main() {

         if [ -n "${post_process}" ] ; then
                 ${OLDPWD}/support/download/${post_process}-post-process \
+                         -g "${gomod_init}" \
                          -o "${tmpf}" \
                          -n "${raw_base_name}"
         fi
diff --git a/support/download/go-post-process b/support/download/go-post-process
index 01073aee8b..796fa08e38 100755
--- a/support/download/go-post-process
+++ b/support/download/go-post-process
@@ -5,8 +5,9 @@ set -e
 . $(dirname $0)/post-process-helpers

 # Parse our options
-while getopts "n:o:" OPT; do
+while getopts "n:g:o:" OPT; do
         case "${OPT}" in
+        g)  gomod_init="${OPTARG}";;
         o)  output="${OPTARG}";;
         n)  base_name="${OPTARG}";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
@@ -23,6 +24,10 @@ unpack ${base_name} ${output}

 # Do the Go vendoring
 pushd ${base_name} > /dev/null
+if [ -n "${gomod_init}" ]; then
+    # Note: does nothing if go.mod already exists.
+    go mod init ${gomod_init}
+fi
 go mod vendor -v
 popd > /dev/null

Not sure if this is the best way to pass the go module identifier into
the script, but running "go mod init" will conditionally create the
go.mod file if it doesn't already exist, just before running "go mod
vendor."

Best,
Christian

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart]

Hi Thomas,

There are a number of packages that don't have go.mod files yet, but
might have files from other package managers in the Go ecosystem.

On Wed, Nov 25, 2020 at 12:37 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Christian,
>
> Thanks for the feedback!
>
> On Wed, 25 Nov 2020 12:28:31 -0800
> Christian Stewart <christian@paral.in> wrote:
>
> > This looks good. There's only one problem - if the package does not
> > come with a go.mod file (which is the case with
> > nvidia-container-toolkit).
> >
> > In this case we need to run the $(2)_GEN_GOMOD script before running
> > the download post-process.
> >
> > Today, this is written as so and creates a go.mod if it doesn't already exist:
> >
> > # Generate a go.mod file if it doesn't exist. Note: Go is configured
> > # to use the "vendor" dir and not make network calls.
> > define $(2)_GEN_GOMOD
> > if [ ! -f $$(@D)/go.mod ]; then \
> >   printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \
> > fi
> > endef
> > $(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD
>
> Ah, good point. Do you have an example of such a package (publicly
> available)? That would help testing a solution.

Yes: https://github.com/NVIDIA/nvidia-container-runtime

Buildroot package:
https://github.com/skiffos/buildroot/commit/983f30d31c7fc8011fbfb57dd3abef1751c3e4d7

And some more like it in this tree:

https://github.com/skiffos/buildroot/tree/pkg-mgr

>
> I guess the best solution is to modify the POST_PATCH_HOOKS to be done
> by the download post-process script ?

The go module system will actually convert those other formats
faithfully into the go.mod and go.sum format.

If we add the patch that I sent in the previous email, which runs "go
mod init" just before "go mod vendor," it will read the legacy formats
in this step.

Alternatively we could do something which was rejected in one of my
original go.mod support revisions - adding the go.mod and go.sum files
into the Buildroot tree.

To generate these:

 1. Clone the repo
 2. "go mod init" and "go mod vendor"
 3. Copy go.mod and go.sum to buildroot tree
 4. These are copied (if exist) into the package just before running
"go mod vendor" in the post-process.

The reasoning behind this is to provide an exact version for all of
the dependencies locked in the Buildroot tree so that the output is
deterministic. We might get hash mismatches otherwise.

Best,
Christian


>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Thomas Petazzoni]

Hello,

On Wed, 25 Nov 2020 12:45:59 -0800
Christian Stewart <christian@paral.in> wrote:


> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index c914d016e2..29462ebaa4 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -112,6 +112,7 @@ define DOWNLOAD
>   -o '$($(2)_DL_DIR)/$(notdir $(1))' \
>   $(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
>   $(if $($(2)_GIT_SUBMODULES),-r) \
> + $(if $($(2)_GOMOD),-g '$($(2)_GOMOD)') \
>   $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
>   $(QUIET) \
>   -- \
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index 2d74554213..2fc530f24f 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -17,7 +17,7 @@
>  # We want to catch any unexpected failure, and exit immediately.
>  set -e
> 
> -export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
> +export BR_BACKEND_DL_GETOPTS=":hc:d:g:o:n:N:H:ru:qf:e"
> 
>  main() {
>      local OPT OPTARG
> @@ -25,11 +25,12 @@ main() {
>      local -a uris
> 
>      # Parse our options; anything after '--' is for the backend
> -    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
> +    while getopts ":c:d:D:g:o:n:N:H:rf:u:qp:" OPT; do
>          case "${OPT}" in
>          c)  cset="${OPTARG}";;
>          d)  dl_dir="${OPTARG}";;
>          D)  old_dl_dir="${OPTARG}";;
> +        g)  gomod_init="${OPTARG}";;

Unfortunately, this is not really a very good solution, because the
dl-wrapper is generic, and if we start adding language-specific and
backend-specific options, it's really going to become a nightmare :-/

>  # Do the Go vendoring
>  pushd ${base_name} > /dev/null
> +if [ -n "${gomod_init}" ]; then
> +    # Note: does nothing if go.mod already exists.
> +    go mod init ${gomod_init}

So this cannot simply be conditional on go.mod existing or not, because
we need the name of the Go module ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart]

Hi Thomas,

On Wed, Nov 25, 2020 at 12:52 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> >
> >      # Parse our options; anything after '--' is for the backend
> > -    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
> > +    while getopts ":c:d:D:g:o:n:N:H:rf:u:qp:" OPT; do
> >          case "${OPT}" in
> >          c)  cset="${OPTARG}";;
> >          d)  dl_dir="${OPTARG}";;
> >          D)  old_dl_dir="${OPTARG}";;
> > +        g)  gomod_init="${OPTARG}";;
>
> Unfortunately, this is not really a very good solution, because the
> dl-wrapper is generic, and if we start adding language-specific and
> backend-specific options, it's really going to become a nightmare :-/

Pass it as an environment variable? I don't see what the problem is
aside from the optargs.

> >  # Do the Go vendoring
> >  pushd ${base_name} > /dev/null
> > +if [ -n "${gomod_init}" ]; then
> > +    # Note: does nothing if go.mod already exists.
> > +    go mod init ${gomod_init}
>
> So this cannot simply be conditional on go.mod existing or not, because
> we need the name of the Go module ?

The "go mod init" command will actually do nothing if go.mod already exists.

We do need the name of the go module though in the case that go.mod
does not exist.

Best regards,
Christian

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart]

Thomas,

On Wed, Nov 25, 2020 at 12:54 PM Christian Stewart <christian@paral.in> wrote:
> > So this cannot simply be conditional on go.mod existing or not, because
> > we need the name of the Go module ?
>
> The "go mod init" command will actually do nothing if go.mod already exists.

Scratch that, "go mod init" will actually fail if go.mod already
exists. So we should instead:

if [ ! -f go.mod ] && [ -n "${gomod_init}" ]; then
    go mod init ${gomod_init}
fi

Apologies for confusion on that.

Best,
Christian

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Ryan Barnett]

Thomas, All,

I've tried out the new go vendoring support and I am running into an
issue with cleaning the build directory when using this series. I've
added a new package on your pkg-mgr branch called maddy with the
following at package/maddy/maddy.mk:

MADDY_VERSION = 0.4.2
MADDY_SITE = $(call github,foxcpp,maddy,v$(MADDY_VERSION))

$(eval $(golang-package))

Then I run the following commands:

make maddy
make clean

The error I get is as follows:

rm -rf /home/ryan/projects/br/br-pkg-mgr/output/target
/home/ryan/projects/br/br-pkg-mgr/output/images
/home/ryan/projects/br/br-pkg-mgr/output/host  \
        /home/ryan/projects/br/br-pkg-mgr/output/build
/home/ryan/projects/br/br-pkg-mgr/output/staging \
        /home/ryan/projects/br/br-pkg-mgr/output/legal-info
/home/ryan/projects/br/br-pkg-mgr/output/graphs
/home/ryan/projects/br/br-pkg-mgr/output/per-package
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/README.md':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_amd64.go':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_amd64.s':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_test.go':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash.go':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/go.mod':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhsum/xxhsum.go':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhsum/.gitignore':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe.go':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_other.go':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/LICENSE.txt':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/go.sum':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/.travis.yml':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_safe.go':
Permission denied
rm: cannot remove
'/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go':
Permission denied

There are about 1000 more lines with this error. The only way I can do
a clean is by running:

sudo rm -rf output/*

Here are the permissions on one of the files:

$ ls -al /home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go
-r--r--r-- 1 ryan ryan 402 Nov 25 15:01
/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go

I've tried that same package on the latest version of master and I do
not see this error. With that same maddy.mk file on master I am able
to successfully build the package and run `make clean`. I believe the
error I see it is due to the vendoring of go modules. Before today,
I've never build a go package so I'm not familiar with how those files
could be have incorrect permissions.

Thanks,
-Ryan

On Thu, Nov 19, 2020 at 3:37 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> This commit introduces the download post-process script
> support/download/go-post-process, and hooks it into the Go package
> infrastructure.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-golang.mk            |  7 ++++++-
>  support/download/go-post-process | 29 +++++++++++++++++++++++++++++
>  2 files changed, 35 insertions(+), 1 deletion(-)
>  create mode 100755 support/download/go-post-process

[...]

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart]

Hi Ryan, all,

On Wed, Nov 25, 2020 at 1:07 PM Ryan Barnett <ryanbarnett3@gmail.com> wrote:
> I've tried out the new go vendoring support and I am running into an
> issue with cleaning the build directory when using this series. I've
> added a new package on your pkg-mgr branch called maddy with the
> following at package/maddy/maddy.mk:

> make maddy
> make clean

> The error I get is as follows:
>
> rm -rf /home/ryan/projects/br/br-pkg-mgr/output/target
> /home/ryan/projects/br/br-pkg-mgr/output/images
> /home/ryan/projects/br/br-pkg-mgr/output/host  \
>         /home/ryan/projects/br/br-pkg-mgr/output/build
> /home/ryan/projects/br/br-pkg-mgr/output/staging \
>         /home/ryan/projects/br/br-pkg-mgr/output/legal-info
> /home/ryan/projects/br/br-pkg-mgr/output/graphs
> /home/ryan/projects/br/br-pkg-mgr/output/per-package
> rm: cannot remove
> '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/README.md':
> Permission denied

> There are about 1000 more lines with this error. The only way I can do
> a clean is by running:
>
> sudo rm -rf output/*
>
> Here are the permissions on one of the files:
>
> $ ls -al /home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go
> -r--r--r-- 1 ryan ryan 402 Nov 25 15:01
> /home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go

This is because the Go tool sets restrictive permissions on the
gopath/pkg/mod tree to prevent source code tools from editing the
dependencies code on the filesystem.

Relevant Go issues:

 - https://github.com/golang/go/issues/27455

myitcv says: "This is, as @mark-rushakoff pointed out, working as
intended. The integrity of pkg/mod/** is important. Hence it is not
writable."

 - https://github.com/golang/go/issues/31481

We can solve this by adding "-modcacherw" flag to the "go mod"
commands. I will test this and send some more info.

Best regards,
Christian Stewart

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Christian Stewart]

Hi Ryan, Thomas, all,

I have a modified version of the go-post-process patch with fixes here:

https://github.com/skiffos/buildroot/commit/5f31774f54402f07c1d83aa407430db5152e6de3

On Wed, Nov 25, 2020 at 1:12 PM Christian Stewart <christian@paral.in> wrote:
> On Wed, Nov 25, 2020 at 1:07 PM Ryan Barnett <ryanbarnett3@gmail.com> wrote:
> > I've tried out the new go vendoring support and I am running into an
> > issue with cleaning the build directory when using this series. I've
> > added a new package on your pkg-mgr branch called maddy with the
> > following at package/maddy/maddy.mk:
>
> > make maddy
> > make clean
>
> > The error I get is as follows:
> >
> > rm -rf /home/ryan/projects/br/br-pkg-mgr/output/target
> > /home/ryan/projects/br/br-pkg-mgr/output/images
> > /home/ryan/projects/br/br-pkg-mgr/output/host  \
> >         /home/ryan/projects/br/br-pkg-mgr/output/build
> > /home/ryan/projects/br/br-pkg-mgr/output/staging \
> >         /home/ryan/projects/br/br-pkg-mgr/output/legal-info
> > /home/ryan/projects/br/br-pkg-mgr/output/graphs
> > /home/ryan/projects/br/br-pkg-mgr/output/per-package
> > rm: cannot remove
> > '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/README.md':
> > Permission denied

Here is the fix for this issue specifically (using the modcacherw flag):

diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
index 150445bf17..b22720add6 100644
--- a/package/pkg-golang.mk
+++ b/package/pkg-golang.mk
@@ -42,6 +42,7 @@ define inner-golang-package

 $(2)_BUILD_OPTS += \
  -ldflags "$$($(2)_LDFLAGS)" \
+ -modcacherw \
  -tags "$$($(2)_TAGS)" \
  -trimpath \
  -p $(PARALLEL_JOBS)
diff --git a/support/download/go-post-process b/support/download/go-post-process
index 8830ee56c1..1e5441f6e6 100755
--- a/support/download/go-post-process
+++ b/support/download/go-post-process
@@ -24,10 +24,12 @@ unpack ${base_name} ${output}

 # Do the Go vendoring
 pushd ${base_name} > /dev/null
+# modcacherw option leaves directories in the module cache at their default
+# permissions rather than making them read-only.
 if [ ! -f go.mod ] && [ -n "${gomod_init}" ]; then
-    go mod init ${gomod_init}
+    go mod init -modcacherw ${gomod_init}
 fi
-go mod vendor -v
+go mod vendor -modcacherw -v
 popd > /dev/null

 repack ${base_name} ${output}

Best regards,
Christian Stewart

[Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support

[Ryan Barnett]

Christian, Thomas,

On Wed, Nov 25, 2020 at 3:21 PM Christian Stewart <christian@paral.in> wrote:
>
> Hi Ryan, Thomas, all,
>
> I have a modified version of the go-post-process patch with fixes here:
>
> https://github.com/skiffos/buildroot/commit/5f31774f54402f07c1d83aa407430db5152e6de3
>
> On Wed, Nov 25, 2020 at 1:12 PM Christian Stewart <christian@paral.in> wrote:
> > On Wed, Nov 25, 2020 at 1:07 PM Ryan Barnett <ryanbarnett3@gmail.com> wrote:
> > > I've tried out the new go vendoring support and I am running into an
> > > issue with cleaning the build directory when using this series. I've
> > > added a new package on your pkg-mgr branch called maddy with the
> > > following at package/maddy/maddy.mk:
> >
> > > make maddy
> > > make clean
> >
> > > The error I get is as follows:
> > >
> > > rm -rf /home/ryan/projects/br/br-pkg-mgr/output/target
> > > /home/ryan/projects/br/br-pkg-mgr/output/images
> > > /home/ryan/projects/br/br-pkg-mgr/output/host  \
> > >         /home/ryan/projects/br/br-pkg-mgr/output/build
> > > /home/ryan/projects/br/br-pkg-mgr/output/staging \
> > >         /home/ryan/projects/br/br-pkg-mgr/output/legal-info
> > > /home/ryan/projects/br/br-pkg-mgr/output/graphs
> > > /home/ryan/projects/br/br-pkg-mgr/output/per-package
> > > rm: cannot remove
> > > '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/README.md':
> > > Permission denied
>
> Here is the fix for this issue specifically (using the modcacherw flag):

[...]

I manually added the '-modcacherw' flag to the pkg-golang.mk and
go-post-process files and I was able to now successfully run 'make
clean'

Thanks for the quick response and the fix.

-Ryan

[Buildroot] [PATCH 1/1] package/pkg-cargo.mk: make sure .cargo/config is used

[Patrick Havelange]

The way cargo searches for config file is based (among other rules)
on the current directory. This means that if cargo is started outside
of a package directory, its configuration file will not be taken
into account.
So just 'cd' into it before running cargo build/install

This fix the issue where the build is failing in offline mode and
only the dl/package.tar.gz is available.

Without this, it would have worked only if the CARGO_HOME cache had
been populated (by running cargo vendor for example).

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
---
 package/pkg-cargo.mk | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
index 6653f64594..2047aa80cc 100644
--- a/package/pkg-cargo.mk
+++ b/package/pkg-cargo.mk
@@ -51,6 +51,7 @@ $(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo
 ifndef $(2)_BUILD_CMDS
 ifeq ($(4),target)
 define $(2)_BUILD_CMDS
+	cd $$(@D) && \
 	$$(TARGET_MAKE_ENV) \
 		$$(TARGET_CONFIGURE_OPTS) \
 		$$($(2)_CARGO_ENV) \
@@ -58,19 +59,20 @@ define $(2)_BUILD_CMDS
 			--offline \
 			--target $$(RUSTC_TARGET_NAME) \
 			$$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
-			--manifest-path $$(@D)/Cargo.toml \
+			--manifest-path Cargo.toml \
 			--locked \
 			$$($(2)_CARGO_BUILD_OPTS)
 endef
 else # ifeq ($(4),target)
 define $(2)_BUILD_CMDS
+	cd $$(@D) && \
 	$$(HOST_MAKE_ENV) \
 		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
 		$$($(2)_CARGO_ENV) \
 		cargo build \
 			--offline \
 			--release \
-			--manifest-path $$(@D)/Cargo.toml \
+			--manifest-path Cargo.toml \
 			--locked \
 			$$($(2)_CARGO_BUILD_OPTS)
 endef
@@ -83,13 +85,14 @@ endif # ifndef $(2)_BUILD_CMDS
 #
 ifndef $(2)_INSTALL_TARGET_CMDS
 define $(2)_INSTALL_TARGET_CMDS
+	cd $$(@D) && \
 	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
 		cargo install \
 			--target $$(RUSTC_TARGET_NAME) \
 			--offline \
 			--root $$(TARGET_DIR)/usr/ \
 			--bins \
-			--path $$(@D) \
+			--path ./ \
 			--force \
 			--locked \
 			$$($(2)_CARGO_INSTALL_OPTS)
@@ -98,6 +101,7 @@ endif
 
 ifndef $(2)_INSTALL_CMDS
 define $(2)_INSTALL_CMDS
+	cd $$(@D) && \
 	$$(HOST_MAKE_ENV) \
 		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
 		$$($(2)_CARGO_ENV) \
@@ -105,7 +109,7 @@ define $(2)_INSTALL_CMDS
 			--offline \
 			--root $$(HOST_DIR) \
 			--bins \
-			--path $$(@D) \
+			--path ./ \
 			--force \
 			--locked \
 			$$($(2)_CARGO_INSTALL_OPTS)
-- 
2.17.1

[Buildroot] [PATCH 1/1] package/pkg-cargo.mk: make sure .cargo/config is used

[Thomas Petazzoni]

Hello Patrick,

On Thu, 10 Dec 2020 16:48:50 +0100
Patrick Havelange <patrick.havelange@essensium.com> wrote:

> The way cargo searches for config file is based (among other rules)
> on the current directory. This means that if cargo is started outside
> of a package directory, its configuration file will not be taken
> into account.
> So just 'cd' into it before running cargo build/install
> 
> This fix the issue where the build is failing in offline mode and
> only the dl/package.tar.gz is available.
> 
> Without this, it would have worked only if the CARGO_HOME cache had
> been populated (by running cargo vendor for example).
> 
> Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>

Thanks a lot! This does fix the "make ripgrep; make clean ripgrep"
issue that Ryan reported. Also, for ripgrep, I no longer see anything
being rebuilt during the install step, which is good.

*However*, for host-sentry-cli, even with this patch, quite a bit of
stuff gets rebuilt during the install step. Here is how it goes:

>> host-sentry-cli 1.59.0 Extracting
gzip -d -c /home/thomas/dl/sentry-cli/sentry-cli-1.59.0.tar.gz | tar --strip-components=1 -C /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0   -xf -
>>> host-sentry-cli 1.59.0 Patching
>>> host-sentry-cli 1.59.0 Configuring
>>> host-sentry-cli 1.59.0 Building
cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo build --offline --release --manifest-path Cargo.toml --locked
  
   Compiling libc v0.2.71
   Compiling cfg-if v0.1.10
   Compiling autocfg v1.0.0
   Compiling proc-macro2 v1.0.18
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.33
   Compiling serde_derive v1.0.114
   Compiling serde v1.0.114
   Compiling lazy_static v1.4.0
   Compiling pkg-config v0.3.17
   Compiling getrandom v0.1.14
   Compiling memchr v2.3.3
   Compiling ppv-lite86 v0.2.8
   Compiling fallible-iterator v0.2.0
   Compiling regex-syntax v0.6.18
   Compiling byteorder v1.3.4
   Compiling stable_deref_trait v1.1.1
   Compiling adler32 v1.1.0
   Compiling rand_core v0.4.2
   Compiling autocfg v0.1.7
   Compiling failure_derive v0.1.8
   Compiling log v0.4.8
   Compiling scopeguard v1.1.0
   Compiling ryu v1.0.5
   Compiling rustc-demangle v0.1.16
   Compiling tinyvec v0.3.3
   Compiling object v0.20.0
   Compiling matches v0.1.8
   Compiling semver-parser v0.7.0
   Compiling maybe-uninit v2.0.0
   Compiling itoa v0.4.6
   Compiling serde_json v1.0.56
   Compiling proc-macro2 v0.4.30
   Compiling encoding_index_tests v0.1.4
   Compiling typenum v1.12.0
   Compiling unicode-xid v0.1.0
   Compiling ucd-trie v0.1.3
   Compiling crc32fast v1.2.0
   Compiling smallvec v1.4.0
   Compiling syn v0.15.44
   Compiling maplit v1.0.2
   Compiling unicode-width v0.1.8
   Compiling adler v0.2.2
   Compiling version_check v0.9.2
   Compiling percent-encoding v2.1.0
   Compiling bitflags v1.2.1
   Compiling rayon-core v1.7.1
   Compiling curl v0.4.30
   Compiling plain v0.2.3
   Compiling podio v0.1.7
   Compiling ahash v0.3.8
   Compiling siphasher v0.3.3
   Compiling fnv v1.0.7
   Compiling xml-rs v0.8.3
   Compiling dmsort v1.0.0
   Compiling new_debug_unreachable v1.0.4
   Compiling openssl-probe v0.1.2
   Compiling arc-swap v0.4.7
   Compiling match_cfg v0.1.0
   Compiling safemem v0.3.3
   Compiling percent-encoding v1.0.1
   Compiling lazycell v1.2.1
   Compiling either v1.5.3
   Compiling precomputed-hash v0.1.1
   Compiling same-file v1.0.6
   Compiling sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
   Compiling xdg v2.2.0
   Compiling number_prefix v0.3.0
   Compiling encode_unicode v0.3.6
   Compiling httpdate v0.3.2
   Compiling instant v0.1.5
   Compiling if_chain v0.1.3
   Compiling strsim v0.8.0
   Compiling chardet v0.2.4
   Compiling glob v0.3.0
   Compiling dotenv v0.15.0
   Compiling username v0.2.0
   Compiling open v1.4.0
   Compiling if_chain v1.0.0
   Compiling thread_local v1.0.1
   Compiling crossbeam-utils v0.7.2
   Compiling num-traits v0.2.12
   Compiling num-integer v0.1.43
   Compiling memoffset v0.5.4
   Compiling crossbeam-epoch v0.8.2
   Compiling hashbrown v0.7.2
   Compiling indexmap v1.4.0
   Compiling rayon v1.3.1
   Compiling gimli v0.21.0
   Compiling miniz_oxide v0.3.7
   Compiling rand_core v0.3.1
   Compiling rand_jitter v0.1.4
   Compiling rand_pcg v0.1.2
   Compiling rand_chacha v0.1.1
   Compiling rand v0.6.5
   Compiling lock_api v0.3.4
   Compiling unicode-normalization v0.1.13
   Compiling unicode-bidi v0.3.4
   Compiling semver v0.9.0
   Compiling encoding-index-japanese v1.20141219.5
   Compiling encoding-index-tradchinese v1.20141219.5
   Compiling encoding-index-simpchinese v1.20141219.5
   Compiling encoding-index-korean v1.20141219.5
   Compiling encoding-index-singlebyte v1.20141219.5
   Compiling pest v2.1.3
   Compiling miniz_oxide v0.4.0
   Compiling im v14.3.0
   Compiling phf_shared v0.8.0
   Compiling line-wrap v0.1.1
   Compiling itertools v0.9.0
   Compiling walkdir v2.3.1
   Compiling app_dirs v1.2.1
   Compiling rand_hc v0.1.0
   Compiling rand_xorshift v0.1.1
   Compiling rand_isaac v0.1.1
   Compiling idna v0.2.0
   Compiling idna v0.1.5
   Compiling rustc_version v0.2.3
   Compiling addr2line v0.12.2
   Compiling encoding v0.2.33
   Compiling pest_meta v2.1.3
   Compiling time v0.1.43
   Compiling parking_lot_core v0.7.2
   Compiling memmap v0.7.0
   Compiling socket2 v0.3.12
   Compiling rand_os v0.1.3
   Compiling term_size v0.3.2
   Compiling num_cpus v1.13.0
   Compiling termios v0.3.2
   Compiling dirs v1.0.5
   Compiling terminal_size v0.1.12
   Compiling uname v0.1.1
   Compiling atty v0.2.14
   Compiling signal-hook-registry v1.2.0
   Compiling dirs-sys v0.3.5
   Compiling hostname v0.3.1
   Compiling jobserver v0.1.21
   Compiling quote v1.0.7
   Compiling aho-corasick v0.7.13
   Compiling csv-core v0.1.10
   Compiling regex-automata v0.1.9
   Compiling base64 v0.10.1
   Compiling backtrace v0.3.49
   Compiling url v1.7.2
   Compiling scroll v0.9.2
   Compiling sentry v0.18.1
   Compiling quote v0.6.13
   Compiling bitmaps v2.1.0
   Compiling flate2 v1.0.16
   Compiling crossbeam-queue v0.2.3
   Compiling crossbeam-channel v0.4.2
   Compiling rand_core v0.5.1
   Compiling parking_lot v0.10.2
   Compiling textwrap v0.11.0
   Compiling term v0.5.2
   Compiling signal-hook v0.1.16
   Compiling dirs v2.0.2
   Compiling cc v1.0.56
   Compiling regex v1.3.9
   Compiling sized-chunks v0.5.3
   Compiling rand_chacha v0.2.2
   Compiling rand_xoshiro v0.4.0
   Compiling scheduled-thread-pool v0.2.4
   Compiling clap v2.33.1
   Compiling crossbeam-deque v0.7.3
   Compiling openssl-src v111.10.0+1.1.1g
   Compiling libz-sys v1.0.25
   Compiling bzip2-sys v0.1.9+1.0.8
   Compiling curl-sys v0.4.32+curl-7.70.0
   Compiling brotli-sys v0.3.2
   Compiling libgit2-sys v0.12.7+1.0.0
   Compiling runas v0.2.1
   Compiling backoff v0.1.6
   Compiling console v0.11.3
   Compiling java-properties v1.2.0
   Compiling might-be-minified v0.3.0
   Compiling rand v0.7.3
   Compiling r2d2 v0.8.8
   Compiling synstructure v0.12.4
   Compiling pest_generator v2.1.3
   Compiling scroll_derive v0.9.5
   Compiling openssl-sys v0.9.58
   Compiling indicatif v0.14.0
   Compiling scroll_derive v0.10.2
   Compiling dlv-list v0.2.2
   Compiling pest_derive v2.1.0
   Compiling scroll v0.10.1
   Compiling ordered-multimap v0.2.4
   Compiling failure v0.1.8
   Compiling goblin v0.2.3
   Compiling sha1 v0.6.0
   Compiling bstr v0.2.13
   Compiling chrono v0.4.11
   Compiling url v2.1.1
   Compiling string_cache v0.8.0
   Compiling rust-ini v0.15.3
   Compiling which v3.1.1
   Compiling uuid v0.8.1
   Compiling csv v1.1.3
   Compiling globset v0.4.5
   Compiling sourcemap v5.0.0
   Compiling plist v0.5.5
   Compiling anylog v0.5.0
   Compiling elementtree v0.5.0
   Compiling brotli2 v0.3.2
   Compiling debugid v0.7.2
   Compiling pdb v0.6.0
   Compiling proguard v4.0.1
   Compiling prettytable-rs v0.8.0
   Compiling ignore v0.4.16
   Compiling symbolic-common v7.5.0
   Compiling sentry-types v0.14.1
   Compiling bzip2 v0.3.3
   Compiling zip v0.5.6
   Compiling symbolic-debuginfo v7.5.0
   Compiling symbolic v7.5.0
   Compiling git2 v0.13.6
    Finished release [optimized] target(s) in 6m 10s
>>> host-sentry-cli 1.59.0 Installing to host directory
cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo install --offline --root /home/thomas/projets/buildroot/outpu
 t/host --bins --path ./ --force --locked 
  Installing sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
   Compiling syn v1.0.33
   Compiling synstructure v0.12.4
   Compiling pest_generator v2.1.3
   Compiling serde_derive v1.0.114
   Compiling failure_derive v0.1.8
   Compiling scroll_derive v0.10.2
   Compiling pest_derive v2.1.0
   Compiling failure v0.1.8
   Compiling which v3.1.1
   Compiling scroll v0.10.1
   Compiling runas v0.2.1
   Compiling goblin v0.2.3
   Compiling serde v1.0.114
   Compiling sha1 v0.6.0
   Compiling serde_json v1.0.56
   Compiling bstr v0.2.13
   Compiling chrono v0.4.11
   Compiling url v2.1.1
   Compiling string_cache v0.8.0
   Compiling uuid v0.8.1
   Compiling csv v1.1.3
   Compiling globset v0.4.5
   Compiling sourcemap v5.0.0
   Compiling plist v0.5.5
   Compiling anylog v0.5.0
   Compiling git2 v0.13.6
   Compiling elementtree v0.5.0
   Compiling debugid v0.7.2
   Compiling pdb v0.6.0
   Compiling proguard v4.0.1
   Compiling prettytable-rs v0.8.0
   Compiling ignore v0.4.16
   Compiling symbolic-common v7.5.0
   Compiling sentry-types v0.14.1
   Compiling symbolic-debuginfo v7.5.0
    Building [====================================================>  ] 299/305: sentry-types, symbolic-debuginfo, ignore                                                                                                                    

   Compiling symbolic v7.5.0
   Compiling sentry v0.18.1
   Compiling sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
    Finished release [optimized] target(s) in 2m 35s
  Installing /home/thomas/projets/buildroot/output/host/bin/sentry-cli
   Installed package `sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)` (executable `sentry-cli`)

Any idea ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH next 06/12] package/tinifier: new package

[Thomas Petazzoni]

Hello,

On Sat, 21 Nov 2020 19:04:18 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> legal-info is also something Thomas and I discussed and IRC when he
> posted his series.
> 
> We know it is not perfect, but this can be extended in a followup
> series.

Right.

> 
> > When I ran 'make legal-info' for the tinifier package all that is
> > mentioned in the 'manifest.csv' file for the package is:
> > 
> >    "tinifier","2.1.0","MIT","LICENSE","tinifier-2.1.0.tar.gz","https://github.com/tarampampam/tinifier/archive/v2.1.0","skeleton-init-common
> > [unknown] skeleton-init-none [unknown] toolchain-external-bootlin
> > [unknown]"
> > 
> > This doesn't give any indication or warnings that dependencies were
> > downloaded or that other open source license could be needed by
> > including this package.  
> 
> To simplify the series, my position as a first step would be to extend
> the FOO_LICENSE list in the infra, with just a very short notice,
> something like:
> 
>     FOO_LICENSE += , vendored licenses not listed

The problem is that we do not know if the vendoring was done by
Buildroot itself, or if vendored dependencies are provided directly in
the upstream repository, so it's difficult to add this only if
Buildroot has done the vendoring.

In addition, in both cases, the FOO_LICENSE of the package may very
well be completely accurate, taking into account all vendored
dependencies. Indeed, now that they are properly downloaded, nothing
prevents from having correct FOO_LICENSE and FOO_LICENSE_FILES values
for those packages.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] package/pkg-cargo.mk: make sure .cargo/config is used

[Patrick Havelange]

On 2020-12-10 21:31, Thomas Petazzoni wrote:
> Hello Patrick,

Hi,

> 
> On Thu, 10 Dec 2020 16:48:50 +0100
> Patrick Havelange <patrick.havelange@essensium.com> wrote:
> 
>> The way cargo searches for config file is based (among other rules)
>> on the current directory. This means that if cargo is started outside
>> of a package directory, its configuration file will not be taken
>> into account.
>> So just 'cd' into it before running cargo build/install
>>
>> This fix the issue where the build is failing in offline mode and
>> only the dl/package.tar.gz is available.
>>
>> Without this, it would have worked only if the CARGO_HOME cache had
>> been populated (by running cargo vendor for example).
>>
>> Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
> 
> Thanks a lot! This does fix the "make ripgrep; make clean ripgrep"
> issue that Ryan reported. Also, for ripgrep, I no longer see anything
> being rebuilt during the install step, which is good.

glad it fixes those issues indeed.

> 
> *However*, for host-sentry-cli, even with this patch, quite a bit of
> stuff gets rebuilt during the install step. Here is how it goes:
> 
>>> host-sentry-cli 1.59.0 Extracting
> gzip -d -c /home/thomas/dl/sentry-cli/sentry-cli-1.59.0.tar.gz | tar --strip-components=1 -C /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0   -xf -
>>>> host-sentry-cli 1.59.0 Patching
>>>> host-sentry-cli 1.59.0 Configuring
>>>> host-sentry-cli 1.59.0 Building
> cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo build --offline --release --manifest-path Cargo.toml --locked
>    
>     Compiling libc v0.2.71
>     Compiling cfg-if v0.1.10
>     Compiling autocfg v1.0.0
>     Compiling proc-macro2 v1.0.18

[snip]

>     Compiling git2 v0.13.6
>      Finished release [optimized] target(s) in 6m 10s
>>>> host-sentry-cli 1.59.0 Installing to host directory
> cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo install --offline --root /home/thomas/projets/buildroot/outpu
>   t/host --bins --path ./ --force --locked
>    Installing sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
>     Compiling syn v1.0.33
>     Compiling synstructure v0.12.4
>     Compiling pest_generator v2.1.3
>     Compiling serde_derive v1.0.114
>     Compiling failure_derive v0.1.8
>     Compiling scroll_derive v0.10.2
>     Compiling pest_derive v2.1.0
>     Compiling failure v0.1.8
>     Compiling which v3.1.1
>     Compiling scroll v0.10.1
>     Compiling runas v0.2.1
>     Compiling goblin v0.2.3
>     Compiling serde v1.0.114
>     Compiling sha1 v0.6.0
>     Compiling serde_json v1.0.56
>     Compiling bstr v0.2.13
>     Compiling chrono v0.4.11
>     Compiling url v2.1.1
>     Compiling string_cache v0.8.0
>     Compiling uuid v0.8.1
>     Compiling csv v1.1.3
>     Compiling globset v0.4.5
>     Compiling sourcemap v5.0.0
>     Compiling plist v0.5.5
>     Compiling anylog v0.5.0
>     Compiling git2 v0.13.6
>     Compiling elementtree v0.5.0
>     Compiling debugid v0.7.2
>     Compiling pdb v0.6.0
>     Compiling proguard v4.0.1
>     Compiling prettytable-rs v0.8.0
>     Compiling ignore v0.4.16
>     Compiling symbolic-common v7.5.0
>     Compiling sentry-types v0.14.1
>     Compiling symbolic-debuginfo v7.5.0
>      Building [====================================================>  ] 299/305: sentry-types, symbolic-debuginfo, ignore
> 
>     Compiling symbolic v7.5.0
>     Compiling sentry v0.18.1
>     Compiling sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
>      Finished release [optimized] target(s) in 2m 35s
>    Installing /home/thomas/projets/buildroot/output/host/bin/sentry-cli
>     Installed package `sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)` (executable `sentry-cli`)
> 
> Any idea ?

Sadly no. I've spent some time trying various combinations of flags for 
cargo build & cargo install, without luck.

I've found this bug-report though : 
https://github.com/rust-lang/cargo/issues/8703 .

Since the first package being rebuild in this case is 'syn', I've looked 
at it and indeed it does make (non trivial) use of the [features] 
section inside its Cargo.toml. So maybe this can explain why syn is 
being rebuild (and also all similar packages and packages depending on 
those)?  I'm absolutely not sure of that explanation though.

> 
> Thomas
> 

Patrick H.

[Buildroot] [PATCH next 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure

[Patrick Havelange]

Hi Thomas,

On 2020-11-19 22:36, Thomas Petazzoni wrote:
> From: Patrick Havelange <patrick.havelange@essensium.com>
> 
> In order to be package agnostic, the install phase is now using cargo
> instead of install. TARGET_CONFIGURE_OPTS is now also set when running
> cargo in order to support cross compiling C code within cargo.
> 
> Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
> [Thomas: add support for host-cargo-package]
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>   package/Makefile.in  |   1 +
>   package/pkg-cargo.mk | 123 +++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 124 insertions(+)
>   create mode 100644 package/pkg-cargo.mk
> 
> diff --git a/package/Makefile.in b/package/Makefile.in
> index 51f5cbce4f..2af123f36d 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -427,3 +427,4 @@ include package/pkg-waf.mk
>   include package/pkg-golang.mk
>   include package/pkg-meson.mk
>   include package/pkg-qmake.mk
> +include package/pkg-cargo.mk
> diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
> new file mode 100644
> index 0000000000..52c237aa3c
> --- /dev/null
> +++ b/package/pkg-cargo.mk
> @@ -0,0 +1,123 @@
> +################################################################################
> +# Cargo package infrastructure
> +#
> +# This file implements an infrastructure that eases development of package
> +# .mk files for Cargo packages. It should be used for all packages that use
> +# Cargo as their build system.
> +#
> +# See the Buildroot documentation for details on the usage of this
> +# infrastructure
> +#
> +# In terms of implementation, this Cargo infrastructure requires the .mk file
> +# to only specify metadata information about the package: name, version,
> +# download URL, etc.
> +#
> +# We still allow the package .mk file to override what the different steps
> +# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined,
> +# it is used as the list of commands to perform to build the package,
> +# instead of the default Cargo behaviour. The package can also define some
> +# post operation hooks.
> +#
> +################################################################################
> +
> +################################################################################
> +# inner-cargo-package -- defines how the configuration, compilation and
> +# installation of a cargo package should be done, implements a few hooks
> +# to tune the build process for cargo specifities and calls the generic
> +# package infrastructure to generate the necessary make targets
> +#
> +#  argument 1 is the lowercase package name
> +#  argument 2 is the uppercase package name, including a HOST_ prefix
> +#             for host packages
> +#  argument 3 is the uppercase package name, without the HOST_ prefix
> +#             for host packages
> +#  argument 4 is the type (target or host)
> +################################################################################
> +
> +define inner-cargo-package
> +
> +# We need host-rustc to run cargo
> +$(2)_DEPENDENCIES += host-rustc
> +
> +$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
> +
> +#
> +# Build step. Only define it if not already defined by the package .mk
> +# file.
> +#
> +ifndef $(2)_BUILD_CMDS
> +ifeq ($(4),target)
> +define $(2)_BUILD_CMDS
> +	$$(TARGET_MAKE_ENV) \
> +		$$(TARGET_CONFIGURE_OPTS) \
> +		$$($(2)_CARGO_ENV) \
> +		cargo build \
> +			--offline \
> +			--target $$(RUSTC_TARGET_NAME) \
> +			$$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
> +			--manifest-path $$(@D)/Cargo.toml \
> +			--locked \
> +			$$($(2)_CARGO_BUILD_OPTS)
> +endef
> +else # ifeq ($(4),target)
> +define $(2)_BUILD_CMDS
> +	$$(HOST_MAKE_ENV) \
> +		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
> +		$$($(2)_CARGO_ENV) \
> +		cargo build \
> +			--offline \
> +			--release \
> +			--manifest-path $$(@D)/Cargo.toml \
> +			--locked \
> +			$$($(2)_CARGO_BUILD_OPTS)
> +endef
> +endif # ifeq ($(4),target)
> +endif # ifndef $(2)_BUILD_CMDS
> +
> +#
> +# Target installation step. Only define it if not already defined by
> +# the package .mk file.
> +#
> +ifndef $(2)_INSTALL_TARGET_CMDS
> +define $(2)_INSTALL_TARGET_CMDS
> +	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
> +		cargo install \
> +			--target $$(RUSTC_TARGET_NAME) \
> +			--offline \
> +			--root $$(TARGET_DIR)/usr/ \
> +			--bins \
> +			--path $$(@D) \
> +			--force \
> +			--locked \
> +			$$($(2)_CARGO_INSTALL_OPTS)
> +endef
> +endif
> +
> +ifndef $(2)_INSTALL_CMDS
> +define $(2)_INSTALL_CMDS
> +	$$(HOST_MAKE_ENV) \
> +		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
> +		$$($(2)_CARGO_ENV) \
> +		cargo install \
> +			--offline \
> +			--root $$(HOST_DIR) \
> +			--bins \
> +			--path $$(@D) \
> +			--force \
> +			--locked \
> +			$$($(2)_CARGO_INSTALL_OPTS)
> +endef
> +endif

Nitpick: This is the only minor thing that bothers me a bit: the 
duplication for the host/target cases of the cargo command flags. Would 
it not be better if they were merged ?



BR,

Patrick H.

[Buildroot] [PATCH next 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure

[Thomas Petazzoni]

Hello,

On Wed, 16 Dec 2020 14:58:39 +0100
Patrick Havelange <patrick.havelange@essensium.com> wrote:

> > +#
> > +# Target installation step. Only define it if not already defined by
> > +# the package .mk file.
> > +#
> > +ifndef $(2)_INSTALL_TARGET_CMDS
> > +define $(2)_INSTALL_TARGET_CMDS
> > +	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
> > +		cargo install \
> > +			--target $$(RUSTC_TARGET_NAME) \
> > +			--offline \
> > +			--root $$(TARGET_DIR)/usr/ \
> > +			--bins \
> > +			--path $$(@D) \
> > +			--force \
> > +			--locked \
> > +			$$($(2)_CARGO_INSTALL_OPTS)
> > +endef
> > +endif
> > +
> > +ifndef $(2)_INSTALL_CMDS
> > +define $(2)_INSTALL_CMDS
> > +	$$(HOST_MAKE_ENV) \
> > +		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
> > +		$$($(2)_CARGO_ENV) \
> > +		cargo install \
> > +			--offline \
> > +			--root $$(HOST_DIR) \
> > +			--bins \
> > +			--path $$(@D) \
> > +			--force \
> > +			--locked \
> > +			$$($(2)_CARGO_INSTALL_OPTS)
> > +endef
> > +endif  
> 
> Nitpick: This is the only minor thing that bothers me a bit: the 
> duplication for the host/target cases of the cargo command flags. Would 
> it not be better if they were merged ?

I agree. In fact while re-using your code and extending it, I did quite
a bit of back and forth in using variables to share options, not using
variables. I was not sure myself. But OK, if you also feel like we
should share those common options, I guess it's a sign that we should
go with that!

> BR,

Is this "Best Regards", "BuildRoot" or both ? :-)

Cheers!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] package/pkg-cargo.mk: make sure .cargo/config is used

[Sam Voss]

Hey Patrick, all,

On Wed, Dec 16, 2020 at 7:47 AM Patrick Havelange
<patrick.havelange@essensium.com> wrote:
>
> On 2020-12-10 21:31, Thomas Petazzoni wrote:
> > Hello Patrick,
>
> Hi,
>
> >
> > On Thu, 10 Dec 2020 16:48:50 +0100
> > Patrick Havelange <patrick.havelange@essensium.com> wrote:
[snip]
> > *However*, for host-sentry-cli, even with this patch, quite a bit of
> > stuff gets rebuilt during the install step. Here is how it goes:
> >
> >>> host-sentry-cli 1.59.0 Extracting
> > gzip -d -c /home/thomas/dl/sentry-cli/sentry-cli-1.59.0.tar.gz | tar --strip-components=1 -C /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0   -xf -
> >>>> host-sentry-cli 1.59.0 Patching
> >>>> host-sentry-cli 1.59.0 Configuring
> >>>> host-sentry-cli 1.59.0 Building
> > cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo build --offline --release --manifest-path Cargo.tom
>  l --locked
> >
> >     Compiling libc v0.2.71
> >     Compiling cfg-if v0.1.10
> >     Compiling autocfg v1.0.0
> >     Compiling proc-macro2 v1.0.18
>
> [snip]
>
> >     Compiling git2 v0.13.6
> >      Finished release [optimized] target(s) in 6m 10s
> >>>> host-sentry-cli 1.59.0 Installing to host directory
> > cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo install --offline --root /home/thomas/projets/build
>  root/outpu
> >   t/host --bins --path ./ --force --locked
> >    Installing sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
> >     Compiling syn v1.0.33
> >     Compiling synstructure v0.12.4
> >     Compiling pest_generator v2.1.3
> >     Compiling serde_derive v1.0.114
> >     Compiling failure_derive v0.1.8
> >     Compiling scroll_derive v0.10.2
> >     Compiling pest_derive v2.1.0
> >     Compiling failure v0.1.8
> >     Compiling which v3.1.1
> >     Compiling scroll v0.10.1
> >     Compiling runas v0.2.1
> >     Compiling goblin v0.2.3
> >     Compiling serde v1.0.114
> >     Compiling sha1 v0.6.0
> >     Compiling serde_json v1.0.56
> >     Compiling bstr v0.2.13
> >     Compiling chrono v0.4.11
> >     Compiling url v2.1.1
> >     Compiling string_cache v0.8.0
> >     Compiling uuid v0.8.1
> >     Compiling csv v1.1.3
> >     Compiling globset v0.4.5
> >     Compiling sourcemap v5.0.0
> >     Compiling plist v0.5.5
> >     Compiling anylog v0.5.0
> >     Compiling git2 v0.13.6
> >     Compiling elementtree v0.5.0
> >     Compiling debugid v0.7.2
> >     Compiling pdb v0.6.0
> >     Compiling proguard v4.0.1
> >     Compiling prettytable-rs v0.8.0
> >     Compiling ignore v0.4.16
> >     Compiling symbolic-common v7.5.0
> >     Compiling sentry-types v0.14.1
> >     Compiling symbolic-debuginfo v7.5.0
> >      Building [====================================================>  ] 299/305: sentry-types, symbolic-debuginfo, ignore
> >
> >     Compiling symbolic v7.5.0
> >     Compiling sentry v0.18.1
> >     Compiling sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
> >      Finished release [optimized] target(s) in 2m 35s
> >    Installing /home/thomas/projets/buildroot/output/host/bin/sentry-cli
> >     Installed package `sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)` (executable `sentry-cli`)
> >
> > Any idea ?
>
> Sadly no. I've spent some time trying various combinations of flags for
> cargo build & cargo install, without luck.
>
> I've found this bug-report though :
> https://github.com/rust-lang/cargo/issues/8703 .
>
> Since the first package being rebuild in this case is 'syn', I've looked
> at it and indeed it does make (non trivial) use of the [features]
> section inside its Cargo.toml. So maybe this can explain why syn is
> being rebuild (and also all similar packages and packages depending on
> those)?  I'm absolutely not sure of that explanation though.

I meant to mention this on the mailing thread but it fell between the
cracks - I tested this without buildroot and had the same outcome.
Seems like you already got to the root of "why" this happens, but just
to show it's not buildroot-centric:

sentry-cli(master)? cargo build --locked --offline --release
   Compiling libc v0.2.71
   Compiling cfg-if v0.1.10
   Compiling proc-macro2 v1.0.18
   Compiling autocfg v1.0.0
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.33
   Compiling serde_derive v1.0.114
   Compiling serde v1.0.114
   Compiling lazy_static v1.4.0
   Compiling pkg-config v0.3.17
   Compiling getrandom v0.1.14
   Compiling memchr v2.3.3
   Compiling ppv-lite86 v0.2.8
[snip compile logs]
  Compiling symbolic-debuginfo v7.5.0
(https://github.com/getsentry/symbolic#075ee512)
   Compiling symbolic v7.5.0 (https://github.com/getsentry/symbolic#075ee512)
    Finished release [optimized] target(s) in 52.72s

sentry-cli(master)? cargo install --path . --root ./exroot --locked
--offline --bins --force
  Installing sentry-cli v1.60.1
(/accts/smvoss/projects/rust-buildroot/sentry-cli)
warning: package `smallvec v1.4.0` in Cargo.lock is yanked in registry
`crates.io`, consider running without --locked
warning: package `socket2 v0.3.12` in Cargo.lock is yanked in registry
`crates.io`, consider running without --locked
   Compiling syn v1.0.33
   Compiling synstructure v0.12.4
   Compiling pest_generator v2.1.3
   Compiling serde_derive v1.0.114
[snip compile logs]
   Compiling sentry-cli v1.60.1
(/accts/smvoss/projects/rust-buildroot/sentry-cli)
    Finished release [optimized] target(s) in 44.14s
  Installing ./exroot/bin/sentry-cli
   Installed package `sentry-cli v1.60.1
(/accts/smvoss/projects/rust-buildroot/sentry-cli)` (executable
`sentry-cli`)
warning: be sure to add `./exroot/bin` to your PATH to be able to run
the installed binaries


-- 

Sam Voss | Sr. Software Engineer | Commercial Avionics
COLLINS AEROSPACE
400 Collins Road NE, Cedar Rapids, Iowa 52498, USA
Tel: +1 319 263 4039
sam.voss at collins.com | collinsaerospace.com

CONFIDENTIALITY WARNING: This message may contain proprietary and/or
privileged information of Collins Aerospace and its affiliated
companies. If you are not the intended recipient, please 1) Do not
disclose, copy, distribute or use this message or its contents. 2)
Advise the sender by return email. 3) Delete all copies (including all
attachments) from your computer. Your cooperation is greatly
appreciated.