[Buildroot] [PATCH-NEXT v3 1/6] support/download/dl-wrapper: add concept of download post-processing

[Buildroot] [PATCH-NEXT v3 1/6] support/download/dl-wrapper: add concept of download post-processing

[Christian Stewart via buildroot]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

In order to support package managers such as Cargo (Rust) or Go, we
want to run some custom logic after the main download, but before
packing the tarball and checking the hash.

To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then
support/download/<something>-post-process will be called.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Christian Stewart <christian@paral.in>
---
 support/download/dl-wrapper | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 3315bd410e..2d74554213 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -25,7 +25,7 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() {
         r)  recurse="-r";;
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
+        p)  post_process="${OPTARG}";;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -135,6 +136,12 @@ main() {
             continue
         fi
 
+        if [ -n "${post_process}" ] ; then
+                ${OLDPWD}/support/download/${post_process}-post-process \
+                         -o "${tmpf}" \
+                         -n "${raw_base_name}"
+        fi
+
         # cd back to free the temp-dir, so we can remove it later
         cd "${OLDPWD}"
 
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH-NEXT v3 2/6] package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable

[Christian Stewart via buildroot]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

This will allow packages to register than a download post-processing
is needed. Note that this variable is intentionally not documented: it
is an internal variable meant to be set by package infrastructures,
not directly by packages.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/pkg-download.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 2527ba5c60..66a415cce0 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -117,6 +117,7 @@ define DOWNLOAD
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH-NEXT v3 3/6] support/download/post-process-helpers: add helper function for post process scripts

[Christian Stewart via buildroot]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

download post process scripts will often need to unpack the source
code tarball, do some operation, and then repack it. In order to help
with this, post-process-helpers provide an unpack() function and a
repack() function.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Christian Stewart <christian@paral.in>
---
 support/download/post-process-helpers | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100755 support/download/post-process-helpers

diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers
new file mode 100755
index 0000000000..b3231207a6
--- /dev/null
+++ b/support/download/post-process-helpers
@@ -0,0 +1,26 @@
+# Helpers for the post-process step to re-pack vendored archives.
+
+. "${0%/*}/helpers"
+
+unpack() {
+    dest="$1"
+    tarball="$2"
+
+    mkdir ${dest}
+    tar -C ${dest} --strip-components=1 -xf ${tarball}
+}
+
+repack() {
+    local in_dir="${1}"
+    local base_dir="${2}"
+    local out="${3}"
+
+    # let's use a fixed hardcoded date to be reproducible
+    date="2020-02-06 01:02:03 +0000"
+
+    # use the helper to create a reproducible archive
+    mk_tar_gz "${in_dir}/${base_dir}" "${base_dir}" "${date}" "${out}"
+}
+
+# Keep this line and the following as last lines in this file.
+# vim: ft=bash
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH-NEXT v3 4/6] support/download/go-post-process: implement Go vendoring support

[Christian Stewart via buildroot]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

This commit introduces the download post-process script
support/download/go-post-process, and hooks it into the Go package
infrastructure.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Christian Stewart <christian@paral.in>

v1 -> v2:

 - re-submitting Thomas's series with adjustments:
 - run "go mod init" just before "go mod vendor"
 - this fixes the case when go.mod does not exist
 - use -modcacherw to fix "make clean" permissions errors
 - use the mk_tar_gz helper in the post-process step

Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/pkg-download.mk          |  1 +
 package/pkg-golang.mk            |  8 +++++++-
 support/download/dl-wrapper      |  6 ++++--
 support/download/go-post-process | 35 ++++++++++++++++++++++++++++++++
 4 files changed, 47 insertions(+), 3 deletions(-)
 create mode 100755 support/download/go-post-process

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 66a415cce0..378b2c66f0 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -119,6 +119,7 @@ define DOWNLOAD
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
 		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
+		$(if $($(2)_GOMOD),-g '$($(2)_GOMOD)') \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
 		-- \
diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
index d07242310d..3d2c45fa09 100644
--- a/package/pkg-golang.mk
+++ b/package/pkg-golang.mk
@@ -42,12 +42,13 @@ define inner-golang-package
 
 $(2)_BUILD_OPTS += \
 	-ldflags "$$($(2)_LDFLAGS)" \
+	-modcacherw \
 	-tags "$$($(2)_TAGS)" \
 	-trimpath \
 	-p $(PARALLEL_JOBS)
 
 # Target packages need the Go compiler on the host.
-$(2)_DEPENDENCIES += host-go
+$(2)_DOWNLOAD_DEPENDENCIES += host-go
 
 $(2)_BUILD_TARGETS ?= .
 
@@ -72,6 +73,11 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
 # If the go.mod file does not exist, one is written with this root path.
 $(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
 
+$(2)_DOWNLOAD_POST_PROCESS = go
+$(2)_DL_ENV = \
+	$(HOST_GO_COMMON_ENV) \
+	GOPROXY=direct
+
 # Generate a go.mod file if it doesn't exist. Note: Go is configured
 # to use the "vendor" dir and not make network calls.
 define $(2)_GEN_GOMOD
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 2d74554213..2fc530f24f 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -17,7 +17,7 @@
 # We want to catch any unexpected failure, and exit immediately.
 set -e
 
-export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
+export BR_BACKEND_DL_GETOPTS=":hc:d:g:o:n:N:H:ru:qf:e"
 
 main() {
     local OPT OPTARG
@@ -25,11 +25,12 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
+    while getopts ":c:d:D:g:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
         D)  old_dl_dir="${OPTARG}";;
+        g)  gomod_init="${OPTARG}";;
         o)  output="${OPTARG}";;
         n)  raw_base_name="${OPTARG}";;
         N)  base_name="${OPTARG}";;
@@ -138,6 +139,7 @@ main() {
 
         if [ -n "${post_process}" ] ; then
                 ${OLDPWD}/support/download/${post_process}-post-process \
+                         -g "${gomod_init}" \
                          -o "${tmpf}" \
                          -n "${raw_base_name}"
         fi
diff --git a/support/download/go-post-process b/support/download/go-post-process
new file mode 100755
index 0000000000..b7f05b37c1
--- /dev/null
+++ b/support/download/go-post-process
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -e
+
+. "${0%/*}/post-process-helpers"
+
+# Parse our options
+while getopts "n:g:o:" OPT; do
+        case "${OPT}" in
+        g)  gomod_init="${OPTARG}";;
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
+	exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Go vendoring
+pushd ${base_name} > /dev/null
+# modcacherw option leaves directories in the module cache at their default
+# permissions rather than making them read-only.
+if [ ! -f go.mod ] && [ -n "${gomod_init}" ]; then
+    go mod init -modcacherw ${gomod_init}
+fi
+go mod vendor -modcacherw -v
+popd > /dev/null
+
+repack $(pwd) ${base_name} ${output}
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH-NEXT v3 5/6] package/embiggen-disk: new package

[Christian Stewart via buildroot]

embiggen-disk is a tool to automatically resize disks to fill available space.

Patch submitted upstream: https://github.com/bradfitz/embiggen-disk/pull/13

Adds support for /dev/mmcblk0pN type paths.

Signed-off-by: Christian Stewart <christian@paral.in>

---

v1 -> v2:

 - BR2_TOOLCHAN_USES_UCLIBC -> BR2_TOOLCHAIN_USES_UCLIBC
 - remove old TODO comment

Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/Config.in                             |  1 +
 ...-Fix-resizing-of-dev-mmcblk0pN-paths.patch | 45 +++++++++++++++++++
 package/embiggen-disk/Config.in               | 17 +++++++
 package/embiggen-disk/embiggen-disk.hash      |  3 ++
 package/embiggen-disk/embiggen-disk.mk        | 12 +++++
 5 files changed, 78 insertions(+)
 create mode 100644 package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch
 create mode 100644 package/embiggen-disk/Config.in
 create mode 100644 package/embiggen-disk/embiggen-disk.hash
 create mode 100644 package/embiggen-disk/embiggen-disk.mk

diff --git a/package/Config.in b/package/Config.in
index 9fed0ab4cb..7a833909aa 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -2471,6 +2471,7 @@ menu "System tools"
 	source "package/earlyoom/Config.in"
 	source "package/efibootmgr/Config.in"
 	source "package/efivar/Config.in"
+	source "package/embiggen-disk/Config.in"
 	source "package/emlog/Config.in"
 	source "package/ftop/Config.in"
 	source "package/getent/Config.in"
diff --git a/package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch b/package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch
new file mode 100644
index 0000000000..31cfea9dde
--- /dev/null
+++ b/package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch
@@ -0,0 +1,45 @@
+From b628e4c561cb4d3ffc92a5d7aac8f4967e44977e Mon Sep 17 00:00:00 2001
+From: Christian Stewart <christian@paral.in>
+Date: Thu, 27 May 2021 20:07:07 -0700
+Subject: [PATCH] Fix resizing of /dev/mmcblk0pN paths
+
+Signed-off-by: Christian Stewart <christian@paral.in>
+---
+ fs.go   | 4 +++-
+ part.go | 5 +++++
+ 2 files changed, 8 insertions(+), 1 deletion(-)
+
+diff --git a/fs.go b/fs.go
+index 0b83e24..8ab3690 100644
+--- a/fs.go
++++ b/fs.go
+@@ -67,7 +67,9 @@ func (e fsResizer) DepResizer() (Resizer, error) {
+ 	if dev == "/dev/root" {
+ 		return nil, errors.New("unexpected device /dev/root from statFS")
+ 	}
+-	if (strings.HasPrefix(dev, "/dev/sd") || strings.HasPrefix(dev, "/dev/nvme")) &&
++	if (strings.HasPrefix(dev, "/dev/sd") ||
++		strings.HasPrefix(dev, "/dev/mmcblk") ||
++		strings.HasPrefix(dev, "/dev/nvme")) &&
+ 		devEndsInNumber(dev) {
+ 		vlogf("fsResizer.DepResizer: returning partitionResizer(%q)", dev)
+ 		return partitionResizer(dev), nil
+diff --git a/part.go b/part.go
+index 0315a6e..f3e280f 100644
+--- a/part.go
++++ b/part.go
+@@ -51,6 +51,11 @@ func diskDev(partDev string) string {
+ 	if strings.HasPrefix(partDev, "/dev/sd") {
+ 		return strings.TrimRight(partDev, "0123456789")
+ 	}
++	if strings.HasPrefix(partDev, "/dev/mmcblk") {
++		v := strings.TrimRight(partDev, "0123456789")
++		v = strings.TrimSuffix(v, "p")
++		return v
++	}
+ 	if strings.HasPrefix(partDev, "/dev/nvme") {
+ 		chopP := regexp.MustCompile(`p\d+$`)
+ 		if !chopP.MatchString(partDev) {
+-- 
+2.31.1
+
diff --git a/package/embiggen-disk/Config.in b/package/embiggen-disk/Config.in
new file mode 100644
index 0000000000..c0e19b104e
--- /dev/null
+++ b/package/embiggen-disk/Config.in
@@ -0,0 +1,17 @@
+config BR2_PACKAGE_EMBIGGEN_DISK
+	bool "embiggen-disk"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
+	depends on BR2_TOOLCHAIN_HAS_THREADS
+	depends on !BR2_TOOLCHAIN_USES_UCLIBC # no fexecve
+	select BR2_PACKAGE_UTIL_LINUX # sfdisk
+	select BR2_PACKAGE_UTIL_LINUX_BINARIES # sfdisk
+	help
+	  embiggen-disk is a tool to resize disk partitions at runtime.
+
+	  https://github.com/bradfitz/embiggen-disk
+
+comment "embiggen-disk needs a glibc or musl toolchain w/ threads"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS && \
+		BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
+	depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_TOOLCHAIN_USES_UCLIBC
diff --git a/package/embiggen-disk/embiggen-disk.hash b/package/embiggen-disk/embiggen-disk.hash
new file mode 100644
index 0000000000..1a642ea9e8
--- /dev/null
+++ b/package/embiggen-disk/embiggen-disk.hash
@@ -0,0 +1,3 @@
+# Locally computed
+sha256  834bf52babc370e275319e4fff2d85ea3b4dfa94eda18b864532896976333213  embiggen-disk-c554fc1c93a4004ce0b6a3f69b0dabe0481a2308.tar.gz
+sha256  063aedec1652c5a05c2d04c40e032b932453142ee8ef7fd53c04a9acc127fc95  LICENSE
diff --git a/package/embiggen-disk/embiggen-disk.mk b/package/embiggen-disk/embiggen-disk.mk
new file mode 100644
index 0000000000..5d5ac15ae8
--- /dev/null
+++ b/package/embiggen-disk/embiggen-disk.mk
@@ -0,0 +1,12 @@
+################################################################################
+#
+# embiggen-disk
+#
+################################################################################
+
+EMBIGGEN_DISK_VERSION = c554fc1c93a4004ce0b6a3f69b0dabe0481a2308
+EMBIGGEN_DISK_SITE = $(call github,bradfitz,embiggen-disk,$(EMBIGGEN_DISK_VERSION))
+EMBIGGEN_DISK_LICENSE = Apache-2.0
+EMBIGGEN_DISK_LICENSE_FILES = LICENSE
+
+$(eval $(golang-package))
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH-NEXT v3 6/6] package/gocryptfs: new package

[Christian Stewart via buildroot]

Adds the gocryptfs encrypted FUSE filesystem.

Currently uses without_openssl build tag, to use the native Go cryptography.
However, the package could be improved by conditionally enabling openssl if it
is also configured to be built by Buildroot.

Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/Config.in                |  1 +
 package/gocryptfs/Config.in      | 12 ++++++++++++
 package/gocryptfs/gocryptfs.hash |  3 +++
 package/gocryptfs/gocryptfs.mk   | 19 +++++++++++++++++++
 4 files changed, 35 insertions(+)
 create mode 100644 package/gocryptfs/Config.in
 create mode 100644 package/gocryptfs/gocryptfs.hash
 create mode 100644 package/gocryptfs/gocryptfs.mk

diff --git a/package/Config.in b/package/Config.in
index 7a833909aa..30c2abbb6a 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -220,6 +220,7 @@ menu "Filesystem and flash utilities"
 	source "package/genext2fs/Config.in"
 	source "package/genpart/Config.in"
 	source "package/genromfs/Config.in"
+	source "package/gocryptfs/Config.in"
 	source "package/imx-usb-loader/Config.in"
 	source "package/mmc-utils/Config.in"
 	source "package/mtd/Config.in"
diff --git a/package/gocryptfs/Config.in b/package/gocryptfs/Config.in
new file mode 100644
index 0000000000..2630cafdcf
--- /dev/null
+++ b/package/gocryptfs/Config.in
@@ -0,0 +1,12 @@
+config BR2_PACKAGE_GOCRYPTFS
+	bool "gocryptfs"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	depends on BR2_TOOLCHAIN_HAS_THREADS
+	help
+	  gocryptfs is an encrypted FUSE overlay filesystem.
+
+	  https://github.com/rfjakob/gocryptfs
+
+comment "gocryptfs needs a toolchain w/ threads"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	depends on !BR2_TOOLCHAIN_HAS_THREADS
diff --git a/package/gocryptfs/gocryptfs.hash b/package/gocryptfs/gocryptfs.hash
new file mode 100644
index 0000000000..f54dce1626
--- /dev/null
+++ b/package/gocryptfs/gocryptfs.hash
@@ -0,0 +1,3 @@
+# Locally computed
+sha256  9581bd579fd1c527bc7a7fe2cd1a8761947797a36bad493902f186a4f5967cf8  gocryptfs-2.2.0.tar.gz
+sha256  322a7e3b02cf18e38b7e6b18cafefb773df8676c65634b34e8a2beb931294a4b  LICENSE
diff --git a/package/gocryptfs/gocryptfs.mk b/package/gocryptfs/gocryptfs.mk
new file mode 100644
index 0000000000..2a00e9a959
--- /dev/null
+++ b/package/gocryptfs/gocryptfs.mk
@@ -0,0 +1,19 @@
+################################################################################
+#
+# gocryptfs
+#
+################################################################################
+
+GOCRYPTFS_VERSION = 2.2.0
+GOCRYPTFS_SITE = $(call github,rfjakob,gocryptfs,v$(GOCRYPTFS_VERSION))
+GOCRYPTFS_LICENSE = MIT
+GOCRYPTFS_LICENSE_FILES = LICENSE
+
+GOCRYPTFS_GOMOD = github.com/rfjakob/gocryptfs/v2
+
+GOCRYPTFS_LDFLAGS = \
+	-X main.GitVersion=$(GOCRYPTFS_VERSION) \
+	-X main.GitVersionFuse=[vendored]
+GOCRYPTFS_TAGS = without_openssl
+
+$(eval $(golang-package))
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v3 1/6] support/download/dl-wrapper: add concept of download post-processing

[Thomas Petazzoni]

Hello Christian,

Thanks for your work on this topic! I didn't review the patches in
details, but here are a couple of comments on the form:

 - The initial series also had support for Cargo, I think we want to
   retain that as the idea was to solve both Go and Cargo at the same
   time, to make sure we have a solution that works for both.

 - PATCH-NEXT doesn't make any sense right now, there is no "next"
   branch.

 - Your patch series lacks a cover letter with a description of the
   changes between versions. So of your commits have a v1 -> v2
   changelog, but we don't know what changed in v3.

Best regards,

Thomas

On Sun, 10 Oct 2021 16:46:50 -0700
Christian Stewart <christian@paral.in> wrote:

> From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> In order to support package managers such as Cargo (Rust) or Go, we
> want to run some custom logic after the main download, but before
> packing the tarball and checking the hash.
> 
> To implement this, this commit introduces a concept of download
> post-processing: if -p <something> is passed to the dl-wrapper, then
> support/download/<something>-post-process will be called.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Christian Stewart <christian@paral.in>
> ---
>  support/download/dl-wrapper | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index 3315bd410e..2d74554213 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -25,7 +25,7 @@ main() {
>      local -a uris
>  
>      # Parse our options; anything after '--' is for the backend
> -    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
> +    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
>          case "${OPT}" in
>          c)  cset="${OPTARG}";;
>          d)  dl_dir="${OPTARG}";;
> @@ -37,6 +37,7 @@ main() {
>          r)  recurse="-r";;
>          f)  filename="${OPTARG}";;
>          u)  uris+=( "${OPTARG}" );;
> +        p)  post_process="${OPTARG}";;
>          q)  quiet="-q";;
>          :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
>          \?) error "unknown option '%s'\n" "${OPTARG}";;
> @@ -135,6 +136,12 @@ main() {
>              continue
>          fi
>  
> +        if [ -n "${post_process}" ] ; then
> +                ${OLDPWD}/support/download/${post_process}-post-process \
> +                         -o "${tmpf}" \
> +                         -n "${raw_base_name}"
> +        fi
> +
>          # cd back to free the temp-dir, so we can remove it later
>          cd "${OLDPWD}"
>  



-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v3 1/6] support/download/dl-wrapper: add concept of download post-processing

[Christian Stewart via buildroot]

Hi Thomas,

On Mon, Oct 11, 2021 at 12:04 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Christian,
>
> Thanks for your work on this topic! I didn't review the patches in
> details, but here are a couple of comments on the form:
>
>  - The initial series also had support for Cargo, I think we want to
>    retain that as the idea was to solve both Go and Cargo at the same
>    time, to make sure we have a solution that works for both.

I don't have the experience with Rust to sign off on the Cargo
portion, and I was unsure if it was confirmed to be ready for merging.
So in the interest of getting these packages in (including gocryptfs)
I sent the Go portion only.

Can send the full thing with the Cargo as well... if really necessary,
but can't we just do it in 2 patch series?

>  - PATCH-NEXT doesn't make any sense right now, there is no "next"
>    branch.

Ok, was just indicating that it's somewhat "next release" material
(not for backport), I guess.

>  - Your patch series lacks a cover letter with a description of the
>    changes between versions. So of your commits have a v1 -> v2
>    changelog, but we don't know what changed in v3.

I just dropped a "TODO."

Best regards,
Christian
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v3 1/6] support/download/dl-wrapper: add concept of download post-processing

[Thomas Petazzoni]

Hello Christian,

On Mon, 11 Oct 2021 00:13:02 -0700
Christian Stewart <christian@paral.in> wrote:

> >  - The initial series also had support for Cargo, I think we want to
> >    retain that as the idea was to solve both Go and Cargo at the same
> >    time, to make sure we have a solution that works for both.  
> 
> I don't have the experience with Rust to sign off on the Cargo
> portion, and I was unsure if it was confirmed to be ready for merging.
> So in the interest of getting these packages in (including gocryptfs)
> I sent the Go portion only.

When I sent the series, I had neither the experience with Go nor Rust :-)

> Can send the full thing with the Cargo as well... if really necessary,
> but can't we just do it in 2 patch series?

The idea of having a single series was to make sure we are able to
implement a mechanism that works (at least) for two different
vendoring-based package management systems.

> >  - PATCH-NEXT doesn't make any sense right now, there is no "next"
> >    branch.  
> 
> Ok, was just indicating that it's somewhat "next release" material
> (not for backport), I guess.

Whether something is to be considered for backport or not is a decision
that Peter Korsgaard does. And new features are by nature generally not
considered for backporting.

Cheers!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v3 4/6] support/download/go-post-process: implement Go vendoring support

[Thomas Petazzoni]

Hello Christian,

On Sun, 10 Oct 2021 16:46:53 -0700
Christian Stewart via buildroot <buildroot@buildroot.org> wrote:

> From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> This commit introduces the download post-process script
> support/download/go-post-process, and hooks it into the Go package
> infrastructure.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Christian Stewart <christian@paral.in>
> 
> v1 -> v2:
> 
>  - re-submitting Thomas's series with adjustments:
>  - run "go mod init" just before "go mod vendor"
>  - this fixes the case when go.mod does not exist
>  - use -modcacherw to fix "make clean" permissions errors
>  - use the mk_tar_gz helper in the post-process step
> 
> Signed-off-by: Christian Stewart <christian@paral.in>

I am a bit confused by the changes you introduced here. You're
labelling this patch series v3, so I assume it comes after my v2:

  https://patchwork.ozlabs.org/project/buildroot/patch/20201219153525.1361175-6-thomas.petazzoni@bootlin.com/

I'm particularly confused by the addition of the -g option to pass the
argument of "go mod init" as opposed to using the BR_GOMOD environment
variable.

In the patch I linked above, you reported that it was causing some hash
issues, but I don't understand which of your changes fixed that
problem. Could you clarify?

Also, my patch dropped the creation of go.mod from
package/pkg-golang.mk (macro $(2)_GEN_GOMOD), but I'm not sure why your
patch doesn't drop that.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v3 1/6] support/download/dl-wrapper: add concept of download post-processing

[Thomas Petazzoni]

On Sun, 10 Oct 2021 16:46:50 -0700
Christian Stewart via buildroot <buildroot@buildroot.org> wrote:

> From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> In order to support package managers such as Cargo (Rust) or Go, we
> want to run some custom logic after the main download, but before
> packing the tarball and checking the hash.
> 
> To implement this, this commit introduces a concept of download
> post-processing: if -p <something> is passed to the dl-wrapper, then
> support/download/<something>-post-process will be called.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Christian Stewart <christian@paral.in>
> ---
>  support/download/dl-wrapper | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)

I have marked this patch series as Superseded, as I have posted:

  https://patchwork.ozlabs.org/project/buildroot/list/?series=279698

which replaces it.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v3 4/6] support/download/go-post-process: implement Go vendoring support

[Thomas Petazzoni]

Hello,

On Sun, 10 Oct 2021 16:46:53 -0700
Christian Stewart via buildroot <buildroot@buildroot.org> wrote:

> +# Do the Go vendoring
> +pushd ${base_name} > /dev/null
> +# modcacherw option leaves directories in the module cache at their default
> +# permissions rather than making them read-only.
> +if [ ! -f go.mod ] && [ -n "${gomod_init}" ]; then
> +    go mod init -modcacherw ${gomod_init}

So here you're using "go mod init" to create the go.mod file, but in
package/pkg-golang.mk, we're manually creating it.

So for the sake of consistency, I tried using "go mod init" in
package/pkg-golang.mk as well, but running that on balena-engine caused
go mod init to start downloading stuff... which is wrong because
balena-engine already comes with its vendored dependencies.

So in the end, I ended up creating the go.mod manually (with the same
logic) in both pkg-golang.mk and in this download post-process script.
But if we find why "go mod init" didn't work in the context of
pkg-golang.mk, we could switch back to using that.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v3 4/6] support/download/go-post-process: implement Go vendoring support

[Christian Stewart via buildroot]

Hi Thomas,

The usage of "go mod init" converts package files from older package
managers like Glide and vendor.conf into a go.mod and go.sum file with
the checksums maintained.

This is important for compatibility with some packages that don't have
go.mod but use other approaches.

On Thu, Jan 6, 2022 at 1:08 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> On Sun, 10 Oct 2021 16:46:53 -0700
> Christian Stewart via buildroot <buildroot@buildroot.org> wrote:
>
> > +# Do the Go vendoring
> > +pushd ${base_name} > /dev/null
> > +# modcacherw option leaves directories in the module cache at their default
> > +# permissions rather than making them read-only.
> > +if [ ! -f go.mod ] && [ -n "${gomod_init}" ]; then
> > +    go mod init -modcacherw ${gomod_init}
>
> So here you're using "go mod init" to create the go.mod file, but in
> package/pkg-golang.mk, we're manually creating it.

In this context it's OK to make network lookups and download things.
GEN_GOMOD, it's not OK.

> So for the sake of consistency, I tried using "go mod init" in
> package/pkg-golang.mk as well, but running that on balena-engine caused
> go mod init to start downloading stuff... which is wrong because
> balena-engine already comes with its vendored dependencies.

In this context we don't want downloads to occur, so generating
manually makes sense.

> So in the end, I ended up creating the go.mod manually (with the same
> logic) in both pkg-golang.mk and in this download post-process script.
> But if we find why "go mod init" didn't work in the context of
> pkg-golang.mk, we could switch back to using that.

I'd prefer to keep the "go mod init" in the download script for better
compatibility with packages that don't yet use go.mod and go.sum. If
you want to keep this in a separate patch & merge the manual-create in
the meantime, that'd also work.

Best regards,
Christian
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v3 4/6] support/download/go-post-process: implement Go vendoring support

[Christian Stewart via buildroot]

Hi all,

On Thu, Jan 6, 2022 at 1:20 PM Christian Stewart <christian@paral.in> wrote:
> The usage of "go mod init" converts package files from older package
> managers like Glide and vendor.conf into a go.mod and go.sum file with
> the checksums maintained.

> > > +# Do the Go vendoring
> > > +pushd ${base_name} > /dev/null
> > > +# modcacherw option leaves directories in the module cache at their default
> > > +# permissions rather than making them read-only.
> > > +if [ ! -f go.mod ] && [ -n "${gomod_init}" ]; then
> > > +    go mod init -modcacherw ${gomod_init}
>
> > So in the end, I ended up creating the go.mod manually (with the same
> > logic) in both pkg-golang.mk and in this download post-process script.
> > But if we find why "go mod init" didn't work in the context of
> > pkg-golang.mk, we could switch back to using that.

One other potential issue is Go will append a line with "go 1.16" -
the major Go version - for compatibility reasons.

So if we update Go to a new major version the go.mod might not be the
same, causing a hash mismatch.

Something to take a look at. Maybe when manually creating go.mod, we
can append "go 1.16" to make the go.mod follow the 1.16 format.

Best regards,
Christian
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot