[Buildroot] [PATCH-NEXT v2 1/5] support/download/dl-wrapper: add concept of download post-processing

[Buildroot] [PATCH-NEXT v2 1/5] support/download/dl-wrapper: add concept of download post-processing

[Christian Stewart via buildroot]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

In order to support package managers such as Cargo (Rust) or Go, we
want to run some custom logic after the main download, but before
packing the tarball and checking the hash.

To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then
support/download/<something>-post-process will be called.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Christian Stewart <christian@paral.in>
---
 support/download/dl-wrapper | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 3315bd410e..2d74554213 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -25,7 +25,7 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() {
         r)  recurse="-r";;
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
+        p)  post_process="${OPTARG}";;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -135,6 +136,12 @@ main() {
             continue
         fi
 
+        if [ -n "${post_process}" ] ; then
+                ${OLDPWD}/support/download/${post_process}-post-process \
+                         -o "${tmpf}" \
+                         -n "${raw_base_name}"
+        fi
+
         # cd back to free the temp-dir, so we can remove it later
         cd "${OLDPWD}"
 
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH-NEXT v2 2/5] package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable

[Christian Stewart via buildroot]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

This will allow packages to register than a download post-processing
is needed. Note that this variable is intentionally not documented: it
is an internal variable meant to be set by package infrastructures,
not directly by packages.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/pkg-download.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 2527ba5c60..66a415cce0 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -117,6 +117,7 @@ define DOWNLOAD
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH-NEXT v2 3/5] support/download/post-process-helpers: add helper function for post process scripts

[Christian Stewart via buildroot]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

download post process scripts will often need to unpack the source
code tarball, do some operation, and then repack it. In order to help
with this, post-process-helpers provide an unpack() function and a
repack() function.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Christian Stewart <christian@paral.in>
---
 support/download/post-process-helpers | 30 +++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 support/download/post-process-helpers

diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers
new file mode 100644
index 0000000000..bed8df2577
--- /dev/null
+++ b/support/download/post-process-helpers
@@ -0,0 +1,30 @@
+
+unpack() {
+        dest="$1"
+        tarball="$2"
+
+        mkdir ${dest}
+        tar -C ${dest} --strip-components=1 -xf ${tarball}
+}
+
+repack() {
+        src="$1"
+        tarball="$2"
+
+        # Generate the archive, sort with the C locale so that it is reproducible.
+        find "$(basename ${src})" -not -type d -print0 >files.list
+        LC_ALL=C sort -z <files.list >files.list.sorted
+
+        # let's use a fixed hardcoded date to be reproducible
+        date="2020-02-06 01:02:03 +0000"
+
+        # Create GNU-format tarballs, since that's the format of the tarballs on
+        # sources.buildroot.org and used in the *.hash files
+        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
+            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
+        gzip -6 -n <new.tar >new.tar.gz
+        mv "${tarball}" "${tarball}".old
+        mv new.tar.gz "${tarball}"
+        rm "${tarball}".old
+        rm -rf ${src}
+}
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH-NEXT v2 4/5] support/download/go-post-process: implement Go vendoring support

[Christian Stewart via buildroot]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

This commit introduces the download post-process script
support/download/go-post-process, and hooks it into the Go package
infrastructure.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Christian Stewart <christian@paral.in>

v1 -> v2:

 - re-submitting Thomas's series with adjustments:
 - run "go mod init" just before "go mod vendor"
 - this fixes the case when go.mod does not exist
 - use -modcacherw to fix "make clean" permissions errors
 - use the mk_tar_gz helper in the post-process step

Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/pkg-download.mk               |  1 +
 package/pkg-golang.mk                 |  8 +++++-
 support/download/dl-wrapper           |  6 +++--
 support/download/go-post-process      | 35 ++++++++++++++++++++++++
 support/download/post-process-helpers | 38 ++++++++++++---------------
 5 files changed, 64 insertions(+), 24 deletions(-)
 create mode 100755 support/download/go-post-process
 mode change 100644 => 100755 support/download/post-process-helpers

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 66a415cce0..378b2c66f0 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -119,6 +119,7 @@ define DOWNLOAD
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
 		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
+		$(if $($(2)_GOMOD),-g '$($(2)_GOMOD)') \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
 		-- \
diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
index d07242310d..3d2c45fa09 100644
--- a/package/pkg-golang.mk
+++ b/package/pkg-golang.mk
@@ -42,12 +42,13 @@ define inner-golang-package
 
 $(2)_BUILD_OPTS += \
 	-ldflags "$$($(2)_LDFLAGS)" \
+	-modcacherw \
 	-tags "$$($(2)_TAGS)" \
 	-trimpath \
 	-p $(PARALLEL_JOBS)
 
 # Target packages need the Go compiler on the host.
-$(2)_DEPENDENCIES += host-go
+$(2)_DOWNLOAD_DEPENDENCIES += host-go
 
 $(2)_BUILD_TARGETS ?= .
 
@@ -72,6 +73,11 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
 # If the go.mod file does not exist, one is written with this root path.
 $(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
 
+$(2)_DOWNLOAD_POST_PROCESS = go
+$(2)_DL_ENV = \
+	$(HOST_GO_COMMON_ENV) \
+	GOPROXY=direct
+
 # Generate a go.mod file if it doesn't exist. Note: Go is configured
 # to use the "vendor" dir and not make network calls.
 define $(2)_GEN_GOMOD
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 2d74554213..2fc530f24f 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -17,7 +17,7 @@
 # We want to catch any unexpected failure, and exit immediately.
 set -e
 
-export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
+export BR_BACKEND_DL_GETOPTS=":hc:d:g:o:n:N:H:ru:qf:e"
 
 main() {
     local OPT OPTARG
@@ -25,11 +25,12 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
+    while getopts ":c:d:D:g:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
         D)  old_dl_dir="${OPTARG}";;
+        g)  gomod_init="${OPTARG}";;
         o)  output="${OPTARG}";;
         n)  raw_base_name="${OPTARG}";;
         N)  base_name="${OPTARG}";;
@@ -138,6 +139,7 @@ main() {
 
         if [ -n "${post_process}" ] ; then
                 ${OLDPWD}/support/download/${post_process}-post-process \
+                         -g "${gomod_init}" \
                          -o "${tmpf}" \
                          -n "${raw_base_name}"
         fi
diff --git a/support/download/go-post-process b/support/download/go-post-process
new file mode 100755
index 0000000000..b7f05b37c1
--- /dev/null
+++ b/support/download/go-post-process
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -e
+
+. "${0%/*}/post-process-helpers"
+
+# Parse our options
+while getopts "n:g:o:" OPT; do
+        case "${OPT}" in
+        g)  gomod_init="${OPTARG}";;
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
+	exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Go vendoring
+pushd ${base_name} > /dev/null
+# modcacherw option leaves directories in the module cache at their default
+# permissions rather than making them read-only.
+if [ ! -f go.mod ] && [ -n "${gomod_init}" ]; then
+    go mod init -modcacherw ${gomod_init}
+fi
+go mod vendor -modcacherw -v
+popd > /dev/null
+
+repack $(pwd) ${base_name} ${output}
diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers
old mode 100644
new mode 100755
index bed8df2577..b3231207a6
--- a/support/download/post-process-helpers
+++ b/support/download/post-process-helpers
@@ -1,30 +1,26 @@
+# Helpers for the post-process step to re-pack vendored archives.
+
+. "${0%/*}/helpers"
 
 unpack() {
-        dest="$1"
-        tarball="$2"
+    dest="$1"
+    tarball="$2"
 
-        mkdir ${dest}
-        tar -C ${dest} --strip-components=1 -xf ${tarball}
+    mkdir ${dest}
+    tar -C ${dest} --strip-components=1 -xf ${tarball}
 }
 
 repack() {
-        src="$1"
-        tarball="$2"
-
-        # Generate the archive, sort with the C locale so that it is reproducible.
-        find "$(basename ${src})" -not -type d -print0 >files.list
-        LC_ALL=C sort -z <files.list >files.list.sorted
+    local in_dir="${1}"
+    local base_dir="${2}"
+    local out="${3}"
 
-        # let's use a fixed hardcoded date to be reproducible
-        date="2020-02-06 01:02:03 +0000"
+    # let's use a fixed hardcoded date to be reproducible
+    date="2020-02-06 01:02:03 +0000"
 
-        # Create GNU-format tarballs, since that's the format of the tarballs on
-        # sources.buildroot.org and used in the *.hash files
-        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
-            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
-        gzip -6 -n <new.tar >new.tar.gz
-        mv "${tarball}" "${tarball}".old
-        mv new.tar.gz "${tarball}"
-        rm "${tarball}".old
-        rm -rf ${src}
+    # use the helper to create a reproducible archive
+    mk_tar_gz "${in_dir}/${base_dir}" "${base_dir}" "${date}" "${out}"
 }
+
+# Keep this line and the following as last lines in this file.
+# vim: ft=bash
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH-NEXT v2 5/5] package/embiggen-disk: new package

[Christian Stewart via buildroot]

embiggen-disk is a tool to automatically resize disks to fill available space.

Patch submitted upstream: https://github.com/bradfitz/embiggen-disk/pull/13

Adds support for /dev/mmcblk0pN type paths.

Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/Config.in                             |  1 +
 ...-Fix-resizing-of-dev-mmcblk0pN-paths.patch | 45 +++++++++++++++++++
 package/embiggen-disk/Config.in               | 18 ++++++++
 package/embiggen-disk/embiggen-disk.hash      |  3 ++
 package/embiggen-disk/embiggen-disk.mk        | 12 +++++
 5 files changed, 79 insertions(+)
 create mode 100644 package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch
 create mode 100644 package/embiggen-disk/Config.in
 create mode 100644 package/embiggen-disk/embiggen-disk.hash
 create mode 100644 package/embiggen-disk/embiggen-disk.mk

diff --git a/package/Config.in b/package/Config.in
index 899b493180..f7029ef723 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -2468,6 +2468,7 @@ menu "System tools"
 	source "package/earlyoom/Config.in"
 	source "package/efibootmgr/Config.in"
 	source "package/efivar/Config.in"
+	source "package/embiggen-disk/Config.in"
 	source "package/emlog/Config.in"
 	source "package/ftop/Config.in"
 	source "package/getent/Config.in"
diff --git a/package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch b/package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch
new file mode 100644
index 0000000000..31cfea9dde
--- /dev/null
+++ b/package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch
@@ -0,0 +1,45 @@
+From b628e4c561cb4d3ffc92a5d7aac8f4967e44977e Mon Sep 17 00:00:00 2001
+From: Christian Stewart <christian@paral.in>
+Date: Thu, 27 May 2021 20:07:07 -0700
+Subject: [PATCH] Fix resizing of /dev/mmcblk0pN paths
+
+Signed-off-by: Christian Stewart <christian@paral.in>
+---
+ fs.go   | 4 +++-
+ part.go | 5 +++++
+ 2 files changed, 8 insertions(+), 1 deletion(-)
+
+diff --git a/fs.go b/fs.go
+index 0b83e24..8ab3690 100644
+--- a/fs.go
++++ b/fs.go
+@@ -67,7 +67,9 @@ func (e fsResizer) DepResizer() (Resizer, error) {
+ 	if dev == "/dev/root" {
+ 		return nil, errors.New("unexpected device /dev/root from statFS")
+ 	}
+-	if (strings.HasPrefix(dev, "/dev/sd") || strings.HasPrefix(dev, "/dev/nvme")) &&
++	if (strings.HasPrefix(dev, "/dev/sd") ||
++		strings.HasPrefix(dev, "/dev/mmcblk") ||
++		strings.HasPrefix(dev, "/dev/nvme")) &&
+ 		devEndsInNumber(dev) {
+ 		vlogf("fsResizer.DepResizer: returning partitionResizer(%q)", dev)
+ 		return partitionResizer(dev), nil
+diff --git a/part.go b/part.go
+index 0315a6e..f3e280f 100644
+--- a/part.go
++++ b/part.go
+@@ -51,6 +51,11 @@ func diskDev(partDev string) string {
+ 	if strings.HasPrefix(partDev, "/dev/sd") {
+ 		return strings.TrimRight(partDev, "0123456789")
+ 	}
++	if strings.HasPrefix(partDev, "/dev/mmcblk") {
++		v := strings.TrimRight(partDev, "0123456789")
++		v = strings.TrimSuffix(v, "p")
++		return v
++	}
+ 	if strings.HasPrefix(partDev, "/dev/nvme") {
+ 		chopP := regexp.MustCompile(`p\d+$`)
+ 		if !chopP.MatchString(partDev) {
+-- 
+2.31.1
+
diff --git a/package/embiggen-disk/Config.in b/package/embiggen-disk/Config.in
new file mode 100644
index 0000000000..d409d36af6
--- /dev/null
+++ b/package/embiggen-disk/Config.in
@@ -0,0 +1,18 @@
+# TODO requires sfdisk
+config BR2_PACKAGE_EMBIGGEN_DISK
+	bool "embiggen-disk"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
+	depends on BR2_TOOLCHAIN_HAS_THREADS
+	depends on !BR2_TOOLCHAIN_USES_UCLIBC # no fexecve
+	select BR2_PACKAGE_UTIL_LINUX # sfdisk
+	select BR2_PACKAGE_UTIL_LINUX_BINARIES # sfdisk
+	help
+	  embiggen-disk is a tool to resize disk partitions at runtime.
+
+	  https://github.com/bradfitz/embiggen-disk
+
+comment "embiggen-disk needs a glibc or musl toolchain w/ threads"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS && \
+		BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
+	depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_TOOLCHAN_USES_UCLIBC
diff --git a/package/embiggen-disk/embiggen-disk.hash b/package/embiggen-disk/embiggen-disk.hash
new file mode 100644
index 0000000000..1a642ea9e8
--- /dev/null
+++ b/package/embiggen-disk/embiggen-disk.hash
@@ -0,0 +1,3 @@
+# Locally computed
+sha256  834bf52babc370e275319e4fff2d85ea3b4dfa94eda18b864532896976333213  embiggen-disk-c554fc1c93a4004ce0b6a3f69b0dabe0481a2308.tar.gz
+sha256  063aedec1652c5a05c2d04c40e032b932453142ee8ef7fd53c04a9acc127fc95  LICENSE
diff --git a/package/embiggen-disk/embiggen-disk.mk b/package/embiggen-disk/embiggen-disk.mk
new file mode 100644
index 0000000000..5d5ac15ae8
--- /dev/null
+++ b/package/embiggen-disk/embiggen-disk.mk
@@ -0,0 +1,12 @@
+################################################################################
+#
+# embiggen-disk
+#
+################################################################################
+
+EMBIGGEN_DISK_VERSION = c554fc1c93a4004ce0b6a3f69b0dabe0481a2308
+EMBIGGEN_DISK_SITE = $(call github,bradfitz,embiggen-disk,$(EMBIGGEN_DISK_VERSION))
+EMBIGGEN_DISK_LICENSE = Apache-2.0
+EMBIGGEN_DISK_LICENSE_FILES = LICENSE
+
+$(eval $(golang-package))
-- 
2.33.0

_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v2 4/5] support/download/go-post-process: implement Go vendoring support

[Christian Stewart via buildroot]

All,

Re-submitted this series to fix determinism issues mentioned here:

 - https://github.com/skiffos/SkiffOS/issues/164
 - https://yhbt.net/lore/all/20210801070830.GP3189549@scaer/t/#r2e8e5fab88a021fa81f6d22e44bd6d06ea740c67

On Sun, Sep 19, 2021 at 12:10 AM Christian Stewart <christian@paral.in> wrote:

> +# Helpers for the post-process step to re-pack vendored archives.
> +
> +. "${0%/*}/helpers"
>
>  unpack() {
> -        dest="$1"
> -        tarball="$2"
> +    dest="$1"
> +    tarball="$2"
>
> -        mkdir ${dest}
> -        tar -C ${dest} --strip-components=1 -xf ${tarball}
> +    mkdir ${dest}
> +    tar -C ${dest} --strip-components=1 -xf ${tarball}
>  }

... probably these changes should be in the commit just before this
one, where the file is introduced.

my mistake.

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v2 5/5] package/embiggen-disk: new package

[Christian Stewart via buildroot]

All,

On Sun, Sep 19, 2021 at 12:10 AM Christian Stewart <christian@paral.in> wrote:
>
> embiggen-disk is a tool to automatically resize disks to fill available space.
>

> diff --git a/package/embiggen-disk/Config.in b/package/embiggen-disk/Config.in
> new file mode 100644
> index 0000000000..d409d36af6
> --- /dev/null
> +++ b/package/embiggen-disk/Config.in
> @@ -0,0 +1,18 @@
> +# TODO requires sfdisk
> +config BR2_PACKAGE_EMBIGGEN_DISK
> +       bool "embiggen-disk"
> +       depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
> +       depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
> +       depends on BR2_TOOLCHAIN_HAS_THREADS
> +       depends on !BR2_TOOLCHAIN_USES_UCLIBC # no fexecve
> +       select BR2_PACKAGE_UTIL_LINUX # sfdisk
> +       select BR2_PACKAGE_UTIL_LINUX_BINARIES # sfdisk

BR2_PACKAGE_UTIL_LINUX_BINARIES provides sfdisk. The TODO here can be
deleted and safely ignored: embiggen-disk has been tested with this
package extensively.

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v2 1/5] support/download/dl-wrapper: add concept of download post-processing

[Yann E. MORIN]

Christian, All,

Can you fixup the patches as you already noticed, and respin a fixed
up series, please? In the meantime., I've marked this one Changes
REquested in Patchwork.

Also, what was still preventing applying this to Buildroot, was a few
still open questions, the most prominent one I can remember is about
BR2_PRIMARY_SITE and BR2_BACKUP_SITE.

Indeed, what we download from the main site is an un-vendored source,
but what we will get in primary or backup sites are vendored sources.

Downloading vendored sources from primary or mirror also means that no
vendoring should be applied afterward, i.e. vendoring should only occur
on sources downloaded from the main site.

Did you try that? Can you report on how that eventually played out?

Regards,
Yann E. MORIN.

On 2021-09-19 00:10 -0700, Christian Stewart via buildroot spake thusly:
> From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> In order to support package managers such as Cargo (Rust) or Go, we
> want to run some custom logic after the main download, but before
> packing the tarball and checking the hash.
> 
> To implement this, this commit introduces a concept of download
> post-processing: if -p <something> is passed to the dl-wrapper, then
> support/download/<something>-post-process will be called.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Christian Stewart <christian@paral.in>
> ---
>  support/download/dl-wrapper | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index 3315bd410e..2d74554213 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -25,7 +25,7 @@ main() {
>      local -a uris
>  
>      # Parse our options; anything after '--' is for the backend
> -    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
> +    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
>          case "${OPT}" in
>          c)  cset="${OPTARG}";;
>          d)  dl_dir="${OPTARG}";;
> @@ -37,6 +37,7 @@ main() {
>          r)  recurse="-r";;
>          f)  filename="${OPTARG}";;
>          u)  uris+=( "${OPTARG}" );;
> +        p)  post_process="${OPTARG}";;
>          q)  quiet="-q";;
>          :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
>          \?) error "unknown option '%s'\n" "${OPTARG}";;
> @@ -135,6 +136,12 @@ main() {
>              continue
>          fi
>  
> +        if [ -n "${post_process}" ] ; then
> +                ${OLDPWD}/support/download/${post_process}-post-process \
> +                         -o "${tmpf}" \
> +                         -n "${raw_base_name}"
> +        fi
> +
>          # cd back to free the temp-dir, so we can remove it later
>          cd "${OLDPWD}"
>  
> -- 
> 2.33.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@lists.buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v2 1/5] support/download/dl-wrapper: add concept of download post-processing

[Arnout Vandecappelle]

On 19/09/2021 11:37, Yann E. MORIN wrote:
> Christian, All,
> 
> Can you fixup the patches as you already noticed, and respin a fixed
> up series, please? In the meantime., I've marked this one Changes
> REquested in Patchwork.
> 
> Also, what was still preventing applying this to Buildroot, was a few
> still open questions, the most prominent one I can remember is about
> BR2_PRIMARY_SITE and BR2_BACKUP_SITE.
> 
> Indeed, what we download from the main site is an un-vendored source,
> but what we will get in primary or backup sites are vendored sources.
> 
> Downloading vendored sources from primary or mirror also means that no
> vendoring should be applied afterward, i.e. vendoring should only occur
> on sources downloaded from the main site.
> 
> Did you try that? Can you report on how that eventually played out?

  I guess that's what this piece of code in go-post-process is about:

# Already vendored tarball, nothing to do
if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
	exit 0
fi


  Regards,
  Arnout


[snip]
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH-NEXT v2 1/5] support/download/dl-wrapper: add concept of download post-processing

[Christian Stewart via buildroot]

Yann, All,

On Sun, Sep 19, 2021 at 4:55 AM Arnout Vandecappelle <arnout@mind.be> wrote:
> On 19/09/2021 11:37, Yann E. MORIN wrote:
> >
> > Downloading vendored sources from primary or mirror also means that no
> > vendoring should be applied afterward, i.e. vendoring should only occur
> > on sources downloaded from the main site.
> >
> > Did you try that? Can you report on how that eventually played out?
>
>   I guess that's what this piece of code in go-post-process is about:
>
> # Already vendored tarball, nothing to do
> if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
>         exit 0
> fi

Indeed; that line of code will catch the case where "vendor" already
exists, and no extra vendoring will be performed.

This seems to work fine in all the cases I've tested.

Is this OK to respin as-is?

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot