* [Buildroot] [PATCH v2 01/12] support/download/dl-wrapper: add concept of download post-processing
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 02/12] package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable Thomas Petazzoni
` (10 subsequent siblings)
11 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
In order to support package managers such as Cargo (Rust) or Go, we
want to run some custom logic after the main download, but before
packing the tarball and checking the hash.
To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then
support/download/<something>-post-process will be called.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
support/download/dl-wrapper | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 3315bd410e..2d74554213 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -25,7 +25,7 @@ main() {
local -a uris
# Parse our options; anything after '--' is for the backend
- while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+ while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
case "${OPT}" in
c) cset="${OPTARG}";;
d) dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() {
r) recurse="-r";;
f) filename="${OPTARG}";;
u) uris+=( "${OPTARG}" );;
+ p) post_process="${OPTARG}";;
q) quiet="-q";;
:) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
\?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -135,6 +136,12 @@ main() {
continue
fi
+ if [ -n "${post_process}" ] ; then
+ ${OLDPWD}/support/download/${post_process}-post-process \
+ -o "${tmpf}" \
+ -n "${raw_base_name}"
+ fi
+
# cd back to free the temp-dir, so we can remove it later
cd "${OLDPWD}"
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 02/12] package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 01/12] support/download/dl-wrapper: add concept of download post-processing Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 03/12] package/pkg-download.mk: add <pkg>_DL_ENV variable Thomas Petazzoni
` (9 subsequent siblings)
11 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
This will allow packages to register than a download post-processing
is needed. Note that this variable is intentionally not documented: it
is an internal variable meant to be set by package infrastructures,
not directly by packages.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/pkg-download.mk | 1 +
1 file changed, 1 insertion(+)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 951d2fb554..175b9d7dba 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -108,6 +108,7 @@ define DOWNLOAD
-n '$($(2)_BASENAME_RAW)' \
-N '$($(2)_RAWNAME)' \
-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+ $(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
$(if $($(2)_GIT_SUBMODULES),-r) \
$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
$(QUIET) \
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 03/12] package/pkg-download.mk: add <pkg>_DL_ENV variable
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 01/12] support/download/dl-wrapper: add concept of download post-processing Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 02/12] package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-29 16:18 ` Yann E. MORIN
2020-12-19 15:35 ` [Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts Thomas Petazzoni
` (8 subsequent siblings)
11 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
This variable can be used by package to pass extra environment
variables to the download logic. It will be used for the Go/Cargo
vendoring.
The <pkg>_DL_ENV variable is intentionally not documented: at this
point, it is not meant to be used by packages directly, but only by
package infrastructures.
Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/pkg-download.mk | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 175b9d7dba..c914d016e2 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -99,7 +99,9 @@ endif
define DOWNLOAD
$(Q)mkdir -p $($(2)_DL_DIR)
- $(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
+ $(Q)$(EXTRA_ENV) \
+ $($(2)_DL_ENV) \
+ flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
-c '$($(2)_DL_VERSION)' \
-d '$($(2)_DL_DIR)' \
-D '$(DL_DIR)' \
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 03/12] package/pkg-download.mk: add <pkg>_DL_ENV variable
2020-12-19 15:35 ` [Buildroot] [PATCH v2 03/12] package/pkg-download.mk: add <pkg>_DL_ENV variable Thomas Petazzoni
@ 2020-12-29 16:18 ` Yann E. MORIN
0 siblings, 0 replies; 24+ messages in thread
From: Yann E. MORIN @ 2020-12-29 16:18 UTC (permalink / raw)
To: buildroot
Thomas, All,
On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
> This variable can be used by package to pass extra environment
> variables to the download logic. It will be used for the Go/Cargo
> vendoring.
>
> The <pkg>_DL_ENV variable is intentionally not documented: at this
> point, it is not meant to be used by packages directly, but only by
> package infrastructures.
>
> Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Although it's not much off your chest on this series: applied to
master, with a little tweak, see below...
> ---
> package/pkg-download.mk | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 175b9d7dba..c914d016e2 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -99,7 +99,9 @@ endif
>
> define DOWNLOAD
> $(Q)mkdir -p $($(2)_DL_DIR)
> - $(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
> + $(Q)$(EXTRA_ENV) \
> + $($(2)_DL_ENV) \
> + flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
I find this code to be ugly to look at. But I could not come up with
something that is nice to look at. So I moved the two _ENV variables
expansions to the same line, and it looks less-ugly now...
Thanks.
Regards,
Yann E. MORIN.
> -c '$($(2)_DL_VERSION)' \
> -d '$($(2)_DL_DIR)' \
> -D '$(DL_DIR)' \
> --
> 2.29.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
` (2 preceding siblings ...)
2020-12-19 15:35 ` [Buildroot] [PATCH v2 03/12] package/pkg-download.mk: add <pkg>_DL_ENV variable Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-19 17:39 ` Yann E. MORIN
2020-12-20 8:40 ` Yann E. MORIN
2020-12-19 15:35 ` [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support Thomas Petazzoni
` (7 subsequent siblings)
11 siblings, 2 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
download post process scripts will often need to unpack the source
code tarball, do some operation, and then repack it. In order to help
with this, post-process-helpers provide an unpack() function and a
repack() function.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
support/download/post-process-helpers | 30 +++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
create mode 100644 support/download/post-process-helpers
diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers
new file mode 100644
index 0000000000..bed8df2577
--- /dev/null
+++ b/support/download/post-process-helpers
@@ -0,0 +1,30 @@
+
+unpack() {
+ dest="$1"
+ tarball="$2"
+
+ mkdir ${dest}
+ tar -C ${dest} --strip-components=1 -xf ${tarball}
+}
+
+repack() {
+ src="$1"
+ tarball="$2"
+
+ # Generate the archive, sort with the C locale so that it is reproducible.
+ find "$(basename ${src})" -not -type d -print0 >files.list
+ LC_ALL=C sort -z <files.list >files.list.sorted
+
+ # let's use a fixed hardcoded date to be reproducible
+ date="2020-02-06 01:02:03 +0000"
+
+ # Create GNU-format tarballs, since that's the format of the tarballs on
+ # sources.buildroot.org and used in the *.hash files
+ tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
+ --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
+ gzip -6 -n <new.tar >new.tar.gz
+ mv "${tarball}" "${tarball}".old
+ mv new.tar.gz "${tarball}"
+ rm "${tarball}".old
+ rm -rf ${src}
+}
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts
2020-12-19 15:35 ` [Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts Thomas Petazzoni
@ 2020-12-19 17:39 ` Yann E. MORIN
2020-12-19 20:29 ` Yann E. MORIN
2020-12-20 8:40 ` Yann E. MORIN
1 sibling, 1 reply; 24+ messages in thread
From: Yann E. MORIN @ 2020-12-19 17:39 UTC (permalink / raw)
To: buildroot
Thomas, All,
On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
> download post process scripts will often need to unpack the source
> code tarball, do some operation, and then repack it. In order to help
> with this, post-process-helpers provide an unpack() function and a
> repack() function.
You forgot to explain in the commit log why we need a hard-coded, fixed
date.
However, I find it ugly that we resort to such an arbitrary constant. At
the very least, I'd like we take a memorable value, like 1970-01-01
00:00:00 +0000.
However, I am afraid that tar would refuse to extract files to that
date, whining about an "implausibly old time stamp" when extracting such
files...
But see below for a proposal...
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> support/download/post-process-helpers | 30 +++++++++++++++++++++++++++
> 1 file changed, 30 insertions(+)
> create mode 100644 support/download/post-process-helpers
>
> diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers
> new file mode 100644
> index 0000000000..bed8df2577
> --- /dev/null
> +++ b/support/download/post-process-helpers
> @@ -0,0 +1,30 @@
> +
> +unpack() {
> + dest="$1"
> + tarball="$2"
This is a shell script, variables are global. But since we run it with
bash, we can declare such variable as local:
local dest="${1}"
local tarball="${2}"
> + mkdir ${dest}
> + tar -C ${dest} --strip-components=1 -xf ${tarball}
local one_file
one_file="$(find "${dest}" -type f |head -n1)"
touch -r "${one_file}" "${dest}.timestamp"
> +}
> +
> +repack() {
> + src="$1"
> + tarball="$2"
Ditto: make them 'local' (this can be fixed when applying).
Also, I suppose that the 'dest' parameter of unpack is expected to be the
'src' parameter of repack, right?
> + # Generate the archive, sort with the C locale so that it is reproducible.
> + find "$(basename ${src})" -not -type d -print0 >files.list
> + LC_ALL=C sort -z <files.list >files.list.sorted
> +
> + # let's use a fixed hardcoded date to be reproducible
> + date="2020-02-06 01:02:03 +0000"
Assuming that unpack's dest and repack's src are pointing to the same
directory:
date="$(stat -c "${src}.timestamp")"
> + # Create GNU-format tarballs, since that's the format of the tarballs on
> + # sources.buildroot.org and used in the *.hash files
> + tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
> + --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
This will have to be changed according to my pending series to change to
the PAX format, whish is even more reproducible.
> + gzip -6 -n <new.tar >new.tar.gz
> + mv "${tarball}" "${tarball}".old
> + mv new.tar.gz "${tarball}"
> + rm "${tarball}".old
> + rm -rf ${src}
Quote all expanded variable, even "${src}". And also rm "${src}.timestamp",
now.
Regards,
Yann E. MORIN.
> +}
> --
> 2.29.2
>
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts
2020-12-19 17:39 ` Yann E. MORIN
@ 2020-12-19 20:29 ` Yann E. MORIN
0 siblings, 0 replies; 24+ messages in thread
From: Yann E. MORIN @ 2020-12-19 20:29 UTC (permalink / raw)
To: buildroot
Thomas, All,
A few mre tweaks I forgot about...
On 2020-12-19 18:39 +0100, Yann E. MORIN spake thusly:
> On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
[--SNIP--]
> > + mkdir ${dest}
Using 'mkdir -p' would ensure that parent direcotries are created as
needed.
>
> one_file="$(find "${dest}" -type f |head -n1)"
Of course, the output should be sorted in the C locale, to ensure the
same file is always used as reference (and hence a better name for that
file):
local ref_file
ref_file="$(find "${dest}" -type f |LC_ALL=C sort |head -n1)"
touch -r "${ref_file}" "${dest}.timestamp"
[--SNIP--]
> > + # let's use a fixed hardcoded date to be reproducible
> > + date="2020-02-06 01:02:03 +0000"
> Assuming that unpack's dest and repack's src are pointing to the same
> directory:
> date="$(stat -c "${src}.timestamp")"
Meh.. I forgot the actual format:
date="$(stat -c '%y' "${src}.timestamp")"
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts
2020-12-19 15:35 ` [Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts Thomas Petazzoni
2020-12-19 17:39 ` Yann E. MORIN
@ 2020-12-20 8:40 ` Yann E. MORIN
1 sibling, 0 replies; 24+ messages in thread
From: Yann E. MORIN @ 2020-12-20 8:40 UTC (permalink / raw)
To: buildroot
Thomas, All,
On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
> download post process scripts will often need to unpack the source
> code tarball, do some operation, and then repack it. In order to help
> with this, post-process-helpers provide an unpack() function and a
> repack() function.
Sorry to come back yet once more on that one, but I may have found quite
an issue. Basically, your code does this:
unpack() {
tar -C ${dest} --strip-components=1 -xf ${tarball}
}
repack() {
tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
--owner=0 --group=0 --mtime="${date}" -T files.list.sorted
gzip -6 -n <new.tar >new.tar.gz
}
So, it takes a tarball in whatever copression scheme and extract it, but
it will alwyas create a gzip-compressed tarball.
So, how would we hanfle a case where the upstream tarball is, say, a
.tar.bz2 ? For example;
FOO_SOURCE = foo=1.2.3.tar.bz2
FOO_SITE = http://foo.example.net/download
FOO_SITE_METOD = wget
$(eval $(go-package))
Besides the fact that the packager would see in DL_DIR a .tar.gz tarball
different from the .tar.bz2 he coded, the most problematic issue wil be
that we would have to list the .tar.gz inthe .hash file. This will be
very confusing to packagers and reviewers...
Furthermore, there is another discrepancy in the post-processing
scripts. For example, the go post-processor does, basically:
if tarball already vendored:
exit leaving the tarball as-is
unpack
vendor
repack as .tar.gz
So some go tarballs will be left as they are upstream, while others will
be re-encoded. This is going to be quite confusing...
I think the post-processing script should always re-encode the tarball,
so that at least we have a consistent behaviour across the board. This
is pretty trivial to come up with, at the exepense of a bit of overhead
for those tarballs that are already vendored, but the majority of
packages will probably be git clones that need to be vendored, so it is
a small price to pay.
The big problem that's left, is how we handle the discrepancy between
the tarball listed in _SOURCE and the one listed in the .hash file...
I have to admit I do not have an easy solution for that for now...
Especially since with the pending series about changing the way we
assemble tarballs from git and svn: the tar format changes, and the
compression changes...
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
` (3 preceding siblings ...)
2020-12-19 15:35 ` [Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2021-07-29 20:17 ` Christian Stewart
2020-12-19 15:35 ` [Buildroot] [PATCH v2 06/12] package/tinifier: new package Thomas Petazzoni
` (6 subsequent siblings)
11 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
This commit introduces the download post-process script
support/download/go-post-process, and hooks it into the Go package
infrastructure.
The -modcacherw flag is added to ensure that the Go cache is
read/write, and can be deleted properly upon "make clean".
The <pkg>_LICENSE variable of golang packages is expanded with ",
vendored dependencies licenses probably not listed" as currently for
all packages, the licenses of the vendored dependencies are not taken
into account.
The logic to generate go.mod when not available is moved to the
download post-process helper, as it must be done before "go mod
vendor" is executed. Also, "go mod init" is used instead of manually
crafting go.mod. This was suggested by Christian Stewart
<christian@paral.in>. The Go module name is passed down to
go-post-process using the BR_GOMOD environment variable.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/pkg-golang.mk | 20 ++++++++++--------
support/download/go-post-process | 35 ++++++++++++++++++++++++++++++++
2 files changed, 46 insertions(+), 9 deletions(-)
create mode 100755 support/download/go-post-process
diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk
index d07242310d..00a74a4d18 100644
--- a/package/pkg-golang.mk
+++ b/package/pkg-golang.mk
@@ -42,12 +42,13 @@ define inner-golang-package
$(2)_BUILD_OPTS += \
-ldflags "$$($(2)_LDFLAGS)" \
+ -modcacherw \
-tags "$$($(2)_TAGS)" \
-trimpath \
-p $(PARALLEL_JOBS)
# Target packages need the Go compiler on the host.
-$(2)_DEPENDENCIES += host-go
+$(2)_DOWNLOAD_DEPENDENCIES += host-go
$(2)_BUILD_TARGETS ?= .
@@ -72,14 +73,15 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
# If the go.mod file does not exist, one is written with this root path.
$(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
-# Generate a go.mod file if it doesn't exist. Note: Go is configured
-# to use the "vendor" dir and not make network calls.
-define $(2)_GEN_GOMOD
- if [ ! -f $$(@D)/go.mod ]; then \
- printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \
- fi
-endef
-$(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD
+$(2)_DOWNLOAD_POST_PROCESS = go
+$(2)_DL_ENV = \
+ $(HOST_GO_COMMON_ENV) \
+ GOPROXY=direct \
+ BR_GOMOD=$$($(2)_GOMOD)
+
+# Due to vendoring, it is pretty likely that not all licenses are
+# listed in <pkg>_LICENSE.
+$(2)_LICENSE += , vendored dependencies licenses probably not listed
# Build step. Only define it if not already defined by the package .mk
# file.
diff --git a/support/download/go-post-process b/support/download/go-post-process
new file mode 100755
index 0000000000..4ca3e9a710
--- /dev/null
+++ b/support/download/go-post-process
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+# Parse our options
+while getopts "n:o:" OPT; do
+ case "${OPT}" in
+ o) output="${OPTARG}";;
+ n) base_name="${OPTARG}";;
+ :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+ \?) error "unknown option '%s'\n" "${OPTARG}";;
+ esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
+ exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Go vendoring
+pushd ${base_name} > /dev/null
+
+# Generate go.mod if it doesn't exist
+if [ ! -f go.mod ] && [ -n "${BR_GOMOD}" ]; then
+ go mod init ${BR_GOMOD}
+fi
+
+go mod vendor -v -modcacherw
+popd > /dev/null
+
+repack ${base_name} ${output}
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
2020-12-19 15:35 ` [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support Thomas Petazzoni
@ 2021-07-29 20:17 ` Christian Stewart
2021-07-29 20:50 ` Thomas Petazzoni
0 siblings, 1 reply; 24+ messages in thread
From: Christian Stewart @ 2021-07-29 20:17 UTC (permalink / raw)
To: Thomas Petazzoni
Cc: Christian Stewart, Matt Weber, Patrick Havelange, Anisse Astier,
Ryan Barnett, Buildroot List, Yann E. MORIN
Thomas, all,
On Sat, Dec 19, 2020 at 7:35 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
> The logic to generate go.mod when not available is moved to the
> download post-process helper, as it must be done before "go mod
> vendor" is executed. Also, "go mod init" is used instead of manually
> crafting go.mod. This was suggested by Christian Stewart
> <christian@paral.in>. The Go module name is passed down to
> go-post-process using the BR_GOMOD environment variable.
> +go mod vendor -v -modcacherw
> +popd > /dev/null
> +
> +repack ${base_name} ${output}
This works fine in most cases, however, this causes inconsistent hashes.
On my system, the hash is correct after re-pack. But on 2 of my users
machines, the hash ends up different.
So something specific to the host machine must be influencing the hash
of the repacked archive.
It's either something in "go mod vendor" that's not deterministic, or
maybe host Go is being used instead of the one compiled by Buildroot,
or perhaps it's not possible to always get deterministic hashes this
way.
Can we possibly switch to hashing the download /before/ re-packing it,
given that Go verifies the hashes in go.sum automatically as part of
the "go mod vendor" step?
Go mod vendor will fail if any of the dependencies are different, the
checksums are included in go.sum
Thanks & best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
2021-07-29 20:17 ` Christian Stewart
@ 2021-07-29 20:50 ` Thomas Petazzoni
2021-07-30 13:18 ` Vincent Fazio
[not found] ` <CA+h8R2onPMjOuvC0U6iM8QbhuAQQ9=aQ-yB-rWQkCbhpxcdiHw@mail.gmail.com>
0 siblings, 2 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2021-07-29 20:50 UTC (permalink / raw)
To: Christian Stewart
Cc: Matt Weber, Patrick Havelange, Anisse Astier, Ryan Barnett,
Buildroot List, Yann E. MORIN
Hello Christian,
On Thu, 29 Jul 2021 13:17:09 -0700
Christian Stewart <christian@paral.in> wrote:
> This works fine in most cases, however, this causes inconsistent hashes.
>
> On my system, the hash is correct after re-pack. But on 2 of my users
> machines, the hash ends up different.
>
> So something specific to the host machine must be influencing the hash
> of the repacked archive.
>
> It's either something in "go mod vendor" that's not deterministic, or
> maybe host Go is being used instead of the one compiled by Buildroot,
> or perhaps it's not possible to always get deterministic hashes this
> way.
Meh :-/
> Can we possibly switch to hashing the download /before/ re-packing it,
> given that Go verifies the hashes in go.sum automatically as part of
> the "go mod vendor" step?
This is going to be difficult, because the whole principle of this
patch series is to integrate the vendoring within the download step,
i.e *before* the hash is verified. For example, the hash is verified
even if you're just extracting, and not downloading.
One possibility that we had discussed in the past, but that Yann didn't
like was to create one tarball for the upstream code, and another for
the vendored dependencies.
Based on the difficulties, I would really like to understand a little
bit why the archive is not deterministic. Could you compare the
contents of the download stuff between your different machines, and the
contents of the tarball, and see what changes?
> Go mod vendor will fail if any of the dependencies are different, the
> checksums are included in go.sum
Is the presence of go.sum mandatory?
Best regards,
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
2021-07-29 20:50 ` Thomas Petazzoni
@ 2021-07-30 13:18 ` Vincent Fazio
[not found] ` <CA+h8R2onPMjOuvC0U6iM8QbhuAQQ9=aQ-yB-rWQkCbhpxcdiHw@mail.gmail.com>
1 sibling, 0 replies; 24+ messages in thread
From: Vincent Fazio @ 2021-07-30 13:18 UTC (permalink / raw)
To: Thomas Petazzoni, Christian Stewart
Cc: Matt Weber, Patrick Havelange, Anisse Astier, Ryan Barnett,
Buildroot List, Yann E. MORIN
Christian, all,
On 7/29/21 3:50 PM, Thomas Petazzoni wrote:
> Hello Christian,
>
> On Thu, 29 Jul 2021 13:17:09 -0700
> Christian Stewart <christian@paral.in> wrote:
>
>> This works fine in most cases, however, this causes inconsistent hashes.
>>
>> On my system, the hash is correct after re-pack. But on 2 of my users
>> machines, the hash ends up different.
>>
>> So something specific to the host machine must be influencing the hash
>> of the repacked archive.
>>
>> It's either something in "go mod vendor" that's not deterministic, or
>> maybe host Go is being used instead of the one compiled by Buildroot,
>> or perhaps it's not possible to always get deterministic hashes this
>> way.
> Meh :-/
>
I think Yann already calls out what may be going on in this patch
https://patchwork.ozlabs.org/project/buildroot/patch/20201219153525.1361175-5-thomas.petazzoni@bootlin.com/
The tarball format will need to follow the same or similar format we use
for other source packs by using the PAX format and trimming some
metadata being included in the archive. GNU format tarballs are not
guaranteed to be consistent across versions.
See:
https://patchwork.ozlabs.org/project/buildroot/patch/bcb281853f0da8cd970446f4afed093b317dcc82.1609239666.git.yann.morin.1998@free.fr/
--
Vincent Fazio
Embedded Software Engineer - ATS
Extreme Engineering Solutions, Inc
http://www.xes-inc.com
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread[parent not found: <CA+h8R2onPMjOuvC0U6iM8QbhuAQQ9=aQ-yB-rWQkCbhpxcdiHw@mail.gmail.com>]
* Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
[not found] ` <CA+h8R2onPMjOuvC0U6iM8QbhuAQQ9=aQ-yB-rWQkCbhpxcdiHw@mail.gmail.com>
@ 2021-08-01 7:08 ` Yann E. MORIN
[not found] ` <CA+h8R2pLN_aYiQ1vp+rTMUsQAcGT88fsAiCC-i9uJpK1y0r4rw@mail.gmail.com>
0 siblings, 1 reply; 24+ messages in thread
From: Yann E. MORIN @ 2021-08-01 7:08 UTC (permalink / raw)
To: Christian Stewart
Cc: Matt Weber, Patrick Havelange, Anisse Astier, Ryan Barnett,
Thomas Petazzoni, Buildroot List
Christian, All,
On 2021-07-31 21:59 -0700, Christian Stewart spake thusly:
> Vincent said:
> >I think Yann already calls out what may be going on in this patch
> >https://patchwork.ozlabs.org/project/buildroot/patch/20201219153525.1361175-5-thomas.petazzoni@bootlin.com/
>
> I think this is probably correct, the .tar.gz needs to have
> deterministic formatting.
Can you try usign the tar helper in support/download/helpers instead?
> In this particular case the user was trying to build on Kali 2.
>
> Doesn't buildroot compile host-gzip and host-tar for this? I'm
> wondering why it's not the same between host machines.
Since 2020.02, we are no longer building host-tar, unless for very old
versions (i.e. before 1.27), thanks to some tar trickery vy Vincent.
See:
cbe95b1a455b support/download: add helper to generate a reproducible archive
We also only build host-gzip if the host gzip is pigz (or missing).
> On Thu, Jul 29, 2021 at 1:50 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
>
> > > This works fine in most cases, however, this causes inconsistent hashes.
> > Based on the difficulties, I would really like to understand a little
> > bit why the archive is not deterministic. Could you compare the
> > contents of the download stuff between your different machines, and the
> > contents of the tarball, and see what changes?
> The two versions are attached. The actual contents of the two are
> identical. But for some reason the formatting of
> embiggen-disk-bad.tar.gz is different:
>
> 00000030: 8232 63de 4282 92dd acff fb79 6740 eae2
>
> Starting at "2dd acff" - in "embiggen-disk-good" it's
>
> 00000030: 8232 63de 4282 96dd acff fb79 6740 eae6
So, I zcat both archives, and the two tar are identical!
$ sha1sum *.tar
dbb8338ab4acaff1e232c67d3b46602d61114b53 embiggen-disk-bad.tar
dbb8338ab4acaff1e232c67d3b46602d61114b53 embiggen-disk-good.tar
So I also hashed the archives you provided, and they too are identical:
$ sha1sum *.tar.gz
bd887a60c4ca60e55d062067132ccab7d85e7d95 embiggen-disk-bad.tar.gz
bd887a60c4ca60e55d062067132ccab7d85e7d95 embiggen-disk-good.tar.gz
Did you mess up when sending the archives?
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 06/12] package/tinifier: new package
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
` (4 preceding siblings ...)
2020-12-19 15:35 ` [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure Thomas Petazzoni
` (5 subsequent siblings)
11 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
This is a Go package that needs vendor modules to be downloaded at
build time.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
DEVELOPERS | 1 +
package/Config.in | 1 +
package/tinifier/Config.in | 10 ++++++++++
package/tinifier/tinifier.hash | 3 +++
package/tinifier/tinifier.mk | 13 +++++++++++++
5 files changed, 28 insertions(+)
create mode 100644 package/tinifier/Config.in
create mode 100644 package/tinifier/tinifier.hash
create mode 100644 package/tinifier/tinifier.mk
diff --git a/DEVELOPERS b/DEVELOPERS
index 9406f30cd5..1a7c22b718 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -2590,6 +2590,7 @@ F: package/rtc-tools/
F: package/sam-ba/
F: package/scons/
F: package/squashfs/
+F: package/tinifier/
F: package/wayland/
F: package/weston/
F: support/testing/tests/boot/test_syslinux.py
diff --git a/package/Config.in b/package/Config.in
index dc7139a49a..d5d8caef47 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -298,6 +298,7 @@ comment "Graphic applications"
source "package/rrdtool/Config.in"
source "package/stellarium/Config.in"
source "package/tesseract-ocr/Config.in"
+ source "package/tinifier/Config.in"
comment "Graphic libraries"
source "package/cegui/Config.in"
diff --git a/package/tinifier/Config.in b/package/tinifier/Config.in
new file mode 100644
index 0000000000..fbadfe6bd9
--- /dev/null
+++ b/package/tinifier/Config.in
@@ -0,0 +1,10 @@
+config BR2_PACKAGE_TINIFIER
+ bool "tinifier"
+ depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+ help
+ CLI tool for images compressing
+
+ This tool uses tinypng.com API endpoint for compressing your
+ local jpg/png images (it supports parallel jobs).
+
+ https://github.com/tarampampam/tinifier
diff --git a/package/tinifier/tinifier.hash b/package/tinifier/tinifier.hash
new file mode 100644
index 0000000000..146700817b
--- /dev/null
+++ b/package/tinifier/tinifier.hash
@@ -0,0 +1,3 @@
+# Locally calculated
+sha256 707a1d9e55aab8c83b65bb10f3ec8c3bc094d77cd576b266f821d9f39133ac3c tinifier-2.1.0.tar.gz
+sha256 791d8fd993ace44d4d83e2f4820a64d5ad3e37412f029afad46d17a9259de2b6 LICENSE
diff --git a/package/tinifier/tinifier.mk b/package/tinifier/tinifier.mk
new file mode 100644
index 0000000000..b47d265a8e
--- /dev/null
+++ b/package/tinifier/tinifier.mk
@@ -0,0 +1,13 @@
+################################################################################
+#
+# tinifier
+#
+################################################################################
+
+TINIFIER_VERSION = 2.1.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
+TINIFIER_LICENSE = MIT
+TINIFIER_LICENSE_FILES = LICENSE
+TINIFIER_GOMOD = tinifier
+
+$(eval $(golang-package))
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
` (5 preceding siblings ...)
2020-12-19 15:35 ` [Buildroot] [PATCH v2 06/12] package/tinifier: new package Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 08/12] docs/manual/cargo: document the cargo-package infrastructure Thomas Petazzoni
` (4 subsequent siblings)
11 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
From: Patrick Havelange <patrick.havelange@essensium.com>
In order to be package agnostic, the install phase is now using cargo
instead of install. TARGET_CONFIGURE_OPTS is now also set when running
cargo in order to support cross compiling C code within cargo.
Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
[Thomas: add support for host-cargo-package]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/Makefile.in | 1 +
package/pkg-cargo.mk | 140 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 141 insertions(+)
create mode 100644 package/pkg-cargo.mk
diff --git a/package/Makefile.in b/package/Makefile.in
index 51f5cbce4f..2af123f36d 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -427,3 +427,4 @@ include package/pkg-waf.mk
include package/pkg-golang.mk
include package/pkg-meson.mk
include package/pkg-qmake.mk
+include package/pkg-cargo.mk
diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
new file mode 100644
index 0000000000..058c6756bb
--- /dev/null
+++ b/package/pkg-cargo.mk
@@ -0,0 +1,140 @@
+################################################################################
+# Cargo package infrastructure
+#
+# This file implements an infrastructure that eases development of package
+# .mk files for Cargo packages. It should be used for all packages that use
+# Cargo as their build system.
+#
+# See the Buildroot documentation for details on the usage of this
+# infrastructure
+#
+# In terms of implementation, this Cargo infrastructure requires the .mk file
+# to only specify metadata information about the package: name, version,
+# download URL, etc.
+#
+# We still allow the package .mk file to override what the different steps
+# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined,
+# it is used as the list of commands to perform to build the package,
+# instead of the default Cargo behaviour. The package can also define some
+# post operation hooks.
+#
+################################################################################
+
+################################################################################
+# inner-cargo-package -- defines how the configuration, compilation and
+# installation of a cargo package should be done, implements a few hooks
+# to tune the build process for cargo specifities and calls the generic
+# package infrastructure to generate the necessary make targets
+#
+# argument 1 is the lowercase package name
+# argument 2 is the uppercase package name, including a HOST_ prefix
+# for host packages
+# argument 3 is the uppercase package name, without the HOST_ prefix
+# for host packages
+# argument 4 is the type (target or host)
+################################################################################
+
+define inner-cargo-package
+
+# We need host-rustc to run cargo
+$(2)_DEPENDENCIES += host-rustc
+
+$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+# Note: in all the steps below, we "cd" into the build directory to
+# execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the
+# manifest-path. Indeed while the latter seems to work, it in fact
+# breaks in subtle ways as the way cargo searches for its
+# configuration file is based (among other rules) on the current
+# directory. This means that if cargo is started outside of a package
+# directory, its configuration file will not be taken into account.
+#
+# Also, we pass:
+# * --offline to prevent cargo from downloading anything
+# * --locked to force cargo to use the Cargo.lock file, which ensures
+# that a fixed set of dependency versions is used
+
+#
+# Build step. Only define it if not already defined by the package .mk
+# file.
+#
+ifndef $(2)_BUILD_CMDS
+ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+ cd $$(@D) && \
+ $$(TARGET_MAKE_ENV) \
+ $$(TARGET_CONFIGURE_OPTS) \
+ $$($(2)_CARGO_ENV) \
+ cargo build \
+ --offline \
+ --target $$(RUSTC_TARGET_NAME) \
+ $$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
+ --manifest-path Cargo.toml \
+ --locked \
+ $$($(2)_CARGO_BUILD_OPTS)
+endef
+else # ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+ cd $$(@D) && \
+ $$(HOST_MAKE_ENV) \
+ RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+ $$($(2)_CARGO_ENV) \
+ cargo build \
+ --offline \
+ --release \
+ --manifest-path Cargo.toml \
+ --locked \
+ $$($(2)_CARGO_BUILD_OPTS)
+endef
+endif # ifeq ($(4),target)
+endif # ifndef $(2)_BUILD_CMDS
+
+#
+# Target installation step. Only define it if not already defined by
+# the package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+ cd $$(@D) && \
+ $$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
+ cargo install \
+ --target $$(RUSTC_TARGET_NAME) \
+ --offline \
+ --root $$(TARGET_DIR)/usr/ \
+ --bins \
+ --path ./ \
+ --force \
+ --locked \
+ $$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+ cd $$(@D) && \
+ $$(HOST_MAKE_ENV) \
+ RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+ $$($(2)_CARGO_ENV) \
+ cargo install \
+ --offline \
+ --root $$(HOST_DIR) \
+ --bins \
+ --path ./ \
+ --force \
+ --locked \
+ $$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+# Call the generic package infrastructure to generate the necessary
+# make targets
+$(call inner-generic-package,$(1),$(2),$(3),$(4))
+
+endef
+
+################################################################################
+# cargo-package -- the target generator macro for Cargo packages
+################################################################################
+
+cargo-package = $(call inner-cargo-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)
+host-cargo-package = $(call inner-cargo-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host)
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 08/12] docs/manual/cargo: document the cargo-package infrastructure
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
` (6 preceding siblings ...)
2020-12-19 15:35 ` [Buildroot] [PATCH v2 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 09/12] package/ripgrep: convert to cargo infrastructure Thomas Petazzoni
` (3 subsequent siblings)
11 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
From: Patrick Havelange <patrick.havelange@essensium.com>
The Buildroot manual was already providing some details on how to
integrate Cargo packages, and those details now need to be updated
with a proper documentation for the cargo-package infrastructure.
Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
[Thomas: numerous updates and extensions.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
docs/manual/adding-packages-cargo.txt | 88 ++++++++++++---------------
1 file changed, 39 insertions(+), 49 deletions(-)
diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt
index 8fcc80bcc6..c65a32f017 100644
--- a/docs/manual/adding-packages-cargo.txt
+++ b/docs/manual/adding-packages-cargo.txt
@@ -1,7 +1,7 @@
// -*- mode:doc; -*-
// vim: set syntax=asciidoc:
-=== Integration of Cargo-based packages
+=== Infrastructure for Cargo-based packages
Cargo is the package manager for the Rust programming language. It allows the
user to build programs or libraries written in Rust, but it also downloads and
@@ -10,7 +10,7 @@ called "crates".
[[cargo-package-tutorial]]
-==== Cargo-based package's +Config.in+ file
+==== +cargo-package+ tutorial
The +Config.in+ file of Cargo-based package 'foo' should contain:
@@ -25,11 +25,7 @@ The +Config.in+ file of Cargo-based package 'foo' should contain:
08: http://foosoftware.org/foo/
---------------------------
-==== Cargo-based package's +.mk+ file
-
-Buildroot does not (yet) provide a dedicated package infrastructure for
-Cargo-based packages. So, we will explain how to write a +.mk+ file for such a
-package. Let's start with an example:
+And the +.mk+ file for this package should contain:
------------------------------
01: ################################################################################
@@ -44,52 +40,48 @@ package. Let's start with an example:
10: FOO_LICENSE = GPL-3.0+
11: FOO_LICENSE_FILES = COPYING
12:
-13: FOO_DEPENDENCIES = host-rustc
-14:
-15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-16:
-17: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE)
-18:
-19: FOO_CARGO_OPTS = \
-20: $(if $(BR2_ENABLE_DEBUG),,--release) \
-21: --target=$(RUSTC_TARGET_NAME) \
-22: --manifest-path=$(@D)/Cargo.toml
-23:
-24: define FOO_BUILD_CMDS
-25: $(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \
-26: cargo build $(FOO_CARGO_OPTS)
-27: endef
-28:
-29: define FOO_INSTALL_TARGET_CMDS
-30: $(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \
-31: $(TARGET_DIR)/usr/bin/foo
-32: endef
-33:
-34: $(eval $(generic-package))
+13: $(eval $(cargo-package))
--------------------------------
-The Makefile starts with the definition of the standard variables for package
-declaration (lines 7 to 11).
+The Makefile starts with the definition of the standard variables for
+package declaration (lines 7 to 11).
+
+As seen in line 13, it is based on the +cargo-package+
+infrastructure. Cargo will be invoked automatically by this
+infrastructure to build and install the package.
+
+It is still possible to define custom build commands or install
+commands (i.e. with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS).
+Those will then replace the commands from the cargo infrastructure.
+
+==== +cargo-package+ reference
+
+The main macros for the Cargo package infrastructure are
++cargo-package+ for target packages and +host-cargo-package+ for host
+packages.
+
+Just like the generic infrastructure, the Cargo infrastructure works
+by defining a number of variables before calling the +cargo-package+
+or +host-cargo-package+ macros.
-As seen in line 34, it is based on the
-xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines
-the variables required by this particular infrastructure, where Cargo is
-invoked:
+First, all the package metadata information variables that exist in
+the generic infrastructure also exist in the Cargo infrastructure:
++FOO_VERSION+, +FOO_SOURCE+, +FOO_PATCH+, +FOO_SITE+,
++FOO_DEPENDENCIES+, +FOO_LICENSE+, +FOO_LICENSE_FILES+, etc.
-* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required
- to configure the cross-compilation of the package are passed via
- +FOO_CONF_OPTS+.
+A few additional variables, specific to the Cargo infrastructure, can
+also be defined. Many of them are only useful in very specific cases,
+typical packages will therefore only use a few of them.
-* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on
- the target.
+* +FOO_CARGO_ENV+ can be used to pass additional variables in the
+ environment of +cargo+ invocations. It used at both build and
+ installation time
-In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to
-contain +host-cargo+.
+* +FOO_CARGO_BUILD_OPTS+ can be used to pass additional options to
+ +cargo+ at build time.
-To sum it up, to add a new Cargo-based package, the Makefile example can be
-copied verbatim then edited to replace all occurences of +FOO+ with the
-uppercase name of the new package and update the values of the standard
-variables.
+* +FOO_CARGO_INSTALL_OPTS+ can be used to pass additional options to
+ +cargo+ at install time.
==== About Dependencies Management
@@ -99,9 +91,7 @@ automatically them. This step can also be performed independently, via the
+cargo fetch+ command.
Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+. As seen in the package
-Makefile example at line 15, this environment variable is set to
-+$(HOST_DIR)/share/cargo+.
+crates, whose location is given by +$CARGO_HOME+.
This dependency download mechanism is not convenient when performing an offline
build, as Cargo will fail to fetch the dependencies. In that case, it is advised
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 09/12] package/ripgrep: convert to cargo infrastructure
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
` (7 preceding siblings ...)
2020-12-19 15:35 ` [Buildroot] [PATCH v2 08/12] docs/manual/cargo: document the cargo-package infrastructure Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 10/12] package/sentry-cli: convert to host-cargo-package infrastructure Thomas Petazzoni
` (2 subsequent siblings)
11 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
From: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/ripgrep/ripgrep.mk | 28 +---------------------------
1 file changed, 1 insertion(+), 27 deletions(-)
diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk
index 9dd8d58de1..97e9e2ce5f 100644
--- a/package/ripgrep/ripgrep.mk
+++ b/package/ripgrep/ripgrep.mk
@@ -9,30 +9,4 @@ RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
RIPGREP_LICENSE = MIT
RIPGREP_LICENSE_FILES = LICENSE-MIT
-RIPGREP_DEPENDENCIES = host-rustc
-RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-
-RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_BIN_SUBDIR)
-
-RIPGREP_CARGO_OPTS = \
- --target=$(RUSTC_TARGET_NAME) \
- --manifest-path=$(@D)/Cargo.toml
-
-ifeq ($(BR2_ENABLE_DEBUG),y)
-RIPGREP_CARGO_BIN_SUBDIR = debug
-else
-RIPGREP_CARGO_OPTS += --release
-RIPGREP_CARGO_BIN_SUBDIR = release
-endif
-
-define RIPGREP_BUILD_CMDS
- $(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \
- cargo build $(RIPGREP_CARGO_OPTS)
-endef
-
-define RIPGREP_INSTALL_TARGET_CMDS
- $(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \
- $(TARGET_DIR)/usr/bin/rg
-endef
-
-$(eval $(generic-package))
+$(eval $(cargo-package))
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 10/12] package/sentry-cli: convert to host-cargo-package infrastructure
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
` (8 preceding siblings ...)
2020-12-19 15:35 ` [Buildroot] [PATCH v2 09/12] package/ripgrep: convert to cargo infrastructure Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 11/12] support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 12/12] docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management Thomas Petazzoni
11 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/sentry-cli/sentry-cli.mk | 22 ++--------------------
1 file changed, 2 insertions(+), 20 deletions(-)
diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk
index 7e53f16026..40ca22fdf9 100644
--- a/package/sentry-cli/sentry-cli.mk
+++ b/package/sentry-cli/sentry-cli.mk
@@ -9,24 +9,6 @@ SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
SENTRY_CLI_LICENSE = BSD-3-clause
SENTRY_CLI_LICENSE_FILES = LICENSE
-HOST_SENTRY_CLI_DEPENDENCIES = host-rustc host-zlib
+HOST_SENTRY_CLI_DEPENDENCIES = host-zlib
-HOST_SENTRY_CLI_CARGO_ENV = \
- CARGO_HOME=$(HOST_DIR)/share/cargo \
- RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))"
-
-HOST_SENTRY_CLI_CARGO_OPTS = \
- --release \
- --manifest-path=$(@D)/Cargo.toml
-
-define HOST_SENTRY_CLI_BUILD_CMDS
- $(HOST_MAKE_ENV) $(HOST_SENTRY_CLI_CARGO_ENV) \
- cargo build $(HOST_SENTRY_CLI_CARGO_OPTS)
-endef
-
-define HOST_SENTRY_CLI_INSTALL_CMDS
- $(INSTALL) -D -m 0755 $(@D)/target/release/sentry-cli \
- $(HOST_DIR)/bin/sentry-cli
-endef
-
-$(eval $(host-generic-package))
+$(eval $(host-cargo-package))
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 11/12] support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
` (9 preceding siblings ...)
2020-12-19 15:35 ` [Buildroot] [PATCH v2 10/12] package/sentry-cli: convert to host-cargo-package infrastructure Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
2020-12-19 15:35 ` [Buildroot] [PATCH v2 12/12] docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management Thomas Petazzoni
11 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
This commit adds support/download/cargo-post-process to perform the
vendoring on Cargo packages, and enables it in package/pkg-cargo.mk.
Since it changes the contents of the tarballs for ripgrep and
sentry-cli, it changes their hashes. To not have a different hash for
the same version of ripgrep and sentry-cli, we bump their versions. It
has to be done in the same commit as the Cargo vendoring to make the
series bisectable.
The <pkg>_LICENSE variable of cargo packages is expanded with ",
vendored dependencies licenses probably not listed" as currently for
all packages, the licenses of the vendored dependencies are not taken
into account.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/pkg-cargo.mk | 13 ++++++++--
package/ripgrep/ripgrep.hash | 2 +-
package/ripgrep/ripgrep.mk | 2 +-
package/sentry-cli/sentry-cli.hash | 2 +-
package/sentry-cli/sentry-cli.mk | 2 +-
support/download/cargo-post-process | 38 +++++++++++++++++++++++++++++
6 files changed, 53 insertions(+), 6 deletions(-)
create mode 100755 support/download/cargo-post-process
diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk
index 058c6756bb..371b8dd424 100644
--- a/package/pkg-cargo.mk
+++ b/package/pkg-cargo.mk
@@ -37,10 +37,17 @@
define inner-cargo-package
# We need host-rustc to run cargo
-$(2)_DEPENDENCIES += host-rustc
+$(2)_DOWNLOAD_DEPENDENCIES += host-rustc
$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
+$(2)_DOWNLOAD_POST_PROCESS = cargo
+$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+# Due to vendoring, it is pretty likely that not all licenses are
+# listed in <pkg>_LICENSE.
+$(2)_LICENSE += , vendored dependencies licenses probably not listed
+
# Note: in all the steps below, we "cd" into the build directory to
# execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the
# manifest-path. Indeed while the latter seems to work, it in fact
@@ -50,7 +57,9 @@ $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
# directory, its configuration file will not be taken into account.
#
# Also, we pass:
-# * --offline to prevent cargo from downloading anything
+# * --offline to prevent cargo from downloading anything: all
+# dependencies should have been built by the download post
+# process logic
# * --locked to force cargo to use the Cargo.lock file, which ensures
# that a fixed set of dependency versions is used
diff --git a/package/ripgrep/ripgrep.hash b/package/ripgrep/ripgrep.hash
index 0841c0185c..81ac905d3d 100644
--- a/package/ripgrep/ripgrep.hash
+++ b/package/ripgrep/ripgrep.hash
@@ -1,3 +1,3 @@
# Locally calculated
-sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz
+sha256 80d7f07325f4d485c5e4baf061b0376b45a497ee7a1c540d7894057bb9ac3a59 ripgrep-12.1.1.tar.gz
sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT
diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk
index 97e9e2ce5f..d21f88a6e8 100644
--- a/package/ripgrep/ripgrep.mk
+++ b/package/ripgrep/ripgrep.mk
@@ -4,7 +4,7 @@
#
################################################################################
-RIPGREP_VERSION = 0.8.1
+RIPGREP_VERSION = 12.1.1
RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
RIPGREP_LICENSE = MIT
RIPGREP_LICENSE_FILES = LICENSE-MIT
diff --git a/package/sentry-cli/sentry-cli.hash b/package/sentry-cli/sentry-cli.hash
index 3b0733a276..63b0c812bc 100644
--- a/package/sentry-cli/sentry-cli.hash
+++ b/package/sentry-cli/sentry-cli.hash
@@ -1,3 +1,3 @@
# locally calculated
-sha256 5d0f7acf6a139d1c1716b9a8ff76c8bfaab09104ba663c957bb9a5dba2ffbabd sentry-cli-1.57.0.tar.gz
+sha256 a657dd1a46e3de044deff4c311b7276c5a9409582e707e6e6a78b0cf712591c4 sentry-cli-1.59.0.tar.gz
sha256 9503def7b54ceb6e3cd182fd59bc05d3a30d7eae481e65aaba4b495133c83c14 LICENSE
diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk
index 40ca22fdf9..58f5f1e325 100644
--- a/package/sentry-cli/sentry-cli.mk
+++ b/package/sentry-cli/sentry-cli.mk
@@ -4,7 +4,7 @@
#
################################################################################
-SENTRY_CLI_VERSION = 1.57.0
+SENTRY_CLI_VERSION = 1.59.0
SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
SENTRY_CLI_LICENSE = BSD-3-clause
SENTRY_CLI_LICENSE_FILES = LICENSE
diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process
new file mode 100755
index 0000000000..5081476385
--- /dev/null
+++ b/support/download/cargo-post-process
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+while getopts "n:o:" OPT; do
+ case "${OPT}" in
+ o) output="${OPTARG}";;
+ n) base_name="${OPTARG}";;
+ :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+ \?) error "unknown option '%s'\n" "${OPTARG}";;
+ esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then
+ exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Cargo vendoring
+pushd ${base_name} > /dev/null
+cargo vendor --locked VENDOR
+echo $?
+# Create the local .cargo/config with vendor info
+mkdir -p .cargo/
+cat <<EOF >.cargo/config
+[source.crates-io]
+replace-with = "vendored-sources"
+
+[source.vendored-sources]
+directory = "VENDOR"
+EOF
+popd > /dev/null
+
+repack ${base_name} ${output}
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH v2 12/12] docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management
2020-12-19 15:35 [Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring Thomas Petazzoni
` (10 preceding siblings ...)
2020-12-19 15:35 ` [Buildroot] [PATCH v2 11/12] support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages Thomas Petazzoni
@ 2020-12-19 15:35 ` Thomas Petazzoni
11 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2020-12-19 15:35 UTC (permalink / raw)
To: buildroot
Now that we have vendoring support for Cargo packages, let's rewrite
the dependency management section in a more accurate way.
We drop the part about the local cache of the registry, because
+CARGO_HOME+ in Buildroot points to $(HOST_DIR)/share/cargo, which is
not shared between builds nor preserved accross builds, so its effect
as a cache is limited.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
docs/manual/adding-packages-cargo.txt | 22 ++++++++++------------
1 file changed, 10 insertions(+), 12 deletions(-)
diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt
index c65a32f017..fb3e7d0780 100644
--- a/docs/manual/adding-packages-cargo.txt
+++ b/docs/manual/adding-packages-cargo.txt
@@ -85,15 +85,13 @@ typical packages will therefore only use a few of them.
==== About Dependencies Management
-A crate can depend on other libraries from crates.io or git repositories, listed
-in its Cargo.toml file. Before starting a build, Cargo usually downloads
-automatically them. This step can also be performed independently, via the
-+cargo fetch+ command.
-
-Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+.
-
-This dependency download mechanism is not convenient when performing an offline
-build, as Cargo will fail to fetch the dependencies. In that case, it is advised
-to generate a tarball of the dependencies using the +cargo vendor+ and add it to
-+FOO_EXTRA_DOWNLOADS+.
+A crate can depend on other libraries from crates.io or git
+repositories, listed in its Cargo.toml file. Buildroot automatically
+takes care of downloading such dependencies as part of the download
+step of packages that use the +cargo-package+ infrastructure. Such
+dependencies are then kept together with the package source code in
+the tarball cached in Buildroot's +DL_DIR+, and therefore the hash of
+the package's tarball includes such dependencies.
+
+This mechanism ensures that any change in the dependencies will be
+detected, and allows the build to be performed completely offline.
--
2.29.2
^ permalink raw reply related [flat|nested] 24+ messages in thread