* [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature
@ 2017-07-04 16:21 Maxime Hadjinlian
2017-07-04 16:21 ` [Buildroot] [PATCH 01/13] pkg-{download, generic}: remove source-check Maxime Hadjinlian
` (13 more replies)
0 siblings, 14 replies; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:21 UTC (permalink / raw)
To: buildroot
Hi all,
This series will progressively reorganize the DL_DIR folder, in order to
hve a subdirectory for each packages.
This way, we store patches into subdirectory called by the package name
instead of flattly into DL_DIR.
We also avoid possible conflicts between packages.
Finally, this series introduce the 'git cache' feature. What it does is
that it keeps a copy of the Git clone in the subdirectory of the
package.
This way, when you change the version of said package, instead of
cloning from scratch, we only fetch the missing objects and create the
tarball from that git clone.
It should speed up the build, especially for some use cases; drawback is
that the disk usage for the DL_DIR will increase significantly. But disk
space are pretty cheap, especially on build servers.
Maxime Hadjinlian (12):
pkg-{download,generic}: remove source-check
download: put most of the infra in dl-wrapper
pkg-generic: make PKG_DL_DIR equal to DL_DIR
packages: use new $($PKG)_DL_DIR) variable
pkg-{download,generic}: use new $($(PKG)_DL_DIR)
support/download: make sure the download folder is created
pkg-generic: add a subdirectory to the DL_DIR
pkg-download: support new subdir for mirrors
pkg-generic: introduce _SAME_SOURCE_AS
help/manual: update help about the new $(LIBFOO_DL_DIR)
download: add flock call before dl-wrapper
download: git: introduce cache feature
Yann E. MORIN (1):
core/pkg-download: change all helpers to use common options
Config.in | 3 +
docs/manual/adding-packages-generic.txt | 6 +-
package/amd-catalyst/amd-catalyst.mk | 2 +-
package/android-tools/android-tools.mk | 2 +-
package/angularjs/angularjs.mk | 2 +-
package/bootstrap/bootstrap.mk | 2 +-
package/cache-calibrator/cache-calibrator.mk | 2 +-
package/cracklib/cracklib.mk | 2 +-
package/cryptopp/cryptopp.mk | 2 +-
package/devmem2/devmem2.mk | 2 +-
package/dhrystone/dhrystone.mk | 2 +-
package/doom-wad/doom-wad.mk | 2 +-
package/espeak/espeak.mk | 2 +-
package/fan-ctrl/fan-ctrl.mk | 2 +-
package/gcc/gcc.mk | 2 +-
package/irrlicht/irrlicht.mk | 2 +-
package/jquery-mobile/jquery-mobile.mk | 2 +-
package/jquery-sparkline/jquery-sparkline.mk | 2 +-
package/jquery-ui-themes/jquery-ui-themes.mk | 2 +-
package/jquery-ui/jquery-ui.mk | 2 +-
package/jquery-validation/jquery-validation.mk | 2 +-
package/jquery/jquery.mk | 2 +-
package/kodi/kodi.mk | 6 +-
package/libfreeimage/libfreeimage.mk | 2 +-
package/libjson/libjson.mk | 2 +-
package/libsoil/libsoil.mk | 2 +-
package/lsof/lsof.mk | 2 +-
package/musl-compat-headers/musl-compat-headers.mk | 2 +-
package/nmon/nmon.mk | 2 +-
package/nvidia-driver/nvidia-driver.mk | 2 +-
package/openobex/openobex.mk | 2 +-
package/opentyrian-data/opentyrian-data.mk | 2 +-
package/perl/perl.mk | 2 +-
package/pkg-download.mk | 216 ++++-----------------
package/pkg-generic.mk | 10 +-
package/python-keyring/python-keyring.mk | 2 +-
.../python-simplegeneric/python-simplegeneric.mk | 2 +-
package/rapidxml/rapidxml.mk | 2 +-
package/rpi-wifi-firmware/rpi-wifi-firmware.mk | 2 +-
package/sam-ba/sam-ba.mk | 2 +-
package/spidev_test/spidev_test.mk | 2 +-
package/tar/tar.mk | 2 +-
package/tesseract-ocr/tesseract-ocr.mk | 2 +-
package/ti-cgt-pru/ti-cgt-pru.mk | 4 +-
package/ti-gfx/ti-gfx.mk | 4 +-
package/ts4900-fpga/ts4900-fpga.mk | 2 +-
package/unscd/unscd.mk | 2 +-
package/urg/urg.mk | 2 +-
package/waf/waf.mk | 2 +-
package/whetstone/whetstone.mk | 2 +-
package/wilc1000-firmware/wilc1000-firmware.mk | 2 +-
package/zynq-boot-bin/zynq-boot-bin.mk | 2 +-
support/download/bzr | 25 +--
support/download/check-hash | 2 +-
support/download/cp | 17 +-
support/download/cvs | 34 ++--
support/download/dl-wrapper | 62 +++---
support/download/git | 96 +++++----
support/download/hg | 25 +--
support/download/scp | 19 +-
support/download/svn | 25 +--
support/download/wget | 24 ++-
62 files changed, 286 insertions(+), 382 deletions(-)
--
2.13.2
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 01/13] pkg-{download, generic}: remove source-check
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
@ 2017-07-04 16:21 ` Maxime Hadjinlian
2017-07-04 17:37 ` Thomas Petazzoni
2017-07-04 16:22 ` [Buildroot] [PATCH 02/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (12 subsequent siblings)
13 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:21 UTC (permalink / raw)
To: buildroot
This feature is not used by anyone in the core developpers and makes a
drastic simplification of the pkg-download infrastructure harder.
The future patch will move much of what's in the current pkg-download.mk
file into the dl-wrapper which is a shell script.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-download.mk | 44 --------------------------------------------
package/pkg-generic.mk | 4 ----
2 files changed, 48 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index fbbc2d7391..3712b9ccc6 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -68,9 +68,6 @@ export BR_NO_CHECK_HASH_FOR =
# ssh authentication. DOWNLOAD_WGET is the normal wget-based download
# mechanism.
#
-# The SOURCE_CHECK_* helpers are in charge of simply checking that the source
-# is available for download. This can be used to make sure one will be able
-# to get all the sources needed for one's build configuration.
################################################################################
define DOWNLOAD_GIT
@@ -86,12 +83,6 @@ define DOWNLOAD_GIT
$($(PKG)_DL_OPTS)
endef
-# TODO: improve to check that the given PKG_DL_VERSION exists on the remote
-# repository
-define SOURCE_CHECK_GIT
- $(GIT) ls-remote --heads $($(PKG)_SITE) > /dev/null
-endef
-
define DOWNLOAD_BZR
$(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
@@ -103,10 +94,6 @@ define DOWNLOAD_BZR
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_BZR
- $(BZR) ls --quiet $($(PKG)_SITE) > /dev/null
-endef
-
define DOWNLOAD_CVS
$(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
@@ -119,11 +106,6 @@ define DOWNLOAD_CVS
$($(PKG)_DL_OPTS)
endef
-# Not all CVS servers support ls/rls, use login to see if we can connect
-define SOURCE_CHECK_CVS
- $(CVS) -d:pserver:anonymous:@$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) login
-endef
-
define DOWNLOAD_SVN
$(EXTRA_ENV) $(DL_WRAPPER) -b svn \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
@@ -135,10 +117,6 @@ define DOWNLOAD_SVN
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_SVN
- $(SVN) ls $($(PKG)_SITE)@$($(PKG)_DL_VERSION) > /dev/null
-endef
-
# SCP URIs should be of the form scp://[user@]host:filepath
# Note that filepath is relative to the user's home directory, so you may want
# to prepend the path with a slash: scp://[user@]host:/absolutepath
@@ -152,10 +130,6 @@ define DOWNLOAD_SCP
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_SCP
- $(SSH) $(call domain,$(1),:) ls '$(call notdomain,$(1),:)' > /dev/null
-endef
-
define DOWNLOAD_HG
$(EXTRA_ENV) $(DL_WRAPPER) -b hg \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
@@ -167,12 +141,6 @@ define DOWNLOAD_HG
$($(PKG)_DL_OPTS)
endef
-# TODO: improve to check that the given PKG_DL_VERSION exists on the remote
-# repository
-define SOURCE_CHECK_HG
- $(HG) incoming --force -l1 $($(PKG)_SITE) > /dev/null
-endef
-
define DOWNLOAD_WGET
$(EXTRA_ENV) $(DL_WRAPPER) -b wget \
-o $(DL_DIR)/$(2) \
@@ -183,10 +151,6 @@ define DOWNLOAD_WGET
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_WGET
- $(WGET) --spider '$(call qstrip,$(1))'
-endef
-
define DOWNLOAD_LOCALFILES
$(EXTRA_ENV) $(DL_WRAPPER) -b cp \
-o $(DL_DIR)/$(2) \
@@ -197,10 +161,6 @@ define DOWNLOAD_LOCALFILES
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_LOCALFILES
- test -e $(call stripurischeme,$(call qstrip,$(1)))
-endef
-
################################################################################
# DOWNLOAD -- Download helper. Will try to download source from:
# 1) BR2_PRIMARY_SITE if enabled
@@ -221,10 +181,6 @@ define DOWNLOAD
$(call DOWNLOAD_INNER,$(1),$(notdir $(1)),DOWNLOAD)
endef
-define SOURCE_CHECK
- $(call DOWNLOAD_INNER,$(1),$(notdir $(1)),SOURCE_CHECK)
-endef
-
define DOWNLOAD_INNER
$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 2916a7bbad..79bea906e2 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -694,10 +694,6 @@ $(1)-legal-source: $$($(2)_TARGET_ACTUAL_SOURCE)
endif # actual sources != sources
endif # actual sources != ""
-$(1)-source-check: PKG=$(2)
-$(1)-source-check:
- $$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep))
-
$(1)-external-deps:
@for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \
echo `basename $$$$p` ; \
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 02/13] core/pkg-download: change all helpers to use common options
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
2017-07-04 16:21 ` [Buildroot] [PATCH 01/13] pkg-{download, generic}: remove source-check Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-07-04 23:09 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 03/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
` (11 subsequent siblings)
13 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
From: "Yann E. MORIN" <yann.morin.1998@free.fr>
Currently all download helpers accepts the local output file, the remote
locations, the changesets and so on... as positional arguments.
This was well and nice when that's was all we needed.
But then we added an option to quiesce their verbosity, and that was
shoehorned with a trivial getopts, still keeping all the existing
positional arguments as... positional arguments.
Adding yet more options while keeping positional arguments will not be
very easy, even if we do not envision any new option in the foreseeable
future (but 640K ought to be enough for everyone, remember? ;-) ).
Change all helpers to accept a set of generic options (-q for quiet and
-o for the output file) as well as helper-specific options (like -r for
the repository, -c for a changeset...).
Maxime:
Changed -R to -r for recurse (only for the git backend)
Changed -r to -u for URI (for all backend)
Change -R to -c for cset (for CVS and SVN backend)
Add the export of the BR_BACKEND_DL_GETOPTS so all the backend wrapper
can use the same option easily
Now all the backends use the same common options.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
package/pkg-download.mk | 38 +++++++++++++++++++-------------------
support/download/bzr | 25 ++++++++++++++-----------
support/download/check-hash | 2 +-
support/download/cp | 17 +++++++++--------
support/download/cvs | 34 +++++++++++++++++++---------------
support/download/dl-wrapper | 7 ++++++-
support/download/git | 33 +++++++++++++++++----------------
support/download/hg | 25 ++++++++++++++-----------
support/download/scp | 19 ++++++++++---------
support/download/svn | 25 ++++++++++++++-----------
support/download/wget | 17 +++++++++--------
11 files changed, 132 insertions(+), 110 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 3712b9ccc6..ce069b9926 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -77,9 +77,9 @@ define DOWNLOAD_GIT
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -88,9 +88,9 @@ define DOWNLOAD_BZR
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -99,10 +99,10 @@ define DOWNLOAD_CVS
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAWNAME) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
+ -c $($(PKG)_DL_VERSION) \
+ -N $($(PKG)_RAWNAME) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -111,9 +111,9 @@ define DOWNLOAD_SVN
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -126,7 +126,7 @@ define DOWNLOAD_SCP
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- '$(call stripurischeme,$(call qstrip,$(1)))' \
+ -u '$(call stripurischeme,$(call qstrip,$(1)))' \
$($(PKG)_DL_OPTS)
endef
@@ -135,9 +135,9 @@ define DOWNLOAD_HG
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -147,7 +147,7 @@ define DOWNLOAD_WGET
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- '$(call qstrip,$(1))' \
+ -u '$(call qstrip,$(1))' \
$($(PKG)_DL_OPTS)
endef
@@ -157,7 +157,7 @@ define DOWNLOAD_LOCALFILES
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- $(call stripurischeme,$(call qstrip,$(1))) \
+ -u $(call stripurischeme,$(call qstrip,$(1))) \
$($(PKG)_DL_OPTS)
endef
diff --git a/support/download/bzr b/support/download/bzr
index 75b7b415c1..a70cb19cf1 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -5,28 +5,31 @@ set -e
# Download helper for bzr, to be called from the download wrapper script
#
-# Call it as:
-# .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
+# Options:
+# -q Be quiet
+# -o FILE Generate archive in FILE.
+# -u URI Clone from repository URI.
+# -c CSET Use changeset (or revision) CSET.
+# -n NAME Use basename NAME.
#
# Environment:
# BZR : the bzr command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) cset="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-rev="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -51,5 +54,5 @@ if [ ${bzr_version} -ge ${bzr_min_version} ]; then
fi
_bzr export ${verbose} --root="'${basename}/'" --format=tgz \
- ${timestamp_opt} - "${@}" "'${repo}'" -r "'${rev}'" \
+ ${timestamp_opt} - "${@}" "'${uri}'" -r "'${cset}'" \
>"${output}"
diff --git a/support/download/check-hash b/support/download/check-hash
index c1ff53c02b..bf2dbb1a8b 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -19,7 +19,7 @@ set -e
# 3: the hash file exists and there was no hash to check the file against
# 4: the hash file exists and at least one hash type is unknown
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) exec >/dev/null;;
\?) exit 1;;
diff --git a/support/download/cp b/support/download/cp
index 0ee1f3ba82..52fe2de83d 100755
--- a/support/download/cp
+++ b/support/download/cp
@@ -5,8 +5,10 @@ set -e
# Download helper for cp, to be called from the download wrapper script
#
-# Call it as:
-# .../cp [-q] OUT_FILE SRC_FILE
+# Options:
+# -q Be quiet.
+# -o FILE Copy to file FILE.
+# -u FILE Copy from file FILE.
#
# Environment:
# LOCALFILES: the cp command to call
@@ -17,18 +19,17 @@ set -e
# Make 'cp' verbose by default, so it behaves a bit like the others.
verbose=-v
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=;;
+ o) output="${OPTARG}";;
+ u) source="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-source="${2}"
-
-shift 2 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
diff --git a/support/download/cvs b/support/download/cvs
index 50050ab1c9..69d5c71f28 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -5,28 +5,32 @@ set -e
# Download helper for cvs, to be called from the download wrapper script
#
-# Call it as:
-# .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
+# Options:
+# -q Be quiet
+# -o FILE Generate archive in FILE.
+# -u URI Checkout from repository at URI.
+# -c REV Use revision REV.
+# -N RAWNAME Use rawname (aka module name) RAWNAME.
+# -n NAME Use basename NAME.
#
# Environment:
# CVS : the cvs command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-Q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) rev="${OPTARG}";;
+ N) rawname="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-rev="${3}"
-rawname="${4}"
-basename="${5}"
-
-shift 5 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -42,14 +46,14 @@ else
select="-r"
fi
-# The absence of an initial : on ${repo} means access method undefined
-if [[ ! "${repo}" =~ ^: ]]; then
+# The absence of an initial : on ${uri} means access method undefined
+if [[ ! "${uri}" =~ ^: ]]; then
# defaults to anonymous pserver
- repo=":pserver:anonymous@${repo}"
+ uri=":pserver:anonymous@${uri}"
fi
export TZ=UTC
-_cvs ${verbose} -z3 -d"'${repo}'" \
+_cvs ${verbose} -z3 -d"'${uri}'" \
co "${@}" -d "'${basename}'" ${select} "'${rev}'" -P "'${rawname}'"
tar czf "${output}" "${basename}"
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index f944b71db5..a29411e0ae 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -19,6 +19,8 @@
# We want to catch any unexpected failure, and exit immediately.
set -e
+export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"
+
main() {
local OPT OPTARG
local backend output hfile recurse quiet
@@ -83,7 +85,10 @@ main() {
# If the backend fails, we can just remove the temporary directory to
# remove all the cruft it may have left behind. Then we just exit in
# error too.
- if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
+ if ! "${OLDPWD}/support/download/${backend}" \
+ ${quiet} ${recurse} \
+ -o "${tmpf}" "${@}"
+ then
rm -rf "${tmpd}"
exit 1
fi
diff --git a/support/download/git b/support/download/git
index fc2957d2ca..a49e448e60 100755
--- a/support/download/git
+++ b/support/download/git
@@ -5,32 +5,33 @@ set -e
# Download helper for git, to be called from the download wrapper script
#
-# Call it as:
-# .../git [-q] [-r] OUT_FILE REPO_URL CSET BASENAME
-#
-# -q Be quiet.
-# -r Clone and archive sub-modules.
+# Options:
+# -q Be quiet.
+# -r Clone and archive sub-modules.
+# -o FILE Generate archive in FILE.
+# -u URI Clone from repository at URI.
+# -c CSET Use changeset CSET.
+# -n NAME Use basename NAME.
#
# Environment:
# GIT : the git command to call
verbose=
recurse=0
-while getopts :qr OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q; exec >/dev/null;;
r) recurse=1;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) cset="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-
-output="${1}"
-repo="${2}"
-cset="${3}"
-basename="${4}"
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -46,9 +47,9 @@ _git() {
# Messages for the type of clone used are provided to ease debugging in case of
# problems
git_done=0
-if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
+if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
printf "Doing shallow clone\n"
- if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
+ if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
git_done=1
else
printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -56,7 +57,7 @@ if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
fi
if [ ${git_done} -eq 0 ]; then
printf "Doing full clone\n"
- _git clone ${verbose} "${@}" "'${repo}'" "'${basename}'"
+ _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
fi
pushd "${basename}" >/dev/null
diff --git a/support/download/hg b/support/download/hg
index 3af01690b3..efb515fca5 100755
--- a/support/download/hg
+++ b/support/download/hg
@@ -5,27 +5,30 @@ set -e
# Download helper for hg, to be called from the download wrapper script
#
-# Call it as:
-# .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
+# Options:
+# -q Be quiet.
+# -o FILE Generate archive in FILE.
+# -u URI Clone from repository at URI.
+# -c CSET Use changeset (or revision) CSET.
+# -n NAME Use basename NAME.
#
# Environment:
# HG : the hg command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) cset="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-cset="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -33,7 +36,7 @@ _hg() {
eval ${HG} "${@}"
}
-_hg clone ${verbose} "${@}" --noupdate "'${repo}'" "'${basename}'"
+_hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'"
_hg archive ${verbose} --repository "'${basename}'" --type tgz \
--prefix "'${basename}'" --rev "'${cset}'" \
diff --git a/support/download/scp b/support/download/scp
index 825fd41c64..8ecf2f4b22 100755
--- a/support/download/scp
+++ b/support/download/scp
@@ -5,25 +5,26 @@ set -e
# Download helper for scp, to be called from the download wrapper script
#
-# Call it as:
-# .../scp [-q] OUT_FILE SRC_URL
+# Options:
+# -q Be quiet.
+# -o FILE Copy to local file FILE.
+# -u FILE Copy from remote file FILE.
#
# Environment:
# SCP : the scp command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-url="${2}"
-
-shift 2 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -31,4 +32,4 @@ _scp() {
eval ${SCP} "${@}"
}
-_scp ${verbose} "${@}" "'${url}'" "'${output}'"
+_scp ${verbose} "${@}" "'${uri}'" "'${output}'"
diff --git a/support/download/svn b/support/download/svn
index 77abf3d02d..542b25c0a2 100755
--- a/support/download/svn
+++ b/support/download/svn
@@ -5,27 +5,30 @@ set -e
# Download helper for svn, to be called from the download wrapper script
#
-# Call it as:
-# .../svn [-q] OUT_FILE REPO_URL REV BASNAME
+# Options:
+# -q Be quiet.
+# -o FILE Generate archive in FILE.
+# -u URI Checkout from repository at URI.
+# -c REV Use revision REV.
+# -n NAME Use basename NAME.
#
# Environment:
# SVN : the svn command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) rev="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-rev="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -33,6 +36,6 @@ _svn() {
eval ${SVN} "${@}"
}
-_svn export ${verbose} "${@}" "'${repo}@${rev}'" "'${basename}'"
+_svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
tar czf "${output}" "${basename}"
diff --git a/support/download/wget b/support/download/wget
index 768de904c3..fece6663ca 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -5,25 +5,26 @@ set -e
# Download helper for wget, to be called from the download wrapper script
#
-# Call it as:
-# .../wget [-q] OUT_FILE URL
+# Options:
+# -q Be quiet.
+# -o FILE Save into file FILE.
+# -u URL Download file at URL.
#
# Environment:
# WGET : the wget command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) url="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-url="${2}"
-
-shift 2 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 03/13] download: put most of the infra in dl-wrapper
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
2017-07-04 16:21 ` [Buildroot] [PATCH 01/13] pkg-{download, generic}: remove source-check Maxime Hadjinlian
2017-07-04 16:22 ` [Buildroot] [PATCH 02/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-07-05 0:32 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 04/13] pkg-generic: make PKG_DL_DIR equal to DL_DIR Maxime Hadjinlian
` (10 subsequent siblings)
13 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
The goal here is to simplify the infrastructure by putting most of the
code in the dl-wrapper as it's easier to implement and to read.
Most of the function where common already, this patch finalize it by
making the pkg-download.mk pass all the parameters needed to the
dl-wrapper which in turns will pass everything to every backend.
The backend will then cherry-pick what it needs from theses arguments
and act accordingly.
It eases the transition to the addition of a sub directory per package
in the DL_DIR, and later on, a git cache.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-download.mk | 166 ++++++++------------------------------------
support/download/bzr | 2 +-
support/download/cp | 2 +-
support/download/cvs | 2 +-
support/download/dl-wrapper | 65 ++++++++++-------
support/download/wget | 7 +-
6 files changed, 77 insertions(+), 167 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index ce069b9926..be61fb130a 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -42,6 +42,8 @@ DL_DIR := $(shell mkdir -p $(DL_DIR) && cd $(DL_DIR) >/dev/null && pwd)
#
# geturischeme: http
geturischeme = $(firstword $(subst ://, ,$(call qstrip,$(1))))
+# getschemeplusuri: git+http://example.com
+getschemeplusuri = $(call geturischeme,$(1))+$(1)
# stripurischeme: www.example.com/dir/file
stripurischeme = $(lastword $(subst ://, ,$(call qstrip,$(1))))
# domain: www.example.com
@@ -61,152 +63,42 @@ github = https://github.com/$(1)/$(2)/archive/$(3)
export BR_NO_CHECK_HASH_FOR =
################################################################################
-# The DOWNLOAD_* helpers are in charge of getting a working copy
-# of the source repository for their corresponding SCM,
-# checking out the requested version / commit / tag, and create an
-# archive out of it. DOWNLOAD_SCP uses scp to obtain a remote file with
-# ssh authentication. DOWNLOAD_WGET is the normal wget-based download
-# mechanism.
+# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download
+# source from:
+# 1) BR2_PRIMARY_SITE if enabled
+# 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
+# 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
+#
+# Argument 1 is the source location
#
################################################################################
-define DOWNLOAD_GIT
- $(EXTRA_ENV) $(DL_WRAPPER) -b git \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(if $($(PKG)_GIT_SUBMODULES),-r) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_BZR
- $(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
+ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
+DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
+endif
-define DOWNLOAD_CVS
- $(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
- -c $($(PKG)_DL_VERSION) \
- -N $($(PKG)_RAWNAME) \
- -n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
+ifeq ($(BR2_PRIMARY_SITE_ONLY),)
+DOWNLOAD_URIS += \
+ -u $($(PKG)_SITE_METHOD)+$(dir $(1))
+ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
+DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
+endif
+endif
-define DOWNLOAD_SVN
- $(EXTRA_ENV) $(DL_WRAPPER) -b svn \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
+define DOWNLOAD
+ $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
+ $(EXTRA_ENV) $(DL_WRAPPER) \
-c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
-
-# SCP URIs should be of the form scp://[user@]host:filepath
-# Note that filepath is relative to the user's home directory, so you may want
-# to prepend the path with a slash: scp://[user@]host:/absolutepath
-define DOWNLOAD_SCP
- $(EXTRA_ENV) $(DL_WRAPPER) -b scp \
- -o $(DL_DIR)/$(2) \
+ -f $(notdir $(1)) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u '$(call stripurischeme,$(call qstrip,$(1)))' \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_HG
- $(EXTRA_ENV) $(DL_WRAPPER) -b hg \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
-n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_WGET
- $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
- -o $(DL_DIR)/$(2) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u '$(call qstrip,$(1))' \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_LOCALFILES
- $(EXTRA_ENV) $(DL_WRAPPER) -b cp \
- -o $(DL_DIR)/$(2) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ -N $($(PKG)_RAWNAME) \
+ -o $(DL_DIR)/$(notdir $(1)) \
+ $(if $($(PKG)_GIT_SUBMODULES),-r) \
+ $(DOWNLOAD_URIS) \
$(QUIET) \
-- \
- -u $(call stripurischeme,$(call qstrip,$(1))) \
$($(PKG)_DL_OPTS)
endef
-
-################################################################################
-# DOWNLOAD -- Download helper. Will try to download source from:
-# 1) BR2_PRIMARY_SITE if enabled
-# 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
-# 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
-#
-# Argument 1 is the source location
-#
-# E.G. use like this:
-# $(call DOWNLOAD,$(FOO_SITE))
-#
-# For PRIMARY and BACKUP site, any ? in the URL is replaced by %3F. A ? in
-# the URL is used to separate query arguments, but the PRIMARY and BACKUP
-# sites serve just plain files.
-################################################################################
-
-define DOWNLOAD
- $(call DOWNLOAD_INNER,$(1),$(notdir $(1)),DOWNLOAD)
-endef
-
-define DOWNLOAD_INNER
- $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
- if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
- case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \
- file) $(call $(3)_LOCALFILES,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
- scp) $(call $(3)_SCP,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
- *) $(call $(3)_WGET,$(BR2_PRIMARY_SITE)/$(subst ?,%3F,$(2)),$(2)) && exit ;; \
- esac ; \
- fi ; \
- if test "$(BR2_PRIMARY_SITE_ONLY)" = "y" ; then \
- exit 1 ; \
- fi ; \
- if test -n "$(1)" ; then \
- case "$($(PKG)_SITE_METHOD)" in \
- git) $($(3)_GIT) && exit ;; \
- svn) $($(3)_SVN) && exit ;; \
- cvs) $($(3)_CVS) && exit ;; \
- bzr) $($(3)_BZR) && exit ;; \
- file) $($(3)_LOCALFILES) && exit ;; \
- scp) $($(3)_SCP) && exit ;; \
- hg) $($(3)_HG) && exit ;; \
- *) $(call $(3)_WGET,$(1),$(2)) && exit ;; \
- esac ; \
- fi ; \
- if test -n "$(call qstrip,$(BR2_BACKUP_SITE))" ; then \
- $(call $(3)_WGET,$(BR2_BACKUP_SITE)/$(subst ?,%3F,$(2)),$(2)) && exit ; \
- fi ; \
- exit 1
-endef
diff --git a/support/download/bzr b/support/download/bzr
index a70cb19cf1..5289a421cd 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -8,7 +8,7 @@ set -e
# Options:
# -q Be quiet
# -o FILE Generate archive in FILE.
-# -u URI Clone from repository URI.
+# -u URI Clone from repository at URI.
# -c CSET Use changeset (or revision) CSET.
# -n NAME Use basename NAME.
#
diff --git a/support/download/cp b/support/download/cp
index 52fe2de83d..9c64b7b70b 100755
--- a/support/download/cp
+++ b/support/download/cp
@@ -23,7 +23,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=;;
o) output="${OPTARG}";;
- u) source="${OPTARG}";;
+ u) source="${OPTARG#*//}";;
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
diff --git a/support/download/cvs b/support/download/cvs
index 69d5c71f28..9caffb4b82 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -21,7 +21,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-Q;;
o) output="${OPTARG}";;
- u) uri="${OPTARG}";;
+ u) uri="${OPTARG#*//}";;
c) rev="${OPTARG}";;
N) rawname="${OPTARG}";;
n) basename="${OPTARG}";;
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index a29411e0ae..50c14a2e16 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -19,31 +19,34 @@
# We want to catch any unexpected failure, and exit immediately.
set -e
-export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"
+export BR_BACKEND_DL_GETOPTS=":hc:o:H:n:N:ru:qf:"
main() {
local OPT OPTARG
local backend output hfile recurse quiet
+ local -a uris
# Parse our options; anything after '--' is for the backend
- while getopts :hb:o:H:rq OPT; do
+ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
h) help; exit 0;;
- b) backend="${OPTARG}";;
+ c) cset="${OPTARG}";;
o) output="${OPTARG}";;
+ n) raw_base_name="${OPTARG}";;
+ N) base_name="${OPTARG}";;
H) hfile="${OPTARG}";;
r) recurse="-r";;
+ f) filename="${OPTARG}";;
+ u) uris+=( "${OPTARG}" );;
q) quiet="-q";;
:) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
\?) error "unknown option '%s'\n" "${OPTARG}";;
esac
done
+
# Forget our options, and keep only those for the backend
shift $((OPTIND-1))
- if [ -z "${backend}" ]; then
- error "no backend specified, use -b\n"
- fi
if [ -z "${output}" ]; then
error "no output specified, use -o\n"
fi
@@ -82,16 +85,29 @@ main() {
# Doing the 'cd' here rather than in all backends is easier.
cd "${tmpd}"
- # If the backend fails, we can just remove the temporary directory to
- # remove all the cruft it may have left behind. Then we just exit in
- # error too.
- if ! "${OLDPWD}/support/download/${backend}" \
- ${quiet} ${recurse} \
- -o "${tmpf}" "${@}"
- then
- rm -rf "${tmpd}"
- exit 1
- fi
+ # Look through all the uris that we were given to downoad the package
+ # source
+ for uri in "${uris[@]}"; do
+ backend=${uri%+*}
+ case "${backend}" in
+ git|svn|cvs|bzr|file|scp|hg) ;;
+ *) backend="wget" ;;
+ esac
+ uri=${uri#*+}
+
+ # If the backend fails, we can just remove the temporary directory to
+ # remove all the cruft it may have left behind. Then we just exit in
+ # error too.
+ if "${OLDPWD}/support/download/${backend}" \
+ -c "${cset}" -n "${raw_base_name}" \
+ -N "${raw_name}" -f "${filename}" -u "${uri}" -o "${tmpf}" \
+ ${quiet} ${recurse} "${@}"
+ then
+ break
+ else
+ rm -rf "${tmpd:?}/*"
+ fi
+ done
# cd back to free the temp-dir, so we can remove it later
cd "${OLDPWD}"
@@ -164,16 +180,13 @@ DESCRIPTION
-h This help text.
- -b BACKEND
- Wrap the specified BACKEND. Known backends are:
- bzr Bazaar
- cp Local files
- cvs Concurrent Versions System
- git Git
- hg Mercurial
- scp Secure copy
- svn Subversion
- wget HTTP download
+ -u URIs
+ The URI to get the file from, the URI must respect the format given in
+ the example.
+ You may give as many '-u URI' as you want, the script will stop at the
+ frist successful download.
+
+ Example: backend+URI; git+http://example.com or http+http://example.com
-o FILE
Store the downloaded archive in FILE.
diff --git a/support/download/wget b/support/download/wget
index fece6663ca..3e6a6b446c 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -8,6 +8,7 @@ set -e
# Options:
# -q Be quiet.
# -o FILE Save into file FILE.
+# -f FILENAME The filename of the tarball to get at URL
# -u URL Download file at URL.
#
# Environment:
@@ -18,6 +19,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
o) output="${OPTARG}";;
+ f) filename="${OPTARG}";;
u) url="${OPTARG}";;
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
@@ -32,4 +34,7 @@ _wget() {
eval ${WGET} "${@}"
}
-_wget ${verbose} "${@}" -O "'${output}'" "'${url}'"
+# Replace every '?' with '%3F' in the URI
+url=${url//\?/%3F}
+
+_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 04/13] pkg-generic: make PKG_DL_DIR equal to DL_DIR
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (2 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 03/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-07-05 8:30 ` Arnout Vandecappelle
2017-07-22 21:37 ` Thomas Petazzoni
2017-07-04 16:22 ` [Buildroot] [PATCH 05/13] packages: use new $($PKG)_DL_DIR) variable Maxime Hadjinlian
` (9 subsequent siblings)
13 siblings, 2 replies; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
This variable is currently not used by anyone.
The value is changed to match the path to DL_DIR.
The next patch will introduce the use of PKG_DL_DIR for packages that
use specific EXTRACT_CMDS.
And then we will be able to change the value of PKG_DL_DIR to create a
new directory structure in DL_DIR with a subdirectory for each package.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-generic.mk | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 79bea906e2..c1cd41fda5 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -434,7 +434,7 @@ endif
$(2)_BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
$(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_BASE_NAME)
+$(2)_DL_DIR = $$(DL_DIR)
$(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME)
ifndef $(2)_SUBDIR
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 05/13] packages: use new $($PKG)_DL_DIR) variable
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (3 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 04/13] pkg-generic: make PKG_DL_DIR equal to DL_DIR Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-07-22 21:39 ` Thomas Petazzoni
2017-10-23 20:00 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 06/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Maxime Hadjinlian
` (8 subsequent siblings)
13 siblings, 2 replies; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the
transition into a new directory structure for DL_DIR.
This commit has been generated with the following scripts:
for i in package/*/*.mk; do
if ! grep -q "\$(DL_DIR)" ${i}; then
continue
fi
pkg_name="$(basename $(dirname ${i}))"
raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
pkg_dl_dir="${raw_pkg_name}_DL_DIR"
sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}
done
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/amd-catalyst/amd-catalyst.mk | 2 +-
package/android-tools/android-tools.mk | 2 +-
package/angularjs/angularjs.mk | 2 +-
package/bootstrap/bootstrap.mk | 2 +-
package/cache-calibrator/cache-calibrator.mk | 2 +-
package/cracklib/cracklib.mk | 2 +-
package/cryptopp/cryptopp.mk | 2 +-
package/devmem2/devmem2.mk | 2 +-
package/dhrystone/dhrystone.mk | 2 +-
package/doom-wad/doom-wad.mk | 2 +-
package/espeak/espeak.mk | 2 +-
package/fan-ctrl/fan-ctrl.mk | 2 +-
package/gcc/gcc.mk | 2 +-
package/irrlicht/irrlicht.mk | 2 +-
package/jquery-mobile/jquery-mobile.mk | 2 +-
package/jquery-sparkline/jquery-sparkline.mk | 2 +-
package/jquery-ui-themes/jquery-ui-themes.mk | 2 +-
package/jquery-ui/jquery-ui.mk | 2 +-
package/jquery-validation/jquery-validation.mk | 2 +-
package/jquery/jquery.mk | 2 +-
package/kodi/kodi.mk | 6 +++---
package/libfreeimage/libfreeimage.mk | 2 +-
package/libjson/libjson.mk | 2 +-
package/libsoil/libsoil.mk | 2 +-
package/lsof/lsof.mk | 2 +-
| 2 +-
package/nmon/nmon.mk | 2 +-
package/nvidia-driver/nvidia-driver.mk | 2 +-
package/openobex/openobex.mk | 2 +-
package/opentyrian-data/opentyrian-data.mk | 2 +-
package/perl/perl.mk | 2 +-
package/python-keyring/python-keyring.mk | 2 +-
package/python-simplegeneric/python-simplegeneric.mk | 2 +-
package/rapidxml/rapidxml.mk | 2 +-
package/rpi-wifi-firmware/rpi-wifi-firmware.mk | 2 +-
package/sam-ba/sam-ba.mk | 2 +-
package/spidev_test/spidev_test.mk | 2 +-
package/tar/tar.mk | 2 +-
package/tesseract-ocr/tesseract-ocr.mk | 2 +-
package/ti-cgt-pru/ti-cgt-pru.mk | 4 ++--
package/ti-gfx/ti-gfx.mk | 4 ++--
package/ts4900-fpga/ts4900-fpga.mk | 2 +-
package/unscd/unscd.mk | 2 +-
package/urg/urg.mk | 2 +-
package/waf/waf.mk | 2 +-
package/whetstone/whetstone.mk | 2 +-
package/wilc1000-firmware/wilc1000-firmware.mk | 2 +-
package/zynq-boot-bin/zynq-boot-bin.mk | 2 +-
48 files changed, 52 insertions(+), 52 deletions(-)
diff --git a/package/amd-catalyst/amd-catalyst.mk b/package/amd-catalyst/amd-catalyst.mk
index b9396e11fa..d864095c31 100644
--- a/package/amd-catalyst/amd-catalyst.mk
+++ b/package/amd-catalyst/amd-catalyst.mk
@@ -17,7 +17,7 @@ AMD_CATALYST_ARCH_DIR = $(@D)/arch/x86$(AMD_CATALYST_SUFFIX)
AMD_CATALYST_LIB_SUFFIX = $(if $(BR2_x86_64),64)
define AMD_CATALYST_EXTRACT_CMDS
- unzip -q $(DL_DIR)/$(AMD_CATALYST_SOURCE) -d $(@D)
+ unzip -q $(AMD_CATALYST_DL_DIR)/$(AMD_CATALYST_SOURCE) -d $(@D)
$(SHELL) $(@D)/AMD-Catalyst-$(AMD_CATALYST_VERSION)-Linux-installer-$(AMD_CATALYST_VERBOSE_VER)-x86.x86_64.run --extract $(@D)
endef
diff --git a/package/android-tools/android-tools.mk b/package/android-tools/android-tools.mk
index 45103923eb..3def537c91 100644
--- a/package/android-tools/android-tools.mk
+++ b/package/android-tools/android-tools.mk
@@ -15,7 +15,7 @@ ANDROID_TOOLS_LICENSE_FILES = debian/copyright
# Extract the Debian tarball inside the sources
define ANDROID_TOOLS_DEBIAN_EXTRACT
$(call suitable-extractor,$(notdir $(ANDROID_TOOLS_EXTRA_DOWNLOADS))) \
- $(DL_DIR)/$(notdir $(ANDROID_TOOLS_EXTRA_DOWNLOADS)) | \
+ $(ANDROID_TOOLS_DL_DIR)/$(notdir $(ANDROID_TOOLS_EXTRA_DOWNLOADS)) | \
$(TAR) -C $(@D) $(TAR_OPTIONS) -
endef
diff --git a/package/angularjs/angularjs.mk b/package/angularjs/angularjs.mk
index 08a4b60845..8d6161125b 100644
--- a/package/angularjs/angularjs.mk
+++ b/package/angularjs/angularjs.mk
@@ -12,7 +12,7 @@ ANGULARJS_LICENSE = MIT
ANGULARJS_LICENSE_FILES = angular.js
define ANGULARJS_EXTRACT_CMDS
- unzip $(DL_DIR)/$(ANGULARJS_SOURCE) -d $(@D)
+ unzip $(ANGULARJS_DL_DIR)/$(ANGULARJS_SOURCE) -d $(@D)
mv $(@D)/angular-$(ANGULARJS_VERSION)/* $(@D)
rmdir $(@D)/angular-$(ANGULARJS_VERSION)
endef
diff --git a/package/bootstrap/bootstrap.mk b/package/bootstrap/bootstrap.mk
index 33161f53ae..d9aeeabe62 100644
--- a/package/bootstrap/bootstrap.mk
+++ b/package/bootstrap/bootstrap.mk
@@ -10,7 +10,7 @@ BOOTSTRAP_SOURCE = bootstrap-$(BOOTSTRAP_VERSION)-dist.zip
BOOTSTRAP_LICENSE = MIT
define BOOTSTRAP_EXTRACT_CMDS
- $(UNZIP) $(DL_DIR)/$(BOOTSTRAP_SOURCE) -d $(@D)
+ $(UNZIP) $(BOOTSTRAP_DL_DIR)/$(BOOTSTRAP_SOURCE) -d $(@D)
mv $(@D)/bootstrap-$(BOOTSTRAP_VERSION)-dist/* $(@D)/
rmdir $(@D)/bootstrap-$(BOOTSTRAP_VERSION)-dist
endef
diff --git a/package/cache-calibrator/cache-calibrator.mk b/package/cache-calibrator/cache-calibrator.mk
index 6f12d119ef..203732e4f7 100644
--- a/package/cache-calibrator/cache-calibrator.mk
+++ b/package/cache-calibrator/cache-calibrator.mk
@@ -10,7 +10,7 @@ CACHE_CALIBRATOR_LICENSE = Cache calibrator license
CACHE_CALIBRATOR_LICENSE_FILES = calibrator.c.license
define CACHE_CALIBRATOR_EXTRACT_CMDS
- cp $(DL_DIR)/$(CACHE_CALIBRATOR_SOURCE) $(@D)
+ cp $(CACHE_CALIBRATOR_DL_DIR)/$(CACHE_CALIBRATOR_SOURCE) $(@D)
endef
define CACHE_CALIBRATOR_EXTRACT_LICENSE
diff --git a/package/cracklib/cracklib.mk b/package/cracklib/cracklib.mk
index 2fa3d3b4ea..aeee60d1e7 100644
--- a/package/cracklib/cracklib.mk
+++ b/package/cracklib/cracklib.mk
@@ -29,7 +29,7 @@ HOST_CRACKLIB_CONF_OPTS += --without-python --without-zlib
ifeq ($(BR2_PACKAGE_CRACKLIB_FULL_DICT),y)
CRACKLIB_EXTRA_DOWNLOADS = cracklib-words-$(CRACKLIB_VERSION).gz
-CRACKLIB_DICT_SOURCE = $(DL_DIR)/cracklib-words-$(CRACKLIB_VERSION).gz
+CRACKLIB_DICT_SOURCE = $(CRACKLIB_DL_DIR)/cracklib-words-$(CRACKLIB_VERSION).gz
else
CRACKLIB_DICT_SOURCE = $(@D)/dicts/cracklib-small
endif
diff --git a/package/cryptopp/cryptopp.mk b/package/cryptopp/cryptopp.mk
index 5c5def5bc5..c45b1a9c52 100644
--- a/package/cryptopp/cryptopp.mk
+++ b/package/cryptopp/cryptopp.mk
@@ -12,7 +12,7 @@ CRYPTOPP_LICENSE_FILES = License.txt
CRYPTOPP_INSTALL_STAGING = YES
define HOST_CRYPTOPP_EXTRACT_CMDS
- $(UNZIP) $(DL_DIR)/$(CRYPTOPP_SOURCE) -d $(@D)
+ $(UNZIP) $(CRYPTOPP_DL_DIR)/$(CRYPTOPP_SOURCE) -d $(@D)
endef
HOST_CRYPTOPP_MAKE_OPTS = \
diff --git a/package/devmem2/devmem2.mk b/package/devmem2/devmem2.mk
index eb92db6098..044e6fa596 100644
--- a/package/devmem2/devmem2.mk
+++ b/package/devmem2/devmem2.mk
@@ -11,7 +11,7 @@ DEVMEM2_LICENSE = GPL-2.0+
DEVMEM2_LICENSE_FILES = devmem2.c.license
define DEVMEM2_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(DEVMEM2_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define DEVMEM2_EXTRACT_LICENSE
diff --git a/package/dhrystone/dhrystone.mk b/package/dhrystone/dhrystone.mk
index c0bca9895d..d54c8d94e8 100644
--- a/package/dhrystone/dhrystone.mk
+++ b/package/dhrystone/dhrystone.mk
@@ -9,7 +9,7 @@ DHRYSTONE_SOURCE = dhry-c
DHRYSTONE_SITE = http://www.netlib.org/benchmark
define DHRYSTONE_EXTRACT_CMDS
- (cd $(@D) && $(SHELL) $(DL_DIR)/$($(PKG)_SOURCE))
+ (cd $(@D) && $(SHELL) $(DHRYSTONE_DL_DIR)/$($(PKG)_SOURCE))
$(Q)cp package/dhrystone/Makefile $(@D)/
endef
diff --git a/package/doom-wad/doom-wad.mk b/package/doom-wad/doom-wad.mk
index d3ac731bd5..f348fc2207 100644
--- a/package/doom-wad/doom-wad.mk
+++ b/package/doom-wad/doom-wad.mk
@@ -11,7 +11,7 @@ DOOM_WAD_SOURCE = doom$(subst .,,$(DOOM_WAD_VERSION))s.zip
DOOM_WAD_SITE = http://www.jbserver.com/downloads/games/doom/misc/shareware
define DOOM_WAD_EXTRACT_CMDS
- $(UNZIP) -p $(DL_DIR)/$($(PKG)_SOURCE) 'DOOMS_19.[12]' > \
+ $(UNZIP) -p $(DOOM_WAD_DL_DIR)/$($(PKG)_SOURCE) 'DOOMS_19.[12]' > \
$(@D)/doom-$(DOOM_WAD_VERSION).zip
$(UNZIP) -d $(@D) $(@D)/doom-$(DOOM_WAD_VERSION).zip DOOM1.WAD
endef
diff --git a/package/espeak/espeak.mk b/package/espeak/espeak.mk
index 226f17fc05..aa0b5b0716 100644
--- a/package/espeak/espeak.mk
+++ b/package/espeak/espeak.mk
@@ -21,7 +21,7 @@ ESPEAK_DEPENDENCIES = pulseaudio
endif
define ESPEAK_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(ESPEAK_SOURCE)
+ $(UNZIP) -d $(@D) $(ESPEAK_DL_DIR)/$(ESPEAK_SOURCE)
mv $(@D)/espeak-$(ESPEAK_VERSION)-source/* $(@D)
$(RM) -r $(@D)/espeak-$(ESPEAK_VERSION)-source
endef
diff --git a/package/fan-ctrl/fan-ctrl.mk b/package/fan-ctrl/fan-ctrl.mk
index 53533a8a17..81a52f1e9f 100644
--- a/package/fan-ctrl/fan-ctrl.mk
+++ b/package/fan-ctrl/fan-ctrl.mk
@@ -12,7 +12,7 @@ FAN_CTRL_LICENSE = GPL-2.0+
FAN_CTRL_LICENSE_FILES = fan-ctrl.c
define FAN_CTRL_EXTRACT_CMDS
- cp $(DL_DIR)/$(FAN_CTRL_SOURCE) $(@D)/fan-ctrl.c
+ cp $(FAN_CTRL_DL_DIR)/$(FAN_CTRL_SOURCE) $(@D)/fan-ctrl.c
endef
define FAN_CTRL_BUILD_CMDS
diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
index b52f9456b3..64fcfc70e7 100644
--- a/package/gcc/gcc.mk
+++ b/package/gcc/gcc.mk
@@ -289,7 +289,7 @@ HOST_GCC_COMMON_MAKE_OPTS = \
gcc_cv_libc_provides_ssp=$(if $(BR2_TOOLCHAIN_HAS_SSP),yes,no)
ifeq ($(BR2_CCACHE),y)
-HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE)
+HOST_GCC_COMMON_CCACHE_HASH_FILES += $(GCC_DL_DIR)/$(GCC_SOURCE)
# Cfr. PATCH_BASE_DIRS in .stamp_patched, but we catch both versioned
# and unversioned patches unconditionally. Moreover, to facilitate the
diff --git a/package/irrlicht/irrlicht.mk b/package/irrlicht/irrlicht.mk
index ccd1045ca0..a1e190d98d 100644
--- a/package/irrlicht/irrlicht.mk
+++ b/package/irrlicht/irrlicht.mk
@@ -25,7 +25,7 @@ IRRLICHT_SUBDIR = source/Irrlicht
IRRLICHT_DEPENDENCIES = libgl xlib_libXxf86vm
define IRRLICHT_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(IRRLICHT_SOURCE)
+ $(UNZIP) -d $(@D) $(IRRLICHT_DL_DIR)/$(IRRLICHT_SOURCE)
mv $(@D)/irrlicht-$(IRRLICHT_VERSION)/* $(@D)
$(RM) -r $(@D)/irrlicht-$(IRRLICHT_VERSION)
endef
diff --git a/package/jquery-mobile/jquery-mobile.mk b/package/jquery-mobile/jquery-mobile.mk
index 1067952904..1171e60d0a 100644
--- a/package/jquery-mobile/jquery-mobile.mk
+++ b/package/jquery-mobile/jquery-mobile.mk
@@ -10,7 +10,7 @@ JQUERY_MOBILE_SOURCE = jquery.mobile-$(JQUERY_MOBILE_VERSION).zip
JQUERY_MOBILE_LICENSE = MIT
define JQUERY_MOBILE_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_MOBILE_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_MOBILE_DL_DIR)/$(JQUERY_MOBILE_SOURCE)
endef
JQUERY_MOBILE_INSTALLED_FILES = \
diff --git a/package/jquery-sparkline/jquery-sparkline.mk b/package/jquery-sparkline/jquery-sparkline.mk
index 165c2fa99c..683d84f4aa 100644
--- a/package/jquery-sparkline/jquery-sparkline.mk
+++ b/package/jquery-sparkline/jquery-sparkline.mk
@@ -10,7 +10,7 @@ JQUERY_SPARKLINE_SOURCE = jquery.sparkline.min.js
JQUERY_SPARKLINE_LICENSE = BSD-3-Clause
define JQUERY_SPARKLINE_EXTRACT_CMDS
- cp $(DL_DIR)/$(JQUERY_SPARKLINE_SOURCE) $(@D)
+ cp $(JQUERY_SPARKLINE_DL_DIR)/$(JQUERY_SPARKLINE_SOURCE) $(@D)
endef
define JQUERY_SPARKLINE_INSTALL_TARGET_CMDS
diff --git a/package/jquery-ui-themes/jquery-ui-themes.mk b/package/jquery-ui-themes/jquery-ui-themes.mk
index cd3dab6d87..5108389a77 100644
--- a/package/jquery-ui-themes/jquery-ui-themes.mk
+++ b/package/jquery-ui-themes/jquery-ui-themes.mk
@@ -12,7 +12,7 @@ JQUERY_UI_THEMES_LICENSE_FILES = MIT-LICENSE.txt
JQUERY_UI_THEMES_DEPENDENCIES = jquery-ui
define JQUERY_UI_THEMES_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_UI_THEMES_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_UI_THEMES_DL_DIR)/$(JQUERY_UI_THEMES_SOURCE)
mv $(@D)/jquery-ui-themes-$(JQUERY_UI_THEMES_VERSION)/* $(@D)
$(RM) -r $(@D)/jquery-ui-themes-$(JQUERY_UI_THEMES_VERSION)
endef
diff --git a/package/jquery-ui/jquery-ui.mk b/package/jquery-ui/jquery-ui.mk
index d829d6241b..9284e683f1 100644
--- a/package/jquery-ui/jquery-ui.mk
+++ b/package/jquery-ui/jquery-ui.mk
@@ -14,7 +14,7 @@ JQUERY_UI_LICENSE = MIT
JQUERY_UI_LICENSE_FILES = MIT-LICENSE.txt
define JQUERY_UI_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_UI_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_UI_DL_DIR)/$(JQUERY_UI_SOURCE)
mv $(@D)/jquery-ui-$(JQUERY_UI_VERSION)/* $(@D)
$(RM) -r $(@D)/jquery-ui-$(JQUERY_UI_VERSION)
endef
diff --git a/package/jquery-validation/jquery-validation.mk b/package/jquery-validation/jquery-validation.mk
index 10d2a92afc..a8c2897983 100644
--- a/package/jquery-validation/jquery-validation.mk
+++ b/package/jquery-validation/jquery-validation.mk
@@ -11,7 +11,7 @@ JQUERY_VALIDATION_LICENSE = MIT
JQUERY_VALIDATION_LICENSE_FILES = README.md
define JQUERY_VALIDATION_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_VALIDATION_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_VALIDATION_DL_DIR)/$(JQUERY_VALIDATION_SOURCE)
endef
define JQUERY_VALIDATION_INSTALL_TARGET_CMDS
diff --git a/package/jquery/jquery.mk b/package/jquery/jquery.mk
index 363ee09633..f75eded8b3 100644
--- a/package/jquery/jquery.mk
+++ b/package/jquery/jquery.mk
@@ -10,7 +10,7 @@ JQUERY_SOURCE = jquery-$(JQUERY_VERSION).min.js
JQUERY_LICENSE = MIT
define JQUERY_EXTRACT_CMDS
- cp $(DL_DIR)/$(JQUERY_SOURCE) $(@D)
+ cp $(JQUERY_DL_DIR)/$(JQUERY_SOURCE) $(@D)
endef
define JQUERY_INSTALL_TARGET_CMDS
diff --git a/package/kodi/kodi.mk b/package/kodi/kodi.mk
index 7a031b7485..3f05503c5b 100644
--- a/package/kodi/kodi.mk
+++ b/package/kodi/kodi.mk
@@ -66,9 +66,9 @@ KODI_CONF_OPTS += \
-DNATIVEPREFIX=$(HOST_DIR)/usr \
-DDEPENDS_PATH=$(@D) \
-DWITH_TEXTUREPACKER=$(HOST_DIR)/usr/bin/TexturePacker \
- -DLIBDVDCSS_URL=$(DL_DIR)/$(KODI_LIBDVDCSS_VERSION).tar.gz \
- -DLIBDVDNAV_URL=$(DL_DIR)/$(KODI_LIBDVDNAV_VERSION).tar.gz \
- -DLIBDVDREAD_URL=$(DL_DIR)/$(KODI_LIBDVDREAD_VERSION).tar.gz
+ -DLIBDVDCSS_URL=$(KODI_DL_DIR)/$(KODI_LIBDVDCSS_VERSION).tar.gz \
+ -DLIBDVDNAV_URL=$(KODI_DL_DIR)/$(KODI_LIBDVDNAV_VERSION).tar.gz \
+ -DLIBDVDREAD_URL=$(KODI_DL_DIR)/$(KODI_LIBDVDREAD_VERSION).tar.gz
ifeq ($(BR2_arm),y)
KODI_CONF_OPTS += -DWITH_ARCH=arm -DWITH_CPU=arm
diff --git a/package/libfreeimage/libfreeimage.mk b/package/libfreeimage/libfreeimage.mk
index 0ca23933a6..e0aa1f0ae8 100644
--- a/package/libfreeimage/libfreeimage.mk
+++ b/package/libfreeimage/libfreeimage.mk
@@ -12,7 +12,7 @@ LIBFREEIMAGE_LICENSE_FILES = license-gplv2.txt license-gplv3.txt license-fi.txt
LIBFREEIMAGE_INSTALL_STAGING = YES
define LIBFREEIMAGE_EXTRACT_CMDS
- $(UNZIP) $(DL_DIR)/$(LIBFREEIMAGE_SOURCE) -d $(@D)
+ $(UNZIP) $(LIBFREEIMAGE_DL_DIR)/$(LIBFREEIMAGE_SOURCE) -d $(@D)
mv $(@D)/FreeImage/* $(@D)
rmdir $(@D)/FreeImage
endef
diff --git a/package/libjson/libjson.mk b/package/libjson/libjson.mk
index 74224ba657..d04ddc40f3 100644
--- a/package/libjson/libjson.mk
+++ b/package/libjson/libjson.mk
@@ -23,7 +23,7 @@ endif
LIBJSON_MAKE_OPTS += BUILD_TYPE= CXXFLAGS="$(LIBJSON_CXXFLAGS)"
define LIBJSON_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(LIBJSON_SOURCE)
+ $(UNZIP) -d $(@D) $(LIBJSON_DL_DIR)/$(LIBJSON_SOURCE)
mv $(@D)/libjson/* $(@D)
$(RM) -r $(@D)/libjson
$(SED) '/ldconfig/d' $(@D)/makefile
diff --git a/package/libsoil/libsoil.mk b/package/libsoil/libsoil.mk
index efa67d9eee..2945edd8dd 100644
--- a/package/libsoil/libsoil.mk
+++ b/package/libsoil/libsoil.mk
@@ -14,7 +14,7 @@ LIBSOIL_LICENSE_FILES = src/stb_image_aug.c src/image_helper.c
LIBSOIL_MAKEFILE = ../projects/makefile/alternate_Makefile.txt
define LIBSOIL_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(LIBSOIL_SOURCE)
+ $(UNZIP) -d $(@D) $(LIBSOIL_DL_DIR)/$(LIBSOIL_SOURCE)
mv $(@D)/Simple\ OpenGL\ Image\ Library/* $(@D)
endef
diff --git a/package/lsof/lsof.mk b/package/lsof/lsof.mk
index e5cd4bce6e..0dc8e2de2f 100644
--- a/package/lsof/lsof.mk
+++ b/package/lsof/lsof.mk
@@ -32,7 +32,7 @@ endif
# The .tar.bz2 contains another .tar, which contains the source code.
define LSOF_EXTRACT_CMDS
- $(call suitable-extractor,$(LSOF_SOURCE)) $(DL_DIR)/$(LSOF_SOURCE) | \
+ $(call suitable-extractor,$(LSOF_SOURCE)) $(LSOF_DL_DIR)/$(LSOF_SOURCE) | \
$(TAR) -O $(TAR_OPTIONS) - lsof_$(LSOF_VERSION)/lsof_$(LSOF_VERSION)_src.tar | \
$(TAR) --strip-components=1 -C $(LSOF_DIR) $(TAR_OPTIONS) -
endef
--git a/package/musl-compat-headers/musl-compat-headers.mk b/package/musl-compat-headers/musl-compat-headers.mk
index 91f5074ef6..1cfa894879 100644
--- a/package/musl-compat-headers/musl-compat-headers.mk
+++ b/package/musl-compat-headers/musl-compat-headers.mk
@@ -20,7 +20,7 @@ MUSL_COMPAT_HEADERS_INSTALL_STAGING = YES
# Copying both headers so legal-info finds them (they are _LICENSE_FILES)
define MUSL_COMPAT_HEADERS_EXTRACT_CMDS
- $(INSTALL) -m 0644 -D $(DL_DIR)/$(notdir $(MUSL_COMPAT_HEADERS_QUEUE_H)) $(@D)/queue.h
+ $(INSTALL) -m 0644 -D $(MUSL_COMPAT_HEADERS_DL_DIR)/$(notdir $(MUSL_COMPAT_HEADERS_QUEUE_H)) $(@D)/queue.h
$(INSTALL) -m 0644 -D $(MUSL_COMPAT_HEADERS_PKGDIR)/cdefs.h $(@D)/cdefs.h
endef
diff --git a/package/nmon/nmon.mk b/package/nmon/nmon.mk
index 2c1ff0220d..5bb2c65f1b 100644
--- a/package/nmon/nmon.mk
+++ b/package/nmon/nmon.mk
@@ -13,7 +13,7 @@ NMON_DEPENDENCIES = ncurses
NMON_CFLAGS = $(TARGET_CFLAGS) -D JFS -D GETUSER -D LARGEMEM -D DEBIAN
define NMON_EXTRACT_CMDS
- cp $(DL_DIR)/$(NMON_SOURCE) $(@D)
+ cp $(NMON_DL_DIR)/$(NMON_SOURCE) $(@D)
endef
define NMON_BUILD_CMDS
diff --git a/package/nvidia-driver/nvidia-driver.mk b/package/nvidia-driver/nvidia-driver.mk
index d8dedd9e3b..cda87363d2 100644
--- a/package/nvidia-driver/nvidia-driver.mk
+++ b/package/nvidia-driver/nvidia-driver.mk
@@ -143,7 +143,7 @@ endif # BR2_PACKAGE_NVIDIA_DRIVER_MODULE == y
# virtually everywhere, and it is fine enough to provide useful options.
# Except it can't extract into an existing (even empty) directory.
define NVIDIA_DRIVER_EXTRACT_CMDS
- $(SHELL) $(DL_DIR)/$(NVIDIA_DRIVER_SOURCE) --extract-only --target \
+ $(SHELL) $(NVIDIA_DRIVER_DL_DIR)/$(NVIDIA_DRIVER_SOURCE) --extract-only --target \
$(@D)/tmp-extract
chmod u+w -R $(@D)
mv $(@D)/tmp-extract/* $(@D)/tmp-extract/.manifest $(@D)
diff --git a/package/openobex/openobex.mk b/package/openobex/openobex.mk
index 8337990eb8..2ef94298d3 100644
--- a/package/openobex/openobex.mk
+++ b/package/openobex/openobex.mk
@@ -18,7 +18,7 @@ OPENOBEX_AUTORECONF = YES
OPENOBEX_INSTALL_STAGING = YES
define OPENOBEX_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(OPENOBEX_SOURCE)
+ $(UNZIP) -d $(@D) $(OPENOBEX_DL_DIR)/$(OPENOBEX_SOURCE)
mv $(@D)/openobex-$(OPENOBEX_VERSION)-Source/* $(@D)
$(RM) -r $(@D)/openobex-$(OPENOBEX_VERSION)-Source
endef
diff --git a/package/opentyrian-data/opentyrian-data.mk b/package/opentyrian-data/opentyrian-data.mk
index 4b879df1c3..9e38b61bfa 100644
--- a/package/opentyrian-data/opentyrian-data.mk
+++ b/package/opentyrian-data/opentyrian-data.mk
@@ -10,7 +10,7 @@ OPENTYRIAN_DATA_SOURCE = tyrian21.zip
OPENTYRIAN_DATA_LICENSE = Freeware
define OPENTYRIAN_DATA_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(OPENTYRIAN_DATA_SOURCE)
+ $(UNZIP) -d $(@D) $(OPENTYRIAN_DATA_DL_DIR)/$(OPENTYRIAN_DATA_SOURCE)
endef
define OPENTYRIAN_DATA_INSTALL_TARGET_CMDS
diff --git a/package/perl/perl.mk b/package/perl/perl.mk
index ca23eb0bf7..434c41ca50 100644
--- a/package/perl/perl.mk
+++ b/package/perl/perl.mk
@@ -24,7 +24,7 @@ PERL_EXTRA_DOWNLOADS = $(PERL_CROSS_SITE)/$(PERL_CROSS_SOURCE)
# as a separate package. Instead, it is downloaded and extracted
# together with perl
define PERL_CROSS_EXTRACT
- $(call suitable-extractor,$(PERL_CROSS_SOURCE)) $(DL_DIR)/$(PERL_CROSS_SOURCE) | \
+ $(call suitable-extractor,$(PERL_CROSS_SOURCE)) $(PERL_DL_DIR)/$(PERL_CROSS_SOURCE) | \
$(TAR) --strip-components=1 -C $(@D) $(TAR_OPTIONS) -
endef
PERL_POST_EXTRACT_HOOKS += PERL_CROSS_EXTRACT
diff --git a/package/python-keyring/python-keyring.mk b/package/python-keyring/python-keyring.mk
index ef8b8af261..a615973912 100644
--- a/package/python-keyring/python-keyring.mk
+++ b/package/python-keyring/python-keyring.mk
@@ -11,7 +11,7 @@ PYTHON_KEYRING_SETUP_TYPE = setuptools
PYTHON_KEYRING_LICENSE = python software foundation license
define PYTHON_KEYRING_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(PYTHON_KEYRING_SOURCE)
+ $(UNZIP) -d $(@D) $(PYTHON_KEYRING_DL_DIR)/$(PYTHON_KEYRING_SOURCE)
mv $(@D)/keyring-$(PYTHON_KEYRING_VERSION)/* $(@D)
$(RM) -r $(@D)/keyring-$(PYTHON_KEYRING_VERSION)
endef
diff --git a/package/python-simplegeneric/python-simplegeneric.mk b/package/python-simplegeneric/python-simplegeneric.mk
index cc84320428..d3bfa69a7e 100644
--- a/package/python-simplegeneric/python-simplegeneric.mk
+++ b/package/python-simplegeneric/python-simplegeneric.mk
@@ -16,7 +16,7 @@ PYTHON_SIMPLEGENERIC_LICENSE = ZPL-2.1
PYTHON_SIMPLEGENERIC_SETUP_TYPE = setuptools
define PYTHON_SIMPLEGENERIC_EXTRACT_CMDS
- unzip $(DL_DIR)/$(PYTHON_SIMPLEGENERIC_SOURCE) -d $(@D)
+ unzip $(PYTHON_SIMPLEGENERIC_DL_DIR)/$(PYTHON_SIMPLEGENERIC_SOURCE) -d $(@D)
mv $(@D)/simplegeneric-$(PYTHON_SIMPLEGENERIC_VERSION)/* $(@D)
rmdir $(@D)/simplegeneric-$(PYTHON_SIMPLEGENERIC_VERSION)
endef
diff --git a/package/rapidxml/rapidxml.mk b/package/rapidxml/rapidxml.mk
index 2bec8fe1f2..9d034d8205 100644
--- a/package/rapidxml/rapidxml.mk
+++ b/package/rapidxml/rapidxml.mk
@@ -15,7 +15,7 @@ RAPIDXML_INSTALL_TARGET = NO
RAPIDXML_INSTALL_STAGING = YES
define RAPIDXML_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(RAPIDXML_SOURCE)
+ $(UNZIP) -d $(@D) $(RAPIDXML_DL_DIR)/$(RAPIDXML_SOURCE)
mv $(@D)/rapidxml-$(RAPIDXML_VERSION)/* $(@D)/
rmdir $(@D)/rapidxml-$(RAPIDXML_VERSION)
endef
diff --git a/package/rpi-wifi-firmware/rpi-wifi-firmware.mk b/package/rpi-wifi-firmware/rpi-wifi-firmware.mk
index 6c855a8e14..83473d5bb9 100644
--- a/package/rpi-wifi-firmware/rpi-wifi-firmware.mk
+++ b/package/rpi-wifi-firmware/rpi-wifi-firmware.mk
@@ -12,7 +12,7 @@ RPI_WIFI_FIRMWARE_SITE = https://raw.githubusercontent.com/RPi-Distro/firmware-n
RPI_WIFI_FIRMWARE_LICENSE = PROPRIETARY
define RPI_WIFI_FIRMWARE_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(RPI_WIFI_FIRMWARE_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define RPI_WIFI_FIRMWARE_INSTALL_TARGET_CMDS
diff --git a/package/sam-ba/sam-ba.mk b/package/sam-ba/sam-ba.mk
index 43e061beea..84c7bbb7aa 100644
--- a/package/sam-ba/sam-ba.mk
+++ b/package/sam-ba/sam-ba.mk
@@ -13,7 +13,7 @@ SAM_BA_LICENSE_FILES = doc/license.txt tcl_lib/boards.tcl \
applets/sam4c/libraries/libchip_sam4c/include/sam4c/sam4c32e-1.h
define HOST_SAM_BA_EXTRACT_CMDS
- $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(SAM_BA_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR) $(SAM_BA_DL_DIR)/$(SAM_BA_SOURCE)
mv $(BUILD_DIR)/sam-ba_cdc_linux/* $(@D)
rmdir $(BUILD_DIR)/sam-ba_cdc_linux/
endef
diff --git a/package/spidev_test/spidev_test.mk b/package/spidev_test/spidev_test.mk
index 1d657803b4..bf8170cd2c 100644
--- a/package/spidev_test/spidev_test.mk
+++ b/package/spidev_test/spidev_test.mk
@@ -32,7 +32,7 @@ endef
SPIDEV_TEST_POST_PATCH_HOOKS += SPIDEV_ADD_LINUX_IOCTL
define SPIDEV_TEST_EXTRACT_CMDS
- cp $(DL_DIR)/$(SPIDEV_TEST_SOURCE) $(@D)/spidev_test.c
+ cp $(SPIDEV_TEST_DL_DIR)/$(SPIDEV_TEST_SOURCE) $(@D)/spidev_test.c
endef
define SPIDEV_TEST_BUILD_CMDS
diff --git a/package/tar/tar.mk b/package/tar/tar.mk
index cb2ddc0eca..ad3f5ef763 100644
--- a/package/tar/tar.mk
+++ b/package/tar/tar.mk
@@ -40,7 +40,7 @@ HOST_TAR_SOURCE = tar-$(TAR_VERSION).cpio.gz
define HOST_TAR_EXTRACT_CMDS
mkdir -p $(@D)
cd $(@D) && \
- $(call suitable-extractor,$(HOST_TAR_SOURCE)) $(DL_DIR)/$(HOST_TAR_SOURCE) | cpio -i --preserve-modification-time
+ $(call suitable-extractor,$(HOST_TAR_SOURCE)) $(TAR_DL_DIR)/$(HOST_TAR_SOURCE) | cpio -i --preserve-modification-time
mv $(@D)/tar-$(TAR_VERSION)/* $(@D)
rmdir $(@D)/tar-$(TAR_VERSION)
endef
diff --git a/package/tesseract-ocr/tesseract-ocr.mk b/package/tesseract-ocr/tesseract-ocr.mk
index 9e315b239a..919a4360ed 100644
--- a/package/tesseract-ocr/tesseract-ocr.mk
+++ b/package/tesseract-ocr/tesseract-ocr.mk
@@ -62,7 +62,7 @@ TESSERACT_OCR_PRE_CONFIGURE_HOOKS += TESSERACT_OCR_PRECONFIGURE
# Language data files installation
define TESSERACT_OCR_INSTALL_LANG_DATA
$(foreach langfile,$(TESSERACT_OCR_DATA_FILES), \
- $(INSTALL) -D -m 0644 $(DL_DIR)/$(langfile) \
+ $(INSTALL) -D -m 0644 $(TESSERACT_OCR_DL_DIR)/$(langfile) \
$(TARGET_DIR)/usr/share/tessdata/$(langfile)
)
endef
diff --git a/package/ti-cgt-pru/ti-cgt-pru.mk b/package/ti-cgt-pru/ti-cgt-pru.mk
index 4f8e70bfce..2492498302 100644
--- a/package/ti-cgt-pru/ti-cgt-pru.mk
+++ b/package/ti-cgt-pru/ti-cgt-pru.mk
@@ -13,8 +13,8 @@ TI_CGT_PRU_LICENSE_FILES = PRU_Code_Generation_Tools_2.1.x_manifest.html \
PRU_CodeGen_Library_2.1_0222433C-30C1-442d-B5C6-2073BD97F80F.spdx.tag
define HOST_TI_CGT_PRU_EXTRACT_CMDS
- chmod +x $(DL_DIR)/$(TI_CGT_PRU_SOURCE)
- $(DL_DIR)/$(TI_CGT_PRU_SOURCE) --prefix $(@D) --mode unattended
+ chmod +x $(TI_CGT_PRU_DL_DIR)/$(TI_CGT_PRU_SOURCE)
+ $(TI_CGT_PRU_DL_DIR)/$(TI_CGT_PRU_SOURCE) --prefix $(@D) --mode unattended
mv $(@D)/ti-cgt-pru_$(TI_CGT_PRU_VERSION)/* $(@D)
rmdir $(@D)/ti-cgt-pru_$(TI_CGT_PRU_VERSION)/
endef
diff --git a/package/ti-gfx/ti-gfx.mk b/package/ti-gfx/ti-gfx.mk
index 428878a08b..9fad553780 100644
--- a/package/ti-gfx/ti-gfx.mk
+++ b/package/ti-gfx/ti-gfx.mk
@@ -94,8 +94,8 @@ TI_GFX_HDR_DIRS = OGLES2/EGL OGLES2/EWS OGLES2/GLES2 OGLES2/KHR \
OGLES/GLES bufferclass_ti/ pvr2d/ wsegl/
define TI_GFX_EXTRACT_CMDS
- chmod +x $(DL_DIR)/$(TI_GFX_SOURCE)
- printf "Y\nY\n qY\n\n" | $(DL_DIR)/$(TI_GFX_SOURCE) \
+ chmod +x $(TI_GFX_DL_DIR)/$(TI_GFX_SOURCE)
+ printf "Y\nY\n qY\n\n" | $(TI_GFX_DL_DIR)/$(TI_GFX_SOURCE) \
--prefix $(@D) \
--mode console
endef
diff --git a/package/ts4900-fpga/ts4900-fpga.mk b/package/ts4900-fpga/ts4900-fpga.mk
index ed951b8c8f..7bb62a3984 100644
--- a/package/ts4900-fpga/ts4900-fpga.mk
+++ b/package/ts4900-fpga/ts4900-fpga.mk
@@ -11,7 +11,7 @@ TS4900_FPGA_SITE = ftp://ftp.embeddedarm.com/ts-socket-macrocontrollers/ts-4900-
# https://github.com/embeddedarm/meta-ts/blob/f31860f1204b64f765a5380d3b93a2cf18234f90/recipes-extras/ts4900-fpga/ts4900-fpga.bb#L6
define TS4900_FPGA_EXTRACT_CMDS
- cp $(DL_DIR)/$(TS4900_FPGA_SOURCE) $(@D)
+ cp $(TS4900_FPGA_DL_DIR)/$(TS4900_FPGA_SOURCE) $(@D)
endef
define TS4900_FPGA_INSTALL_TARGET_CMDS
diff --git a/package/unscd/unscd.mk b/package/unscd/unscd.mk
index f0eb5d6ad6..36f2de271b 100644
--- a/package/unscd/unscd.mk
+++ b/package/unscd/unscd.mk
@@ -11,7 +11,7 @@ UNSCD_LICENSE = GPL-2.0
UNSCD_LICENSE_FILES = $(UNSCD_SOURCE)
define UNSCD_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(UNSCD_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define UNSCD_BUILD_CMDS
diff --git a/package/urg/urg.mk b/package/urg/urg.mk
index 4a41779709..7bb1e807dd 100644
--- a/package/urg/urg.mk
+++ b/package/urg/urg.mk
@@ -26,7 +26,7 @@ URG_CONFIG_SCRIPTS = c_urg-config urg-config
define URG_EXTRACT_CMDS
$(RM) -rf $(URG_DIR)
- $(UNZIP) -d $(BUILD_DIR)/ $(DL_DIR)/$(URG_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR)/ $(URG_DL_DIR)/$(URG_SOURCE)
test -d $(URG_DIR) || \
mv $(BUILD_DIR)/$(subst .zip,,$(URG_SOURCE)) $(URG_DIR)
endef
diff --git a/package/waf/waf.mk b/package/waf/waf.mk
index 3d61f267e6..c54a3e2b38 100644
--- a/package/waf/waf.mk
+++ b/package/waf/waf.mk
@@ -9,7 +9,7 @@ WAF_SOURCE = waf-$(WAF_VERSION)
WAF_SITE = https://waf.io/
define HOST_WAF_EXTRACT_CMDS
- $(INSTALL) -D -m 0755 $(DL_DIR)/waf-$(WAF_VERSION) $(@D)/waf
+ $(INSTALL) -D -m 0755 $(WAF_DL_DIR)/waf-$(WAF_VERSION) $(@D)/waf
endef
define HOST_WAF_INSTALL_CMDS
diff --git a/package/whetstone/whetstone.mk b/package/whetstone/whetstone.mk
index d9b45638c5..3b6ec3419b 100644
--- a/package/whetstone/whetstone.mk
+++ b/package/whetstone/whetstone.mk
@@ -9,7 +9,7 @@ WHETSTONE_SOURCE = whetstone.c
WHETSTONE_SITE = http://www.netlib.org/benchmark
define WHETSTONE_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(WHETSTONE_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define WHETSTONE_BUILD_CMDS
diff --git a/package/wilc1000-firmware/wilc1000-firmware.mk b/package/wilc1000-firmware/wilc1000-firmware.mk
index 5dcf19045b..6f504d67a0 100644
--- a/package/wilc1000-firmware/wilc1000-firmware.mk
+++ b/package/wilc1000-firmware/wilc1000-firmware.mk
@@ -11,7 +11,7 @@ WILC1000_FIRMWARE_SOURCE = v$(WILC1000_FIRMWARE_VERSION)_Firmware.zip
WILC1000_FIRMWARE_LICENSE = PROPRIETARY
define WILC1000_FIRMWARE_EXTRACT_CMDS
- $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(WILC1000_FIRMWARE_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR) $(WILC1000_FIRMWARE_DL_DIR)/$(WILC1000_FIRMWARE_SOURCE)
mv $(BUILD_DIR)/wireless-firmware-$(WILC1000_FIRMWARE_VERSION)_Firmware/* $(@D)
rmdir $(BUILD_DIR)/wireless-firmware-$(WILC1000_FIRMWARE_VERSION)_Firmware
endef
diff --git a/package/zynq-boot-bin/zynq-boot-bin.mk b/package/zynq-boot-bin/zynq-boot-bin.mk
index f5f2f10d1d..b29c48ca60 100644
--- a/package/zynq-boot-bin/zynq-boot-bin.mk
+++ b/package/zynq-boot-bin/zynq-boot-bin.mk
@@ -12,7 +12,7 @@ ZYNQ_BOOT_BIN_LICENSE = GPL-3.0+
HOST_ZYNQ_BOOT_BIN_DEPENDENCIES = host-python
define HOST_ZYNQ_BOOT_BIN_EXTRACT_CMDS
- cp $(DL_DIR)/$(ZYNQ_BOOT_BIN_SOURCE) $(@D)
+ cp $(ZYNQ_BOOT_BIN_DL_DIR)/$(ZYNQ_BOOT_BIN_SOURCE) $(@D)
endef
define HOST_ZYNQ_BOOT_BIN_INSTALL_CMDS
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 06/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR)
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (4 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 05/13] packages: use new $($PKG)_DL_DIR) variable Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-07-04 16:22 ` [Buildroot] [PATCH 07/13] support/download: make sure the download folder is created Maxime Hadjinlian
` (7 subsequent siblings)
13 siblings, 0 replies; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
Let the infrastructure use the new variable $(PKG)_DL_DIR
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-download.mk | 2 +-
package/pkg-generic.mk | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index be61fb130a..9a455dc79b 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -95,7 +95,7 @@ define DOWNLOAD
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
-n $($(PKG)_RAW_BASE_NAME) \
-N $($(PKG)_RAWNAME) \
- -o $(DL_DIR)/$(notdir $(1)) \
+ -o $($(PKG)_DL_DIR)/$(notdir $(1)) \
$(if $($(PKG)_GIT_SUBMODULES),-r) \
$(DOWNLOAD_URIS) \
$(QUIET) \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index c1cd41fda5..5d169b20d6 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -579,7 +579,7 @@ $(2)_TARGET_DIRCLEAN = $$($(2)_DIR)/.stamp_dircleaned
# default extract command
$(2)_EXTRACT_CMDS ?= \
- $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$(DL_DIR)/$$($(2)_SOURCE) | \
+ $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$($(2)_DL_DIR)/$$($(2)_SOURCE) | \
$$(TAR) --strip-components=$$($(2)_STRIP_COMPONENTS) \
-C $$($(2)_DIR) \
$$(foreach x,$$($(2)_EXCLUDES),--exclude='$$(x)' ) \
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 07/13] support/download: make sure the download folder is created
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (5 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 06/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-07-04 16:22 ` [Buildroot] [PATCH 08/13] pkg-generic: add a subdirectory to the DL_DIR Maxime Hadjinlian
` (6 subsequent siblings)
13 siblings, 0 replies; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
At the moment, it means that we make sure that BR2_DL_DIR is created, in
the future, it will make sure that BR2_DL_DIR/PKG_NAME/ is created.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
support/download/dl-wrapper | 1 +
1 file changed, 1 insertion(+)
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 50c14a2e16..f14a60c173 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -50,6 +50,7 @@ main() {
if [ -z "${output}" ]; then
error "no output specified, use -o\n"
fi
+ mkdir -p "$(dirname "${output}")"
# If the output file already exists and:
# - there's no .hash file: do not download it again and exit promptly
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 08/13] pkg-generic: add a subdirectory to the DL_DIR
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (6 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 07/13] support/download: make sure the download folder is created Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-10-23 19:42 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 09/13] pkg-download: support new subdir for mirrors Maxime Hadjinlian
` (5 subsequent siblings)
13 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
With all the previous changes, we are now ready to add a subdirectory to
the DL_DIR.
The structure will now be DL_DIR/PKG_NAME/{FILE1,FILE2}
This is needed for multiple reasons:
- Avoid patches with name like SHA1.patch laying flat in DL_DIR,
which makes it hard to know to which packages they apply
- Avoid possible collisions if two releases have the same name
(e.g: v01.tar)
- Allow the possibility to handle a git cache per package in the
newly created subdirectory.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-generic.mk | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 5d169b20d6..7364551471 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -434,7 +434,7 @@ endif
$(2)_BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
$(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR = $$(DL_DIR)
+$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
$(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME)
ifndef $(2)_SUBDIR
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 09/13] pkg-download: support new subdir for mirrors
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (7 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 08/13] pkg-generic: add a subdirectory to the DL_DIR Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-10-23 19:49 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 10/13] pkg-generic: introduce _SAME_SOURCE_AS Maxime Hadjinlian
` (4 subsequent siblings)
13 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
Since we introduced subdirectory to the DL_DIR, we need to support them
in the PRIMARY and BACKUP mirror as they evolve to the new tree
structure.
We check first the new URI (with the subdir), and in case of failure, we
go check without.
By checking both URIs, we ensure that old mirror are usable.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-download.mk | 2 ++
1 file changed, 2 insertions(+)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 9a455dc79b..47671e21a7 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -75,6 +75,7 @@ export BR_NO_CHECK_HASH_FOR =
ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))/$(notdir $($(PKG)_DL_DIR)) \
-u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
endif
@@ -83,6 +84,7 @@ DOWNLOAD_URIS += \
-u $($(PKG)_SITE_METHOD)+$(dir $(1))
ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_BACKUP_SITE))/$(notdir $($(PKG)_DL_DIR)) \
-u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
endif
endif
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 10/13] pkg-generic: introduce _SAME_SOURCE_AS
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (8 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 09/13] pkg-download: support new subdir for mirrors Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-10-23 19:55 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Maxime Hadjinlian
` (3 subsequent siblings)
13 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
This per package variable can be used to specify that a package shares
the same sources as another package.
The use case here is for example, you have linux-headers and linux, you
don't want to download twice the kernel (because of the introduction of
the subdirs in DL_DIR).
So you want to specify in linux-headers, this variable, so it will use
the linux's source.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-generic.mk | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 7364551471..a8e0686068 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -434,7 +434,9 @@ endif
$(2)_BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
$(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
+$(2)_DL_DIR = $$(if $$($(2)_SAME_SOURCE_AS), \
+ $$(DL_DIR)/$$($$(call UPPERCASE,$$($(2)_SAME_SOURCE_AS))_RAWNAME), \
+ $$(DL_DIR)/$$($(2)_RAWNAME))
$(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME)
ifndef $(2)_SUBDIR
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR)
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (9 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 10/13] pkg-generic: introduce _SAME_SOURCE_AS Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-10-23 20:04 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 12/13] download: add flock call before dl-wrapper Maxime Hadjinlian
` (2 subsequent siblings)
13 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Config.in | 3 +++
docs/manual/adding-packages-generic.txt | 6 +++---
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/Config.in b/Config.in
index 72ceadfde6..0ad43679f3 100644
--- a/Config.in
+++ b/Config.in
@@ -195,6 +195,9 @@ config BR2_DL_DIR
If the Linux shell environment has defined the BR2_DL_DIR
environment variable, then this overrides this configuration
item.
+ The directory is organized with a subdirectory for each package.
+ Each package has its own $(LIBFOO_DL_DIR) variable that can be used
+ to find the correct path.
The default is $(TOPDIR)/dl
diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
index 12cf1605d6..6e3b503f6b 100644
--- a/docs/manual/adding-packages-generic.txt
+++ b/docs/manual/adding-packages-generic.txt
@@ -263,7 +263,7 @@ information is (assuming the package name is +libfoo+) :
the file using this URL. Otherwise, Buildroot will assume the file
to be downloaded is located at +LIBFOO_SITE+. Buildroot will not do
anything with those additional files, except download them: it will
- be up to the package recipe to use them from +$(DL_DIR)+.
+ be up to the package recipe to use them from +$(LIBFOO_DL_DIR)+.
* +LIBFOO_SITE_METHOD+ determines the method used to fetch or copy the
package source code. In many cases, Buildroot guesses the method
@@ -537,8 +537,8 @@ In the action definitions, you can use the following variables:
* +$(@D)+, which contains the directory in which the package source
code has been uncompressed.
-* +$(DL_DIR)+ contains the path to the directory where all the downloads made
- by Buildroot are stored.
+* +$(LIBFOO_DL_DIR)+ contains the path to the directory where all the downloads
+ made by Buildroot are stored.
* +$(TARGET_CC)+, +$(TARGET_LD)+, etc. to get the target
cross-compilation utilities
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 12/13] download: add flock call before dl-wrapper
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (10 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-10-23 20:17 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 13/13] download: git: introduce cache feature Maxime Hadjinlian
2017-07-27 22:37 ` [Buildroot] [PATCH 00/13] New DL_DIR organisation; git " Peter Seiderer
13 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
In order to introduce the cache mechanisms, we need to have a lock on
the $(LIBFOO_DL_DIR), otherwise it would be impossible to do parallel
download (a shared DL_DIR for two buildroot instances).
To make sure the directory exists, the mkdir call has been removed from
the dl-wrapper and put in the infrastructure.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-download.mk | 4 +++-
support/download/dl-wrapper | 1 -
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 47671e21a7..c67f1ba4e1 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -19,6 +19,7 @@ SSH := $(call qstrip,$(BR2_SSH))
export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
DL_WRAPPER = support/download/dl-wrapper
+FLOCK = flock $($(PKG)_DL_DIR)/
# DL_DIR may have been set already from the environment
ifeq ($(origin DL_DIR),undefined)
@@ -91,7 +92,8 @@ endif
define DOWNLOAD
$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
- $(EXTRA_ENV) $(DL_WRAPPER) \
+ $(Q)mkdir -p $($(PKG)_DL_DIR)/
+ $(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
-c $($(PKG)_DL_VERSION) \
-f $(notdir $(1)) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index f14a60c173..50c14a2e16 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -50,7 +50,6 @@ main() {
if [ -z "${output}" ]; then
error "no output specified, use -o\n"
fi
- mkdir -p "$(dirname "${output}")"
# If the output file already exists and:
# - there's no .hash file: do not download it again and exit promptly
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 13/13] download: git: introduce cache feature
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (11 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 12/13] download: add flock call before dl-wrapper Maxime Hadjinlian
@ 2017-07-04 16:22 ` Maxime Hadjinlian
2017-10-27 18:28 ` Peter Seiderer
2017-07-27 22:37 ` [Buildroot] [PATCH 00/13] New DL_DIR organisation; git " Peter Seiderer
13 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-04 16:22 UTC (permalink / raw)
To: buildroot
Now we keep the git clone that we download and generates our tarball
from there.
The main goal here is that if you change the version of a package (say
Linux), instead of cloning all over again, you will simply 'git fetch'
from the repo the missing objects, then generates the tarball again.
This should speed the 'source' part of the build significantly.
The drawback is that the DL_DIR will grow much larger; but time is more
important than disk space nowadays.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
support/download/git | 69 ++++++++++++++++++++++++++++++----------------------
1 file changed, 40 insertions(+), 29 deletions(-)
diff --git a/support/download/git b/support/download/git
index a49e448e60..834345b53a 100755
--- a/support/download/git
+++ b/support/download/git
@@ -39,28 +39,34 @@ _git() {
eval ${GIT} "${@}"
}
-# Try a shallow clone, since it is faster than a full clone - but that only
-# works if the version is a ref (tag or branch). Before trying to do a shallow
-# clone we check if ${cset} is in the list provided by git ls-remote. If not
-# we fall back on a full clone.
-#
-# Messages for the type of clone used are provided to ease debugging in case of
-# problems
-git_done=0
-if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
- printf "Doing shallow clone\n"
- if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
- git_done=1
- else
- printf "Shallow clone failed, falling back to doing a full clone\n"
+# We want to check if a cache of the git clone of this repo already exists.
+git_cache="${BR2_DL_DIR}/${basename%%-*}/git"
+
+# If the cache directory already exists, don't try to clone.
+if [ ! -d "${git_cache}" ]; then
+ # Try a shallow clone, since it is faster than a full clone - but that
+ # only works if the versionis a ref (tag or branch). Before trying to do a
+ # shallow clone we check if ${cset} is in the list provided by git
+ # ls-remote. If not we fall back on a full clone.
+ #
+ # Messages for the type of clone used are provided to ease debugging in
+ # case of problems
+ git_done=0
+ if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
+ printf "Doing shallow clone\n"
+ if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${git_cache}'"; then
+ git_done=1
+ else
+ printf "Shallow clone failed, falling back to doing a full clone\n"
+ fi
+ fi
+ if [ ${git_done} -eq 0 ]; then
+ printf "Doing full clone\n"
+ _git clone ${verbose} "${@}" "'${uri}'" "'${git_cache}'"
fi
-fi
-if [ ${git_done} -eq 0 ]; then
- printf "Doing full clone\n"
- _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
fi
-pushd "${basename}" >/dev/null
+pushd "${git_cache}" >/dev/null
# Try to get the special refs exposed by some forges (pull-requests for
# github, changes for gerrit...). There is no easy way to know whether
@@ -86,20 +92,25 @@ if [ ${recurse} -eq 1 ]; then
_git submodule update --init --recursive
fi
-# We do not want the .git dir; we keep other .git files, in case they
-# are the only files in their directory.
+# Generate the archive, sort with the C locale so that it is reproducible
+# We do not want the .git dir; we keep other .git
+# files, in case they are the only files in their directory.
# The .git dir would generate non reproducible tarballs as it depends on
# the state of the remote server. It also would generate large tarballs
# (gigabytes for some linux trees) when a full clone took place.
-rm -rf .git
+find . -not -type d \
+ -and -not -path "./.git/*" >"${BR2_DL_DIR}/${basename}.list"
+LC_ALL=C sort <"${BR2_DL_DIR}/${basename}.list" >"${BR2_DL_DIR}/${basename}.list.sorted"
-popd >/dev/null
-
-# Generate the archive, sort with the C locale so that it is reproducible
-find "${basename}" -not -type d >"${basename}.list"
-LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
# Create GNU-format tarballs, since that's the format of the tarballs on
# sources.buildroot.org and used in the *.hash files
-tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
- -T "${basename}.list.sorted" >"${output}.tar"
+tar cf - --transform="s/^\./${basename}/" \
+ --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
+ -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}.tar"
gzip -n <"${output}.tar" >"${output}"
+tar tf "${output}"
+
+rm -f "${BR2_DL_DIR}/${basename}.list"
+rm -f "${BR2_DL_DIR}/${basename}.list.sorted"
+
+popd >/dev/null
--
2.13.2
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 01/13] pkg-{download, generic}: remove source-check
2017-07-04 16:21 ` [Buildroot] [PATCH 01/13] pkg-{download, generic}: remove source-check Maxime Hadjinlian
@ 2017-07-04 17:37 ` Thomas Petazzoni
0 siblings, 0 replies; 41+ messages in thread
From: Thomas Petazzoni @ 2017-07-04 17:37 UTC (permalink / raw)
To: buildroot
Hello,
On Tue, 4 Jul 2017 18:21:59 +0200, Maxime Hadjinlian wrote:
> This feature is not used by anyone in the core developpers and makes a
> drastic simplification of the pkg-download infrastructure harder.
>
> The future patch will move much of what's in the current pkg-download.mk
> file into the dl-wrapper which is a shell script.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
You forgot to remove lots of references to source-check, do a "git grep
source-check" to find them.
Thanks!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 02/13] core/pkg-download: change all helpers to use common options
2017-07-04 16:22 ` [Buildroot] [PATCH 02/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
@ 2017-07-04 23:09 ` Arnout Vandecappelle
2017-07-04 23:14 ` Yann E. MORIN
0 siblings, 1 reply; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-07-04 23:09 UTC (permalink / raw)
To: buildroot
On 04-07-17 18:22, Maxime Hadjinlian wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
[snip]
> diff --git a/support/download/check-hash b/support/download/check-hash
> index c1ff53c02b..bf2dbb1a8b 100755
> --- a/support/download/check-hash
> +++ b/support/download/check-hash
> @@ -19,7 +19,7 @@ set -e
> # 3: the hash file exists and there was no hash to check the file against
> # 4: the hash file exists and at least one hash type is unknown
>
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
This doesn't make sense to me: the global DL_GETOPTS are never passed to
check-hash, and you're not using any of them. So I'd keep the explicit :q here.
> case "${OPT}" in
> q) exec >/dev/null;;
> \?) exit 1;;
[snip]
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index f944b71db5..a29411e0ae 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -19,6 +19,8 @@
> # We want to catch any unexpected failure, and exit immediately.
> set -e
>
> +export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"
Since this variable is defined here, it makes sense to also document the
options here. In fact, to make it makes more sense to document them all here
rather than in each individual backend. Of course, no need to go and remove it
in each backend, but I would like to have then documented here. Especially since
a later patch will also parse them here.
> +
> main() {
> local OPT OPTARG
> local backend output hfile recurse quiet
> @@ -83,7 +85,10 @@ main() {
> # If the backend fails, we can just remove the temporary directory to
> # remove all the cruft it may have left behind. Then we just exit in
> # error too.
> - if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
> + if ! "${OLDPWD}/support/download/${backend}" \
> + ${quiet} ${recurse} \
I don't like it very much when options are mixed like this: either fill lines
up completely, or do one option (or option combination, of course, like -o
"...") per line. But since this will be changed again later it's not worth
changing this patch.
Regards,
Arnout
> + -o "${tmpf}" "${@}"
> + then
> rm -rf "${tmpd}"
> exit 1
> fi
[snip]
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 02/13] core/pkg-download: change all helpers to use common options
2017-07-04 23:09 ` Arnout Vandecappelle
@ 2017-07-04 23:14 ` Yann E. MORIN
2017-07-05 0:38 ` Arnout Vandecappelle
0 siblings, 1 reply; 41+ messages in thread
From: Yann E. MORIN @ 2017-07-04 23:14 UTC (permalink / raw)
To: buildroot
Arnout, All,
On 2017-07-05 01:09 +0200, Arnout Vandecappelle spake thusly:
> On 04-07-17 18:22, Maxime Hadjinlian wrote:
> > From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > diff --git a/support/download/check-hash b/support/download/check-hash
> > index c1ff53c02b..bf2dbb1a8b 100755
> > --- a/support/download/check-hash
> > +++ b/support/download/check-hash
> > @@ -19,7 +19,7 @@ set -e
> > # 3: the hash file exists and there was no hash to check the file against
> > # 4: the hash file exists and at least one hash type is unknown
> >
> > -while getopts :q OPT; do
> > +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
>
> This doesn't make sense to me: the global DL_GETOPTS are never passed to
> check-hash, and you're not using any of them. So I'd keep the explicit :q here.
It took me a bit of time to understand what you meant, because
BR_BACKEND_DL_GETOPTS is actually exported by dl-wrapper, and it is
dl-wrapper that calls check-hash. So let me rephrase it, and you'll tell
me if I understood correctly:
This doesn't make sense to me: none of the variabels from
BR_BACKEND_DL_GETOPTS are ever passed to check-hash, so [...]
Right? If so I agree.
Regards,
Yann E. MORIN.
> > case "${OPT}" in
> > q) exec >/dev/null;;
> > \?) exit 1;;
>
> [snip]
> > diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> > index f944b71db5..a29411e0ae 100755
> > --- a/support/download/dl-wrapper
> > +++ b/support/download/dl-wrapper
> > @@ -19,6 +19,8 @@
> > # We want to catch any unexpected failure, and exit immediately.
> > set -e
> >
> > +export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"
>
> Since this variable is defined here, it makes sense to also document the
> options here. In fact, to make it makes more sense to document them all here
> rather than in each individual backend. Of course, no need to go and remove it
> in each backend, but I would like to have then documented here. Especially since
> a later patch will also parse them here.
>
> > +
> > main() {
> > local OPT OPTARG
> > local backend output hfile recurse quiet
> > @@ -83,7 +85,10 @@ main() {
> > # If the backend fails, we can just remove the temporary directory to
> > # remove all the cruft it may have left behind. Then we just exit in
> > # error too.
> > - if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
> > + if ! "${OLDPWD}/support/download/${backend}" \
> > + ${quiet} ${recurse} \
>
> I don't like it very much when options are mixed like this: either fill lines
> up completely, or do one option (or option combination, of course, like -o
> "...") per line. But since this will be changed again later it's not worth
> changing this patch.
>
> Regards,
> Arnout
>
> > + -o "${tmpf}" "${@}"
> > + then
> > rm -rf "${tmpd}"
> > exit 1
> > fi
> [snip]
>
>
> --
> Arnout Vandecappelle arnout at mind be
> Senior Embedded Software Architect +32-16-286500
> Essensium/Mind http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 03/13] download: put most of the infra in dl-wrapper
2017-07-04 16:22 ` [Buildroot] [PATCH 03/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
@ 2017-07-05 0:32 ` Arnout Vandecappelle
0 siblings, 0 replies; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-07-05 0:32 UTC (permalink / raw)
To: buildroot
[Thomas, question for you in the middle]
On 04-07-17 18:22, Maxime Hadjinlian wrote:
> The goal here is to simplify the infrastructure by putting most of the
> code in the dl-wrapper as it's easier to implement and to read.
Yes! I like it!
>
> Most of the function where common already, this patch finalize it by
> making the pkg-download.mk pass all the parameters needed to the
> dl-wrapper which in turns will pass everything to every backend.
>
> The backend will then cherry-pick what it needs from theses arguments
> and act accordingly.
>
> It eases the transition to the addition of a sub directory per package
> in the DL_DIR, and later on, a git cache.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> package/pkg-download.mk | 166 ++++++++------------------------------------
This huge diff could have been simplified a little by
1. Removing the third argument to DOWNLOAD_INNER (could actually be part of the
first patch of this series).
2. Replacing the calls to the DOWNLOAD_* helpers with calls to dl-wrapper - the
same patch can also replace the case with "-b $($(PKG)_SITE_METHOD)".
3. Replacing the entire DOWNLOAD_INNER with a single call to dl-wrapper.
This can stay as a single big patch but it's quite tricky to review it...
> support/download/bzr | 2 +-
> support/download/cp | 2 +-
> support/download/cvs | 2 +-
> support/download/dl-wrapper | 65 ++++++++++-------
> support/download/wget | 7 +-
> 6 files changed, 77 insertions(+), 167 deletions(-)
>
[snip]
> +ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
> +DOWNLOAD_URIS += \
> + -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
I was at first not so enthusiastic about this URI list and was going to suggest
passing BR2_PRIMARY_SITE into dl-wrapper instead, but the loop in dl-wrapper is
in fact quite elegant.
> +endif
>
> -define DOWNLOAD_CVS
> - $(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
> - -o $(DL_DIR)/$($(PKG)_SOURCE) \
> - $(QUIET) \
> - -- \
> - -u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
> - -c $($(PKG)_DL_VERSION) \
> - -N $($(PKG)_RAWNAME) \
> - -n $($(PKG)_RAW_BASE_NAME) \
> - $($(PKG)_DL_OPTS)
> -endef
> +ifeq ($(BR2_PRIMARY_SITE_ONLY),)
> +DOWNLOAD_URIS += \
> + -u $($(PKG)_SITE_METHOD)+$(dir $(1))
Although I find this solution pretty elegant, Thomas was a bit confused by it
and I understand his reasoning. So to make things explicit, and in line with how
we do things elsewhere:
ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
DOWNLOAD_URI_PRIMARY = -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
endif
ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
DOWNLOAD_URI_SECONDARY = -u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
endif
and in the call itself:
$(DOWNLOAD_URI_PRIMARY) \
$(if $(BR2_PRIMARY_SITE_ONLY),,\
-u $($(PKG)_SITE_METHOD)+$(dir $(1)) \
$(DOWNLOAD_URI_SECONDARY)) \
Thomas, do you think this is clearer?
> +ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
> +DOWNLOAD_URIS += \
> + -u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
> +endif
> +endif
>
> -define DOWNLOAD_SVN
> - $(EXTRA_ENV) $(DL_WRAPPER) -b svn \
> - -o $(DL_DIR)/$($(PKG)_SOURCE) \
> - $(QUIET) \
> - -- \
> - -u $($(PKG)_SITE) \
> +define DOWNLOAD
> + $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
No need for the export, there is just one command.
Not related to this patch, but what we do here is actually wrong: when a
package uses one of these download methods, *everything* related to that package
including patches etc. will not be checked... However, apparently we simply
don't support downloading anything extra for the VCS download methods, so it
doesn't matter.
> + $(EXTRA_ENV) $(DL_WRAPPER) \
> -c $($(PKG)_DL_VERSION) \
[snip]
> diff --git a/support/download/cp b/support/download/cp
> index 52fe2de83d..9c64b7b70b 100755
> --- a/support/download/cp
> +++ b/support/download/cp
> @@ -23,7 +23,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=;;
> o) output="${OPTARG}";;
> - u) source="${OPTARG}";;
> + u) source="${OPTARG#*//}";;
^^^ *://
This stuff would also be easier to understand if it was introduced in a
separate patch. Basically, previously it was called with
-u $(call stripurischeme,$(call qstrip,$(1)))
and now it is called with
-u '$(call qstrip,$(1))'
so we still need to strip the urischeme.
> :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> diff --git a/support/download/cvs b/support/download/cvs
> index 69d5c71f28..9caffb4b82 100755
> --- a/support/download/cvs
> +++ b/support/download/cvs
> @@ -21,7 +21,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-Q;;
> o) output="${OPTARG}";;
> - u) uri="${OPTARG}";;
> + u) uri="${OPTARG#*//}";;
> c) rev="${OPTARG}";;
> N) rawname="${OPTARG}";;
> n) basename="${OPTARG}";;
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index a29411e0ae..50c14a2e16 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -19,31 +19,34 @@
> # We want to catch any unexpected failure, and exit immediately.
> set -e
>
> -export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"
> +export BR_BACKEND_DL_GETOPTS=":hc:o:H:n:N:ru:qf:"
Something not right here: you're not adding any options to the download
helpers, so this shouldn't change...
Options that are only valid for the wrapper and not for the helper should not
be included here since they won't be passed down.
>
> main() {
> local OPT OPTARG
> local backend output hfile recurse quiet
> + local -a uris
>
> # Parse our options; anything after '--' is for the backend
> - while getopts :hb:o:H:rq OPT; do
> + while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> h) help; exit 0;;
> - b) backend="${OPTARG}";;
> + c) cset="${OPTARG}";;
> o) output="${OPTARG}";;
> + n) raw_base_name="${OPTARG}";;
> + N) base_name="${OPTARG}";;
> H) hfile="${OPTARG}";;
> r) recurse="-r";;
> + f) filename="${OPTARG}";;
> + u) uris+=( "${OPTARG}" );;
> q) quiet="-q";;
> :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
> \?) error "unknown option '%s'\n" "${OPTARG}";;
> esac
> done
> +
> # Forget our options, and keep only those for the backend
> shift $((OPTIND-1))
>
> - if [ -z "${backend}" ]; then
> - error "no backend specified, use -b\n"
> - fi
> if [ -z "${output}" ]; then
> error "no output specified, use -o\n"
> fi
> @@ -82,16 +85,29 @@ main() {
> # Doing the 'cd' here rather than in all backends is easier.
> cd "${tmpd}"
>
> - # If the backend fails, we can just remove the temporary directory to
> - # remove all the cruft it may have left behind. Then we just exit in
> - # error too.
> - if ! "${OLDPWD}/support/download/${backend}" \
> - ${quiet} ${recurse} \
> - -o "${tmpf}" "${@}"
> - then
> - rm -rf "${tmpd}"
> - exit 1
> - fi
> + # Look through all the uris that we were given to downoad the package
> + # source
> + for uri in "${uris[@]}"; do
> + backend=${uri%+*}
> + case "${backend}" in
> + git|svn|cvs|bzr|file|scp|hg) ;;
The backend for "file" is called "cp", so probably the easiest is to rename
that one to "file".
> + *) backend="wget" ;;
> + esac
> + uri=${uri#*+}
> +
> + # If the backend fails, we can just remove the temporary directory to
> + # remove all the cruft it may have left behind. Then we just exit in
No, we don't exit any more.
> + # error too.
> + if "${OLDPWD}/support/download/${backend}" \
> + -c "${cset}" -n "${raw_base_name}" \
> + -N "${raw_name}" -f "${filename}" -u "${uri}" -o "${tmpf}" \
> + ${quiet} ${recurse} "${@}"
> + then
> + break
Wrong, see below.
> + else
> + rm -rf "${tmpd:?}/*"
There should be a continue here, see below.
> + fi
We have to do the hash check within the loop - if it fails, we should continue
with the next URI. We should break if the hash check is successful. And we
should keep track somewhere if any download+check was successful, so we can give
the appropriate exit code.
> + done
>
> # cd back to free the temp-dir, so we can remove it later
> cd "${OLDPWD}"
> @@ -164,16 +180,13 @@ DESCRIPTION
>
> -h This help text.
>
> - -b BACKEND
> - Wrap the specified BACKEND. Known backends are:
> - bzr Bazaar
> - cp Local files
> - cvs Concurrent Versions System
> - git Git
> - hg Mercurial
> - scp Secure copy
> - svn Subversion
> - wget HTTP download
> + -u URIs
> + The URI to get the file from, the URI must respect the format given in
> + the example.
> + You may give as many '-u URI' as you want, the script will stop at the
> + frist successful download.
> +
> + Example: backend+URI; git+http://example.com or http+http://example.com
>
> -o FILE
> Store the downloaded archive in FILE.
> diff --git a/support/download/wget b/support/download/wget
> index fece6663ca..3e6a6b446c 100755
> --- a/support/download/wget
> +++ b/support/download/wget
> @@ -8,6 +8,7 @@ set -e
> # Options:
> # -q Be quiet.
> # -o FILE Save into file FILE.
> +# -f FILENAME The filename of the tarball to get at URL
> # -u URL Download file at URL.
> #
> # Environment:
> @@ -18,6 +19,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-q;;
> o) output="${OPTARG}";;
> + f) filename="${OPTARG}";;
> u) url="${OPTARG}";;
> :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> @@ -32,4 +34,7 @@ _wget() {
> eval ${WGET} "${@}"
> }
>
> -_wget ${verbose} "${@}" -O "'${output}'" "'${url}'"
> +# Replace every '?' with '%3F' in the URI
> +url=${url//\?/%3F}
We *don't* do that for the normal download, only for the primary/secondary.
Indeed, doing this will fail the download of musl-compat-headers. Not sure how
to solve it elegantly.
Oh, hang on, this is completely wrong: you need to do this substitution in the
filename, not in the url. The url normally doesn't have any ?, and if it does it
should be kept. The subsitution is there to make sure something like
musl-compat-headers, which has a filename like "queue.h?rev=1.70", can still be
downloaded from primary/secondary. But again, you have to distinguish between
the primary/secondary case and the normal case.
Regards,
Arnout
> +
> +_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 02/13] core/pkg-download: change all helpers to use common options
2017-07-04 23:14 ` Yann E. MORIN
@ 2017-07-05 0:38 ` Arnout Vandecappelle
2017-07-05 7:09 ` Yann E. MORIN
0 siblings, 1 reply; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-07-05 0:38 UTC (permalink / raw)
To: buildroot
On 05-07-17 01:14, Yann E. MORIN wrote:
> Arnout, All,
>
> On 2017-07-05 01:09 +0200, Arnout Vandecappelle spake thusly:
>> On 04-07-17 18:22, Maxime Hadjinlian wrote:
>>> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>>> diff --git a/support/download/check-hash b/support/download/check-hash
>>> index c1ff53c02b..bf2dbb1a8b 100755
>>> --- a/support/download/check-hash
>>> +++ b/support/download/check-hash
>>> @@ -19,7 +19,7 @@ set -e
>>> # 3: the hash file exists and there was no hash to check the file against
>>> # 4: the hash file exists and at least one hash type is unknown
>>>
>>> -while getopts :q OPT; do
>>> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
>>
>> This doesn't make sense to me: the global DL_GETOPTS are never passed to
>> check-hash, and you're not using any of them. So I'd keep the explicit :q here.
>
> It took me a bit of time to understand what you meant, because
> BR_BACKEND_DL_GETOPTS is actually exported by dl-wrapper, and it is
> dl-wrapper that calls check-hash. So let me rephrase it, and you'll tell
> me if I understood correctly:
>
> This doesn't make sense to me: none of the variabels from
> BR_BACKEND_DL_GETOPTS are ever passed to check-hash, so [...]
Yes, that's what I meant. I guess I don't formulate very clearly at 01:09 (much
less at 02:38...)
Regards,
Arnout
>
> Right? If so I agree.
>
> Regards,
> Yann E. MORIN.
>
>>> case "${OPT}" in
>>> q) exec >/dev/null;;
>>> \?) exit 1;;
>>
>> [snip]
>>> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
>>> index f944b71db5..a29411e0ae 100755
>>> --- a/support/download/dl-wrapper
>>> +++ b/support/download/dl-wrapper
>>> @@ -19,6 +19,8 @@
>>> # We want to catch any unexpected failure, and exit immediately.
>>> set -e
>>>
>>> +export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"
>>
>> Since this variable is defined here, it makes sense to also document the
>> options here. In fact, to make it makes more sense to document them all here
>> rather than in each individual backend. Of course, no need to go and remove it
>> in each backend, but I would like to have then documented here. Especially since
>> a later patch will also parse them here.
>>
>>> +
>>> main() {
>>> local OPT OPTARG
>>> local backend output hfile recurse quiet
>>> @@ -83,7 +85,10 @@ main() {
>>> # If the backend fails, we can just remove the temporary directory to
>>> # remove all the cruft it may have left behind. Then we just exit in
>>> # error too.
>>> - if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
>>> + if ! "${OLDPWD}/support/download/${backend}" \
>>> + ${quiet} ${recurse} \
>>
>> I don't like it very much when options are mixed like this: either fill lines
>> up completely, or do one option (or option combination, of course, like -o
>> "...") per line. But since this will be changed again later it's not worth
>> changing this patch.
>>
>> Regards,
>> Arnout
>>
>>> + -o "${tmpf}" "${@}"
>>> + then
>>> rm -rf "${tmpd}"
>>> exit 1
>>> fi
>> [snip]
>>
>>
>> --
>> Arnout Vandecappelle arnout at mind be
>> Senior Embedded Software Architect +32-16-286500
>> Essensium/Mind http://www.mind.be
>> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
>> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
>> GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 02/13] core/pkg-download: change all helpers to use common options
2017-07-05 0:38 ` Arnout Vandecappelle
@ 2017-07-05 7:09 ` Yann E. MORIN
0 siblings, 0 replies; 41+ messages in thread
From: Yann E. MORIN @ 2017-07-05 7:09 UTC (permalink / raw)
To: buildroot
Arnout, All,
On 2017-07-05 02:38 +0200, Arnout Vandecappelle spake thusly:
> On 05-07-17 01:14, Yann E. MORIN wrote:
> > Arnout, All,
> >
> > On 2017-07-05 01:09 +0200, Arnout Vandecappelle spake thusly:
> >> On 04-07-17 18:22, Maxime Hadjinlian wrote:
> >>> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >>> diff --git a/support/download/check-hash b/support/download/check-hash
> >>> index c1ff53c02b..bf2dbb1a8b 100755
> >>> --- a/support/download/check-hash
> >>> +++ b/support/download/check-hash
> >>> @@ -19,7 +19,7 @@ set -e
> >>> # 3: the hash file exists and there was no hash to check the file against
> >>> # 4: the hash file exists and at least one hash type is unknown
> >>>
> >>> -while getopts :q OPT; do
> >>> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> >>
> >> This doesn't make sense to me: the global DL_GETOPTS are never passed to
> >> check-hash, and you're not using any of them. So I'd keep the explicit :q here.
> >
> > It took me a bit of time to understand what you meant, because
> > BR_BACKEND_DL_GETOPTS is actually exported by dl-wrapper, and it is
> > dl-wrapper that calls check-hash. So let me rephrase it, and you'll tell
> > me if I understood correctly:
> >
> > This doesn't make sense to me: none of the variabels from
> > BR_BACKEND_DL_GETOPTS are ever passed to check-hash, so [...]
>
> Yes, that's what I meant. I guess I don't formulate very clearly at 01:09 (much
> less at 02:38...)
Oh come on, you woke up at 11 in the morning! Aha! ;-]
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 04/13] pkg-generic: make PKG_DL_DIR equal to DL_DIR
2017-07-04 16:22 ` [Buildroot] [PATCH 04/13] pkg-generic: make PKG_DL_DIR equal to DL_DIR Maxime Hadjinlian
@ 2017-07-05 8:30 ` Arnout Vandecappelle
2017-07-22 21:37 ` Thomas Petazzoni
1 sibling, 0 replies; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-07-05 8:30 UTC (permalink / raw)
To: buildroot
On 04-07-17 18:22, Maxime Hadjinlian wrote:
> This variable is currently not used by anyone.
> The value is changed to match the path to DL_DIR.
>
> The next patch will introduce the use of PKG_DL_DIR for packages that
> use specific EXTRACT_CMDS.
>
> And then we will be able to change the value of PKG_DL_DIR to create a
> new directory structure in DL_DIR with a subdirectory for each package.
Funny, you will later change it to almost the same as what it is now...
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(doesn't depend on the earlier patches in the series).
Regards,
Arnout
> ---
> package/pkg-generic.mk | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 79bea906e2..c1cd41fda5 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -434,7 +434,7 @@ endif
>
> $(2)_BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
> $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
> -$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_BASE_NAME)
> +$(2)_DL_DIR = $$(DL_DIR)
> $(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME)
>
> ifndef $(2)_SUBDIR
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 04/13] pkg-generic: make PKG_DL_DIR equal to DL_DIR
2017-07-04 16:22 ` [Buildroot] [PATCH 04/13] pkg-generic: make PKG_DL_DIR equal to DL_DIR Maxime Hadjinlian
2017-07-05 8:30 ` Arnout Vandecappelle
@ 2017-07-22 21:37 ` Thomas Petazzoni
1 sibling, 0 replies; 41+ messages in thread
From: Thomas Petazzoni @ 2017-07-22 21:37 UTC (permalink / raw)
To: buildroot
Hello,
On Tue, 4 Jul 2017 18:22:02 +0200, Maxime Hadjinlian wrote:
> This variable is currently not used by anyone.
> The value is changed to match the path to DL_DIR.
>
> The next patch will introduce the use of PKG_DL_DIR for packages that
> use specific EXTRACT_CMDS.
>
> And then we will be able to change the value of PKG_DL_DIR to create a
> new directory structure in DL_DIR with a subdirectory for each package.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> package/pkg-generic.mk | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
Applied to master, thanks.
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 05/13] packages: use new $($PKG)_DL_DIR) variable
2017-07-04 16:22 ` [Buildroot] [PATCH 05/13] packages: use new $($PKG)_DL_DIR) variable Maxime Hadjinlian
@ 2017-07-22 21:39 ` Thomas Petazzoni
2017-07-22 21:54 ` Yann E. MORIN
2017-10-23 20:00 ` Arnout Vandecappelle
1 sibling, 1 reply; 41+ messages in thread
From: Thomas Petazzoni @ 2017-07-22 21:39 UTC (permalink / raw)
To: buildroot
Hello,
On Tue, 4 Jul 2017 18:22:03 +0200, Maxime Hadjinlian wrote:
> Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the
> transition into a new directory structure for DL_DIR.
>
> This commit has been generated with the following scripts:
>
> for i in package/*/*.mk; do
> if ! grep -q "\$(DL_DIR)" ${i}; then
> continue
> fi
> pkg_name="$(basename $(dirname ${i}))"
> raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
> pkg_dl_dir="${raw_pkg_name}_DL_DIR"
> sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}
> done
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
I wanted to apply this one, but I believe there is a problem, see below:
> diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
> index b52f9456b3..64fcfc70e7 100644
> --- a/package/gcc/gcc.mk
> +++ b/package/gcc/gcc.mk
> @@ -289,7 +289,7 @@ HOST_GCC_COMMON_MAKE_OPTS = \
> gcc_cv_libc_provides_ssp=$(if $(BR2_TOOLCHAIN_HAS_SSP),yes,no)
>
> ifeq ($(BR2_CCACHE),y)
> -HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE)
> +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(GCC_DL_DIR)/$(GCC_SOURCE)
I'm not sure this one works. Indeed, there is no package named just
"gcc". There are two packages named gcc-initial and gcc-final, but no
package named "gcc", so GCC_DL_DIR doesn't exist. Only
GCC_INITIAL_DL_DIR and GCC_FINAL_DL_DIR exist.
Am I missing something here ?
Thanks!
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 05/13] packages: use new $($PKG)_DL_DIR) variable
2017-07-22 21:39 ` Thomas Petazzoni
@ 2017-07-22 21:54 ` Yann E. MORIN
0 siblings, 0 replies; 41+ messages in thread
From: Yann E. MORIN @ 2017-07-22 21:54 UTC (permalink / raw)
To: buildroot
Thomas, Maxime, All,
On 2017-07-22 23:39 +0200, Thomas Petazzoni spake thusly:
> Hello,
>
> On Tue, 4 Jul 2017 18:22:03 +0200, Maxime Hadjinlian wrote:
> > Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the
> > transition into a new directory structure for DL_DIR.
> >
> > This commit has been generated with the following scripts:
> >
> > for i in package/*/*.mk; do
> > if ! grep -q "\$(DL_DIR)" ${i}; then
> > continue
> > fi
> > pkg_name="$(basename $(dirname ${i}))"
> > raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
> > pkg_dl_dir="${raw_pkg_name}_DL_DIR"
> > sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}
> > done
> >
> > Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
>
> I wanted to apply this one, but I believe there is a problem, see below:
>
> > diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
> > index b52f9456b3..64fcfc70e7 100644
> > --- a/package/gcc/gcc.mk
> > +++ b/package/gcc/gcc.mk
> > @@ -289,7 +289,7 @@ HOST_GCC_COMMON_MAKE_OPTS = \
> > gcc_cv_libc_provides_ssp=$(if $(BR2_TOOLCHAIN_HAS_SSP),yes,no)
> >
> > ifeq ($(BR2_CCACHE),y)
> > -HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE)
> > +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(GCC_DL_DIR)/$(GCC_SOURCE)
>
> I'm not sure this one works. Indeed, there is no package named just
> "gcc". There are two packages named gcc-initial and gcc-final, but no
> package named "gcc", so GCC_DL_DIR doesn't exist. Only
> GCC_INITIAL_DL_DIR and GCC_FINAL_DL_DIR exist.
Indeed, it does not exist;
$ make -s printvars VARS=GCC_DL_DIR
[nothing]
Regards,
Yann E. MORIN.
> Am I missing something here ?
>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
` (12 preceding siblings ...)
2017-07-04 16:22 ` [Buildroot] [PATCH 13/13] download: git: introduce cache feature Maxime Hadjinlian
@ 2017-07-27 22:37 ` Peter Seiderer
2017-07-30 16:32 ` Maxime Hadjinlian
13 siblings, 1 reply; 41+ messages in thread
From: Peter Seiderer @ 2017-07-27 22:37 UTC (permalink / raw)
To: buildroot
Hello Maxime,
On Tue, 4 Jul 2017 18:21:58 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
> Hi all,
>
> This series will progressively reorganize the DL_DIR folder, in order to
> hve a subdirectory for each packages.
> This way, we store patches into subdirectory called by the package name
> instead of flattly into DL_DIR.
> We also avoid possible conflicts between packages.
>
> Finally, this series introduce the 'git cache' feature. What it does is
> that it keeps a copy of the Git clone in the subdirectory of the
> package.
> This way, when you change the version of said package, instead of
> cloning from scratch, we only fetch the missing objects and create the
> tarball from that git clone.
>
Tested your patch series (hoping to speed up RPi kernel/userland/firmware
update which are huge git downloads) but failed with the raspberrypi3_defconfig
while compiling the linux kernel:
DTC arch/arm/boot/dts/bcm2710-rpi-3-b.dtb
In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
from arch/arm/boot/dts/bcm2710-rpi-3-b.dts:3:
arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
#include <dt-bindings/pinctrl/bcm2835.h>
^
compilation terminated.
DTC arch/arm/boot/dts/bcm2710-rpi-cm3.dtb
In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
from arch/arm/boot/dts/bcm2710-rpi-cm3.dts:3:
arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
#include <dt-bindings/pinctrl/bcm2835.h>
^
compilation terminated.
This is due to wrong dt-bindings link, the failing version (with your patchset):
$ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
lrwxrwxrwx 1 seiderer users 79 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> linux-9126e25b0934bd7bd843763310ea4b34c6e139d0./../../../../include/dt-bindings
And from a working version (without your patchset):
$ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
lrwxrwxrwx 1 seiderer users 34 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> ../../../../../include/dt-bindings
Regards,
Peter
> It should speed up the build, especially for some use cases; drawback is
> that the disk usage for the DL_DIR will increase significantly. But disk
> space are pretty cheap, especially on build servers.
>
> Maxime Hadjinlian (12):
> pkg-{download,generic}: remove source-check
> download: put most of the infra in dl-wrapper
> pkg-generic: make PKG_DL_DIR equal to DL_DIR
> packages: use new $($PKG)_DL_DIR) variable
> pkg-{download,generic}: use new $($(PKG)_DL_DIR)
> support/download: make sure the download folder is created
> pkg-generic: add a subdirectory to the DL_DIR
> pkg-download: support new subdir for mirrors
> pkg-generic: introduce _SAME_SOURCE_AS
> help/manual: update help about the new $(LIBFOO_DL_DIR)
> download: add flock call before dl-wrapper
> download: git: introduce cache feature
>
> Yann E. MORIN (1):
> core/pkg-download: change all helpers to use common options
>
> Config.in | 3 +
> docs/manual/adding-packages-generic.txt | 6 +-
> package/amd-catalyst/amd-catalyst.mk | 2 +-
> package/android-tools/android-tools.mk | 2 +-
> package/angularjs/angularjs.mk | 2 +-
> package/bootstrap/bootstrap.mk | 2 +-
> package/cache-calibrator/cache-calibrator.mk | 2 +-
> package/cracklib/cracklib.mk | 2 +-
> package/cryptopp/cryptopp.mk | 2 +-
> package/devmem2/devmem2.mk | 2 +-
> package/dhrystone/dhrystone.mk | 2 +-
> package/doom-wad/doom-wad.mk | 2 +-
> package/espeak/espeak.mk | 2 +-
> package/fan-ctrl/fan-ctrl.mk | 2 +-
> package/gcc/gcc.mk | 2 +-
> package/irrlicht/irrlicht.mk | 2 +-
> package/jquery-mobile/jquery-mobile.mk | 2 +-
> package/jquery-sparkline/jquery-sparkline.mk | 2 +-
> package/jquery-ui-themes/jquery-ui-themes.mk | 2 +-
> package/jquery-ui/jquery-ui.mk | 2 +-
> package/jquery-validation/jquery-validation.mk | 2 +-
> package/jquery/jquery.mk | 2 +-
> package/kodi/kodi.mk | 6 +-
> package/libfreeimage/libfreeimage.mk | 2 +-
> package/libjson/libjson.mk | 2 +-
> package/libsoil/libsoil.mk | 2 +-
> package/lsof/lsof.mk | 2 +-
> package/musl-compat-headers/musl-compat-headers.mk | 2 +-
> package/nmon/nmon.mk | 2 +-
> package/nvidia-driver/nvidia-driver.mk | 2 +-
> package/openobex/openobex.mk | 2 +-
> package/opentyrian-data/opentyrian-data.mk | 2 +-
> package/perl/perl.mk | 2 +-
> package/pkg-download.mk | 216 ++++-----------------
> package/pkg-generic.mk | 10 +-
> package/python-keyring/python-keyring.mk | 2 +-
> .../python-simplegeneric/python-simplegeneric.mk | 2 +-
> package/rapidxml/rapidxml.mk | 2 +-
> package/rpi-wifi-firmware/rpi-wifi-firmware.mk | 2 +-
> package/sam-ba/sam-ba.mk | 2 +-
> package/spidev_test/spidev_test.mk | 2 +-
> package/tar/tar.mk | 2 +-
> package/tesseract-ocr/tesseract-ocr.mk | 2 +-
> package/ti-cgt-pru/ti-cgt-pru.mk | 4 +-
> package/ti-gfx/ti-gfx.mk | 4 +-
> package/ts4900-fpga/ts4900-fpga.mk | 2 +-
> package/unscd/unscd.mk | 2 +-
> package/urg/urg.mk | 2 +-
> package/waf/waf.mk | 2 +-
> package/whetstone/whetstone.mk | 2 +-
> package/wilc1000-firmware/wilc1000-firmware.mk | 2 +-
> package/zynq-boot-bin/zynq-boot-bin.mk | 2 +-
> support/download/bzr | 25 +--
> support/download/check-hash | 2 +-
> support/download/cp | 17 +-
> support/download/cvs | 34 ++--
> support/download/dl-wrapper | 62 +++---
> support/download/git | 96 +++++----
> support/download/hg | 25 +--
> support/download/scp | 19 +-
> support/download/svn | 25 +--
> support/download/wget | 24 ++-
> 62 files changed, 286 insertions(+), 382 deletions(-)
>
> --
> 2.13.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature
2017-07-27 22:37 ` [Buildroot] [PATCH 00/13] New DL_DIR organisation; git " Peter Seiderer
@ 2017-07-30 16:32 ` Maxime Hadjinlian
2017-10-17 19:56 ` Peter Seiderer
0 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-07-30 16:32 UTC (permalink / raw)
To: buildroot
Hi Peter, all
On Fri, Jul 28, 2017 at 12:37 AM, Peter Seiderer <ps.report@gmx.net> wrote:
> Hello Maxime,
>
> On Tue, 4 Jul 2017 18:21:58 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
>
>> Hi all,
>>
>> This series will progressively reorganize the DL_DIR folder, in order to
>> hve a subdirectory for each packages.
>> This way, we store patches into subdirectory called by the package name
>> instead of flattly into DL_DIR.
>> We also avoid possible conflicts between packages.
>>
>> Finally, this series introduce the 'git cache' feature. What it does is
>> that it keeps a copy of the Git clone in the subdirectory of the
>> package.
>> This way, when you change the version of said package, instead of
>> cloning from scratch, we only fetch the missing objects and create the
>> tarball from that git clone.
>>
>
> Tested your patch series (hoping to speed up RPi kernel/userland/firmware
> update which are huge git downloads) but failed with the raspberrypi3_defconfig
> while compiling the linux kernel:
>
> DTC arch/arm/boot/dts/bcm2710-rpi-3-b.dtb
> In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
> from arch/arm/boot/dts/bcm2710-rpi-3-b.dts:3:
> arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
> #include <dt-bindings/pinctrl/bcm2835.h>
> ^
> compilation terminated.
> DTC arch/arm/boot/dts/bcm2710-rpi-cm3.dtb
> In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
> from arch/arm/boot/dts/bcm2710-rpi-cm3.dts:3:
> arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
> #include <dt-bindings/pinctrl/bcm2835.h>
> ^
> compilation terminated.
>
>
> This is due to wrong dt-bindings link, the failing version (with your patchset):
>
> $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
>
> lrwxrwxrwx 1 seiderer users 79 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> linux-9126e25b0934bd7bd843763310ea4b34c6e139d0./../../../../include/dt-bindings
>
> And from a working version (without your patchset):
>
> $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
> lrwxrwxrwx 1 seiderer users 34 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> ../../../../../include/dt-bindings
>
Thanks a lot for your test, made my realise I had a bug when I create
the tarball from the git repo, I use a regex to transform './' into
'$basename' but I forgot to put the '$' in my regex, hence the '../'
becomming 'linux-sha1./'
It's fixed on my branch on github for now.
> Regards,
>
> Peter
>
>
>> It should speed up the build, especially for some use cases; drawback is
>> that the disk usage for the DL_DIR will increase significantly. But disk
>> space are pretty cheap, especially on build servers.
>>
>> Maxime Hadjinlian (12):
>> pkg-{download,generic}: remove source-check
>> download: put most of the infra in dl-wrapper
>> pkg-generic: make PKG_DL_DIR equal to DL_DIR
>> packages: use new $($PKG)_DL_DIR) variable
>> pkg-{download,generic}: use new $($(PKG)_DL_DIR)
>> support/download: make sure the download folder is created
>> pkg-generic: add a subdirectory to the DL_DIR
>> pkg-download: support new subdir for mirrors
>> pkg-generic: introduce _SAME_SOURCE_AS
>> help/manual: update help about the new $(LIBFOO_DL_DIR)
>> download: add flock call before dl-wrapper
>> download: git: introduce cache feature
>>
>> Yann E. MORIN (1):
>> core/pkg-download: change all helpers to use common options
>>
>> Config.in | 3 +
>> docs/manual/adding-packages-generic.txt | 6 +-
>> package/amd-catalyst/amd-catalyst.mk | 2 +-
>> package/android-tools/android-tools.mk | 2 +-
>> package/angularjs/angularjs.mk | 2 +-
>> package/bootstrap/bootstrap.mk | 2 +-
>> package/cache-calibrator/cache-calibrator.mk | 2 +-
>> package/cracklib/cracklib.mk | 2 +-
>> package/cryptopp/cryptopp.mk | 2 +-
>> package/devmem2/devmem2.mk | 2 +-
>> package/dhrystone/dhrystone.mk | 2 +-
>> package/doom-wad/doom-wad.mk | 2 +-
>> package/espeak/espeak.mk | 2 +-
>> package/fan-ctrl/fan-ctrl.mk | 2 +-
>> package/gcc/gcc.mk | 2 +-
>> package/irrlicht/irrlicht.mk | 2 +-
>> package/jquery-mobile/jquery-mobile.mk | 2 +-
>> package/jquery-sparkline/jquery-sparkline.mk | 2 +-
>> package/jquery-ui-themes/jquery-ui-themes.mk | 2 +-
>> package/jquery-ui/jquery-ui.mk | 2 +-
>> package/jquery-validation/jquery-validation.mk | 2 +-
>> package/jquery/jquery.mk | 2 +-
>> package/kodi/kodi.mk | 6 +-
>> package/libfreeimage/libfreeimage.mk | 2 +-
>> package/libjson/libjson.mk | 2 +-
>> package/libsoil/libsoil.mk | 2 +-
>> package/lsof/lsof.mk | 2 +-
>> package/musl-compat-headers/musl-compat-headers.mk | 2 +-
>> package/nmon/nmon.mk | 2 +-
>> package/nvidia-driver/nvidia-driver.mk | 2 +-
>> package/openobex/openobex.mk | 2 +-
>> package/opentyrian-data/opentyrian-data.mk | 2 +-
>> package/perl/perl.mk | 2 +-
>> package/pkg-download.mk | 216 ++++-----------------
>> package/pkg-generic.mk | 10 +-
>> package/python-keyring/python-keyring.mk | 2 +-
>> .../python-simplegeneric/python-simplegeneric.mk | 2 +-
>> package/rapidxml/rapidxml.mk | 2 +-
>> package/rpi-wifi-firmware/rpi-wifi-firmware.mk | 2 +-
>> package/sam-ba/sam-ba.mk | 2 +-
>> package/spidev_test/spidev_test.mk | 2 +-
>> package/tar/tar.mk | 2 +-
>> package/tesseract-ocr/tesseract-ocr.mk | 2 +-
>> package/ti-cgt-pru/ti-cgt-pru.mk | 4 +-
>> package/ti-gfx/ti-gfx.mk | 4 +-
>> package/ts4900-fpga/ts4900-fpga.mk | 2 +-
>> package/unscd/unscd.mk | 2 +-
>> package/urg/urg.mk | 2 +-
>> package/waf/waf.mk | 2 +-
>> package/whetstone/whetstone.mk | 2 +-
>> package/wilc1000-firmware/wilc1000-firmware.mk | 2 +-
>> package/zynq-boot-bin/zynq-boot-bin.mk | 2 +-
>> support/download/bzr | 25 +--
>> support/download/check-hash | 2 +-
>> support/download/cp | 17 +-
>> support/download/cvs | 34 ++--
>> support/download/dl-wrapper | 62 +++---
>> support/download/git | 96 +++++----
>> support/download/hg | 25 +--
>> support/download/scp | 19 +-
>> support/download/svn | 25 +--
>> support/download/wget | 24 ++-
>> 62 files changed, 286 insertions(+), 382 deletions(-)
>>
>> --
>> 2.13.2
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
>
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature
2017-07-30 16:32 ` Maxime Hadjinlian
@ 2017-10-17 19:56 ` Peter Seiderer
2017-10-23 18:24 ` Maxime Hadjinlian
0 siblings, 1 reply; 41+ messages in thread
From: Peter Seiderer @ 2017-10-17 19:56 UTC (permalink / raw)
To: buildroot
Hello Maxime,
On Sun, 30 Jul 2017 18:32:27 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
> Hi Peter, all
>
> On Fri, Jul 28, 2017 at 12:37 AM, Peter Seiderer <ps.report@gmx.net> wrote:
> > Hello Maxime,
> >
> > On Tue, 4 Jul 2017 18:21:58 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> This series will progressively reorganize the DL_DIR folder, in order to
> >> hve a subdirectory for each packages.
> >> This way, we store patches into subdirectory called by the package name
> >> instead of flattly into DL_DIR.
> >> We also avoid possible conflicts between packages.
> >>
> >> Finally, this series introduce the 'git cache' feature. What it does is
> >> that it keeps a copy of the Git clone in the subdirectory of the
> >> package.
> >> This way, when you change the version of said package, instead of
> >> cloning from scratch, we only fetch the missing objects and create the
> >> tarball from that git clone.
> >>
> >
> > Tested your patch series (hoping to speed up RPi kernel/userland/firmware
> > update which are huge git downloads) but failed with the raspberrypi3_defconfig
> > while compiling the linux kernel:
> >
> > DTC arch/arm/boot/dts/bcm2710-rpi-3-b.dtb
> > In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
> > from arch/arm/boot/dts/bcm2710-rpi-3-b.dts:3:
> > arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
> > #include <dt-bindings/pinctrl/bcm2835.h>
> > ^
> > compilation terminated.
> > DTC arch/arm/boot/dts/bcm2710-rpi-cm3.dtb
> > In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
> > from arch/arm/boot/dts/bcm2710-rpi-cm3.dts:3:
> > arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
> > #include <dt-bindings/pinctrl/bcm2835.h>
> > ^
> > compilation terminated.
> >
> >
> > This is due to wrong dt-bindings link, the failing version (with your patchset):
> >
> > $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
> >
> > lrwxrwxrwx 1 seiderer users 79 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> linux-9126e25b0934bd7bd843763310ea4b34c6e139d0./../../../../include/dt-bindings
> >
> > And from a working version (without your patchset):
> >
> > $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
> > lrwxrwxrwx 1 seiderer users 34 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> ../../../../../include/dt-bindings
> >
> Thanks a lot for your test, made my realise I had a bug when I create
> the tarball from the git repo, I use a regex to transform './' into
> '$basename' but I forgot to put the '$' in my regex, hence the '../'
> becomming 'linux-sha1./'
>
> It's fixed on my branch on github for now.
Any updates on this one? Mind to resend an actual version (would like to
see this feature integrated into buildroot, tired of waiting for hours
in case RaspberryPi kernel/firmware/userland git versions are updated)?
Regards,
Peter
> > Regards,
> >
> > Peter
> >
> >
> >> It should speed up the build, especially for some use cases; drawback is
> >> that the disk usage for the DL_DIR will increase significantly. But disk
> >> space are pretty cheap, especially on build servers.
> >>
> >> Maxime Hadjinlian (12):
> >> pkg-{download,generic}: remove source-check
> >> download: put most of the infra in dl-wrapper
> >> pkg-generic: make PKG_DL_DIR equal to DL_DIR
> >> packages: use new $($PKG)_DL_DIR) variable
> >> pkg-{download,generic}: use new $($(PKG)_DL_DIR)
> >> support/download: make sure the download folder is created
> >> pkg-generic: add a subdirectory to the DL_DIR
> >> pkg-download: support new subdir for mirrors
> >> pkg-generic: introduce _SAME_SOURCE_AS
> >> help/manual: update help about the new $(LIBFOO_DL_DIR)
> >> download: add flock call before dl-wrapper
> >> download: git: introduce cache feature
> >>
> >> Yann E. MORIN (1):
> >> core/pkg-download: change all helpers to use common options
> >>
> >> Config.in | 3 +
> >> docs/manual/adding-packages-generic.txt | 6 +-
> >> package/amd-catalyst/amd-catalyst.mk | 2 +-
> >> package/android-tools/android-tools.mk | 2 +-
> >> package/angularjs/angularjs.mk | 2 +-
> >> package/bootstrap/bootstrap.mk | 2 +-
> >> package/cache-calibrator/cache-calibrator.mk | 2 +-
> >> package/cracklib/cracklib.mk | 2 +-
> >> package/cryptopp/cryptopp.mk | 2 +-
> >> package/devmem2/devmem2.mk | 2 +-
> >> package/dhrystone/dhrystone.mk | 2 +-
> >> package/doom-wad/doom-wad.mk | 2 +-
> >> package/espeak/espeak.mk | 2 +-
> >> package/fan-ctrl/fan-ctrl.mk | 2 +-
> >> package/gcc/gcc.mk | 2 +-
> >> package/irrlicht/irrlicht.mk | 2 +-
> >> package/jquery-mobile/jquery-mobile.mk | 2 +-
> >> package/jquery-sparkline/jquery-sparkline.mk | 2 +-
> >> package/jquery-ui-themes/jquery-ui-themes.mk | 2 +-
> >> package/jquery-ui/jquery-ui.mk | 2 +-
> >> package/jquery-validation/jquery-validation.mk | 2 +-
> >> package/jquery/jquery.mk | 2 +-
> >> package/kodi/kodi.mk | 6 +-
> >> package/libfreeimage/libfreeimage.mk | 2 +-
> >> package/libjson/libjson.mk | 2 +-
> >> package/libsoil/libsoil.mk | 2 +-
> >> package/lsof/lsof.mk | 2 +-
> >> package/musl-compat-headers/musl-compat-headers.mk | 2 +-
> >> package/nmon/nmon.mk | 2 +-
> >> package/nvidia-driver/nvidia-driver.mk | 2 +-
> >> package/openobex/openobex.mk | 2 +-
> >> package/opentyrian-data/opentyrian-data.mk | 2 +-
> >> package/perl/perl.mk | 2 +-
> >> package/pkg-download.mk | 216 ++++-----------------
> >> package/pkg-generic.mk | 10 +-
> >> package/python-keyring/python-keyring.mk | 2 +-
> >> .../python-simplegeneric/python-simplegeneric.mk | 2 +-
> >> package/rapidxml/rapidxml.mk | 2 +-
> >> package/rpi-wifi-firmware/rpi-wifi-firmware.mk | 2 +-
> >> package/sam-ba/sam-ba.mk | 2 +-
> >> package/spidev_test/spidev_test.mk | 2 +-
> >> package/tar/tar.mk | 2 +-
> >> package/tesseract-ocr/tesseract-ocr.mk | 2 +-
> >> package/ti-cgt-pru/ti-cgt-pru.mk | 4 +-
> >> package/ti-gfx/ti-gfx.mk | 4 +-
> >> package/ts4900-fpga/ts4900-fpga.mk | 2 +-
> >> package/unscd/unscd.mk | 2 +-
> >> package/urg/urg.mk | 2 +-
> >> package/waf/waf.mk | 2 +-
> >> package/whetstone/whetstone.mk | 2 +-
> >> package/wilc1000-firmware/wilc1000-firmware.mk | 2 +-
> >> package/zynq-boot-bin/zynq-boot-bin.mk | 2 +-
> >> support/download/bzr | 25 +--
> >> support/download/check-hash | 2 +-
> >> support/download/cp | 17 +-
> >> support/download/cvs | 34 ++--
> >> support/download/dl-wrapper | 62 +++---
> >> support/download/git | 96 +++++----
> >> support/download/hg | 25 +--
> >> support/download/scp | 19 +-
> >> support/download/svn | 25 +--
> >> support/download/wget | 24 ++-
> >> 62 files changed, 286 insertions(+), 382 deletions(-)
> >>
> >> --
> >> 2.13.2
> >>
> >> _______________________________________________
> >> buildroot mailing list
> >> buildroot at busybox.net
> >> http://lists.busybox.net/mailman/listinfo/buildroot
> >
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature
2017-10-17 19:56 ` Peter Seiderer
@ 2017-10-23 18:24 ` Maxime Hadjinlian
2017-10-23 18:42 ` Arnout Vandecappelle
0 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-10-23 18:24 UTC (permalink / raw)
To: buildroot
Hi Peter,
On Tue, Oct 17, 2017 at 9:56 PM, Peter Seiderer <ps.report@gmx.net> wrote:
> Hello Maxime,
>
> On Sun, 30 Jul 2017 18:32:27 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
>
>> Hi Peter, all
>>
>> On Fri, Jul 28, 2017 at 12:37 AM, Peter Seiderer <ps.report@gmx.net> wrote:
>> > Hello Maxime,
>> >
>> > On Tue, 4 Jul 2017 18:21:58 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
>> >
>> >> Hi all,
>> >>
>> >> This series will progressively reorganize the DL_DIR folder, in order to
>> >> hve a subdirectory for each packages.
>> >> This way, we store patches into subdirectory called by the package name
>> >> instead of flattly into DL_DIR.
>> >> We also avoid possible conflicts between packages.
>> >>
>> >> Finally, this series introduce the 'git cache' feature. What it does is
>> >> that it keeps a copy of the Git clone in the subdirectory of the
>> >> package.
>> >> This way, when you change the version of said package, instead of
>> >> cloning from scratch, we only fetch the missing objects and create the
>> >> tarball from that git clone.
>> >>
>> >
>> > Tested your patch series (hoping to speed up RPi kernel/userland/firmware
>> > update which are huge git downloads) but failed with the raspberrypi3_defconfig
>> > while compiling the linux kernel:
>> >
>> > DTC arch/arm/boot/dts/bcm2710-rpi-3-b.dtb
>> > In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
>> > from arch/arm/boot/dts/bcm2710-rpi-3-b.dts:3:
>> > arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
>> > #include <dt-bindings/pinctrl/bcm2835.h>
>> > ^
>> > compilation terminated.
>> > DTC arch/arm/boot/dts/bcm2710-rpi-cm3.dtb
>> > In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
>> > from arch/arm/boot/dts/bcm2710-rpi-cm3.dts:3:
>> > arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
>> > #include <dt-bindings/pinctrl/bcm2835.h>
>> > ^
>> > compilation terminated.
>> >
>> >
>> > This is due to wrong dt-bindings link, the failing version (with your patchset):
>> >
>> > $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
>> >
>> > lrwxrwxrwx 1 seiderer users 79 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> linux-9126e25b0934bd7bd843763310ea4b34c6e139d0./../../../../include/dt-bindings
>> >
>> > And from a working version (without your patchset):
>> >
>> > $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
>> > lrwxrwxrwx 1 seiderer users 34 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> ../../../../../include/dt-bindings
>> >
>> Thanks a lot for your test, made my realise I had a bug when I create
>> the tarball from the git repo, I use a regex to transform './' into
>> '$basename' but I forgot to put the '$' in my regex, hence the '../'
>> becomming 'linux-sha1./'
>>
>> It's fixed on my branch on github for now.
>
> Any updates on this one? Mind to resend an actual version (would like to
> see this feature integrated into buildroot, tired of waiting for hours
> in case RaspberryPi kernel/firmware/userland git versions are updated)?
I think the most "up-to-date" is my branch on Github
(https://github.com/maximeh/buildroot/tree/git_cache).
I don't really think it's ready for a review, there's still many bugs
in there, at least that's what I remember.
What would really help me is, if you could test this and tell me what
breaks so I could look it up and try to fix this, this would be a huge
help !
If you come up with patches, that's really nice too ;).
I won't make any promises on the time I can spend on this, we all know
how this goes.. but that being said, I am still really motivated by
this feature, I really need it too.
Sorry for the time it takes :/.
>
> Regards,
> Peter
>
>> > Regards,
>> >
>> > Peter
>> >
>> >
>> >> It should speed up the build, especially for some use cases; drawback is
>> >> that the disk usage for the DL_DIR will increase significantly. But disk
>> >> space are pretty cheap, especially on build servers.
>> >>
>> >> Maxime Hadjinlian (12):
>> >> pkg-{download,generic}: remove source-check
>> >> download: put most of the infra in dl-wrapper
>> >> pkg-generic: make PKG_DL_DIR equal to DL_DIR
>> >> packages: use new $($PKG)_DL_DIR) variable
>> >> pkg-{download,generic}: use new $($(PKG)_DL_DIR)
>> >> support/download: make sure the download folder is created
>> >> pkg-generic: add a subdirectory to the DL_DIR
>> >> pkg-download: support new subdir for mirrors
>> >> pkg-generic: introduce _SAME_SOURCE_AS
>> >> help/manual: update help about the new $(LIBFOO_DL_DIR)
>> >> download: add flock call before dl-wrapper
>> >> download: git: introduce cache feature
>> >>
>> >> Yann E. MORIN (1):
>> >> core/pkg-download: change all helpers to use common options
>> >>
>> >> Config.in | 3 +
>> >> docs/manual/adding-packages-generic.txt | 6 +-
>> >> package/amd-catalyst/amd-catalyst.mk | 2 +-
>> >> package/android-tools/android-tools.mk | 2 +-
>> >> package/angularjs/angularjs.mk | 2 +-
>> >> package/bootstrap/bootstrap.mk | 2 +-
>> >> package/cache-calibrator/cache-calibrator.mk | 2 +-
>> >> package/cracklib/cracklib.mk | 2 +-
>> >> package/cryptopp/cryptopp.mk | 2 +-
>> >> package/devmem2/devmem2.mk | 2 +-
>> >> package/dhrystone/dhrystone.mk | 2 +-
>> >> package/doom-wad/doom-wad.mk | 2 +-
>> >> package/espeak/espeak.mk | 2 +-
>> >> package/fan-ctrl/fan-ctrl.mk | 2 +-
>> >> package/gcc/gcc.mk | 2 +-
>> >> package/irrlicht/irrlicht.mk | 2 +-
>> >> package/jquery-mobile/jquery-mobile.mk | 2 +-
>> >> package/jquery-sparkline/jquery-sparkline.mk | 2 +-
>> >> package/jquery-ui-themes/jquery-ui-themes.mk | 2 +-
>> >> package/jquery-ui/jquery-ui.mk | 2 +-
>> >> package/jquery-validation/jquery-validation.mk | 2 +-
>> >> package/jquery/jquery.mk | 2 +-
>> >> package/kodi/kodi.mk | 6 +-
>> >> package/libfreeimage/libfreeimage.mk | 2 +-
>> >> package/libjson/libjson.mk | 2 +-
>> >> package/libsoil/libsoil.mk | 2 +-
>> >> package/lsof/lsof.mk | 2 +-
>> >> package/musl-compat-headers/musl-compat-headers.mk | 2 +-
>> >> package/nmon/nmon.mk | 2 +-
>> >> package/nvidia-driver/nvidia-driver.mk | 2 +-
>> >> package/openobex/openobex.mk | 2 +-
>> >> package/opentyrian-data/opentyrian-data.mk | 2 +-
>> >> package/perl/perl.mk | 2 +-
>> >> package/pkg-download.mk | 216 ++++-----------------
>> >> package/pkg-generic.mk | 10 +-
>> >> package/python-keyring/python-keyring.mk | 2 +-
>> >> .../python-simplegeneric/python-simplegeneric.mk | 2 +-
>> >> package/rapidxml/rapidxml.mk | 2 +-
>> >> package/rpi-wifi-firmware/rpi-wifi-firmware.mk | 2 +-
>> >> package/sam-ba/sam-ba.mk | 2 +-
>> >> package/spidev_test/spidev_test.mk | 2 +-
>> >> package/tar/tar.mk | 2 +-
>> >> package/tesseract-ocr/tesseract-ocr.mk | 2 +-
>> >> package/ti-cgt-pru/ti-cgt-pru.mk | 4 +-
>> >> package/ti-gfx/ti-gfx.mk | 4 +-
>> >> package/ts4900-fpga/ts4900-fpga.mk | 2 +-
>> >> package/unscd/unscd.mk | 2 +-
>> >> package/urg/urg.mk | 2 +-
>> >> package/waf/waf.mk | 2 +-
>> >> package/whetstone/whetstone.mk | 2 +-
>> >> package/wilc1000-firmware/wilc1000-firmware.mk | 2 +-
>> >> package/zynq-boot-bin/zynq-boot-bin.mk | 2 +-
>> >> support/download/bzr | 25 +--
>> >> support/download/check-hash | 2 +-
>> >> support/download/cp | 17 +-
>> >> support/download/cvs | 34 ++--
>> >> support/download/dl-wrapper | 62 +++---
>> >> support/download/git | 96 +++++----
>> >> support/download/hg | 25 +--
>> >> support/download/scp | 19 +-
>> >> support/download/svn | 25 +--
>> >> support/download/wget | 24 ++-
>> >> 62 files changed, 286 insertions(+), 382 deletions(-)
>> >>
>> >> --
>> >> 2.13.2
>> >>
>> >> _______________________________________________
>> >> buildroot mailing list
>> >> buildroot at busybox.net
>> >> http://lists.busybox.net/mailman/listinfo/buildroot
>> >
>
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature
2017-10-23 18:24 ` Maxime Hadjinlian
@ 2017-10-23 18:42 ` Arnout Vandecappelle
2017-10-23 19:03 ` Maxime Hadjinlian
0 siblings, 1 reply; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-10-23 18:42 UTC (permalink / raw)
To: buildroot
On 23-10-17 20:24, Maxime Hadjinlian wrote:
> Hi Peter,
>
> On Tue, Oct 17, 2017 at 9:56 PM, Peter Seiderer <ps.report@gmx.net> wrote:
>> Hello Maxime,
>>
>> On Sun, 30 Jul 2017 18:32:27 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
>>
[snip]
>>> It's fixed on my branch on github for now.
>> Any updates on this one? Mind to resend an actual version (would like to
>> see this feature integrated into buildroot, tired of waiting for hours
>> in case RaspberryPi kernel/firmware/userland git versions are updated)?
> I think the most "up-to-date" is my branch on Github
> (https://github.com/maximeh/buildroot/tree/git_cache).
> I don't really think it's ready for a review, there's still many bugs
> in there, at least that's what I remember.
> What would really help me is, if you could test this and tell me what
> breaks so I could look it up and try to fix this, this would be a huge
> help !
> If you come up with patches, that's really nice too ;).
>
> I won't make any promises on the time I can spend on this, we all know
> how this goes.. but that being said, I am still really motivated by
> this feature, I really need it too.
Maxime, correct me if I'm wrong, but you'd be happy if Peter does a respin of
the series, right?
It would be nice if we could already apply the first few patches, just to keep
things going.
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature
2017-10-23 18:42 ` Arnout Vandecappelle
@ 2017-10-23 19:03 ` Maxime Hadjinlian
2017-10-25 20:02 ` Peter Seiderer
0 siblings, 1 reply; 41+ messages in thread
From: Maxime Hadjinlian @ 2017-10-23 19:03 UTC (permalink / raw)
To: buildroot
On Mon, Oct 23, 2017 at 8:42 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
> On 23-10-17 20:24, Maxime Hadjinlian wrote:
>> Hi Peter,
>>
>> On Tue, Oct 17, 2017 at 9:56 PM, Peter Seiderer <ps.report@gmx.net> wrote:
>>> Hello Maxime,
>>>
>>> On Sun, 30 Jul 2017 18:32:27 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
>>>
> [snip]
>>>> It's fixed on my branch on github for now.
>>> Any updates on this one? Mind to resend an actual version (would like to
>>> see this feature integrated into buildroot, tired of waiting for hours
>>> in case RaspberryPi kernel/firmware/userland git versions are updated)?
>> I think the most "up-to-date" is my branch on Github
>> (https://github.com/maximeh/buildroot/tree/git_cache).
>> I don't really think it's ready for a review, there's still many bugs
>> in there, at least that's what I remember.
>> What would really help me is, if you could test this and tell me what
>> breaks so I could look it up and try to fix this, this would be a huge
>> help !
>> If you come up with patches, that's really nice too ;).
>>
>> I won't make any promises on the time I can spend on this, we all know
>> how this goes.. but that being said, I am still really motivated by
>> this feature, I really need it too.
>
> Maxime, correct me if I'm wrong, but you'd be happy if Peter does a respin of
> the series, right?
Oh sure, for me, the important part is that it's merged, who gets to
do is not. Since I clearly can't make the time to make it happen; if
someone can respin this, then please do it :). At least the topic is
out there :).
>
> It would be nice if we could already apply the first few patches, just to keep
> things going.
Agreed.
>
> Regards,
> Arnout
>
> --
> Arnout Vandecappelle arnout at mind be
> Senior Embedded Software Architect +32-16-286500
> Essensium/Mind http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 08/13] pkg-generic: add a subdirectory to the DL_DIR
2017-07-04 16:22 ` [Buildroot] [PATCH 08/13] pkg-generic: add a subdirectory to the DL_DIR Maxime Hadjinlian
@ 2017-10-23 19:42 ` Arnout Vandecappelle
0 siblings, 0 replies; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-10-23 19:42 UTC (permalink / raw)
To: buildroot
On 04-07-17 18:22, Maxime Hadjinlian wrote:
> With all the previous changes, we are now ready to add a subdirectory to
> the DL_DIR.
> The structure will now be DL_DIR/PKG_NAME/{FILE1,FILE2}
>
> This is needed for multiple reasons:
> - Avoid patches with name like SHA1.patch laying flat in DL_DIR,
names
> which makes it hard to know to which packages they apply
> - Avoid possible collisions if two releases have the same name
> (e.g: v01.tar)
> - Allow the possibility to handle a git cache per package in the
> newly created subdirectory.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
> package/pkg-generic.mk | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 5d169b20d6..7364551471 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -434,7 +434,7 @@ endif
>
> $(2)_BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
> $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
> -$(2)_DL_DIR = $$(DL_DIR)
> +$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
> $(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME)
>
> ifndef $(2)_SUBDIR
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 09/13] pkg-download: support new subdir for mirrors
2017-07-04 16:22 ` [Buildroot] [PATCH 09/13] pkg-download: support new subdir for mirrors Maxime Hadjinlian
@ 2017-10-23 19:49 ` Arnout Vandecappelle
2017-10-27 18:14 ` Peter Seiderer
0 siblings, 1 reply; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-10-23 19:49 UTC (permalink / raw)
To: buildroot
On 04-07-17 18:22, Maxime Hadjinlian wrote:
> Since we introduced subdirectory to the DL_DIR, we need to support them
subdirectories
> in the PRIMARY and BACKUP mirror as they evolve to the new tree
> structure.
>
> We check first the new URI (with the subdir), and in case of failure, we
> go check without.
> By checking both URIs, we ensure that old mirror are usable.
mirrors
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.co> ---
> package/pkg-download.mk | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 9a455dc79b..47671e21a7 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -75,6 +75,7 @@ export BR_NO_CHECK_HASH_FOR =
>
> ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
> DOWNLOAD_URIS += \
> + -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))/$(notdir $($(PKG)_DL_DIR)) \
I was thinking, it is a bit silly to first calculate that path and then
applying notdir to it, but I guess this is more robust to a change of PKG_DL_DIR.
There is just one thing to check: I think that this DOWNLOAD_URIS may also be
used outside the context of a package, in which case $(PKG) is not defined. It's
not terrible because then we just go to PRIMARY_SITE without subdir, but it's a
pity to do things twice. So maybe better
$(if $(PKG),$(call getschemeplusuri,$(BR2_PRIMARY_SITE))/$(notdir
$($(PKG)_DL_DIR))) \
Oh yeah, completely easy to read :-)
Regards,
Arnout
> -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
> endif
>
> @@ -83,6 +84,7 @@ DOWNLOAD_URIS += \
> -u $($(PKG)_SITE_METHOD)+$(dir $(1))
> ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
> DOWNLOAD_URIS += \
> + -u $(call getschemeplusuri,$(BR2_BACKUP_SITE))/$(notdir $($(PKG)_DL_DIR)) \
> -u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
> endif
> endif
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 10/13] pkg-generic: introduce _SAME_SOURCE_AS
2017-07-04 16:22 ` [Buildroot] [PATCH 10/13] pkg-generic: introduce _SAME_SOURCE_AS Maxime Hadjinlian
@ 2017-10-23 19:55 ` Arnout Vandecappelle
2017-10-23 23:09 ` Yann E. MORIN
0 siblings, 1 reply; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-10-23 19:55 UTC (permalink / raw)
To: buildroot
On 04-07-17 18:22, Maxime Hadjinlian wrote:
> This per package variable can be used to specify that a package shares
> the same sources as another package.
>
> The use case here is for example, you have linux-headers and linux, you
> don't want to download twice the kernel (because of the introduction of
> the subdirs in DL_DIR).
>
> So you want to specify in linux-headers, this variable, so it will use
> the linux's source.
That patch at least should be part of this series. We *really* don't want to
download linux twice.
However, Yann is reviving the package out-of-tree build series, which IIRC will
already do something similar but then reusing the source directory rather than
the source tarball. Therefore, this patch may no longer be needed. Well, on
condition that that series gets applied :-)
Regards,
Arnout
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> package/pkg-generic.mk | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 7364551471..a8e0686068 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -434,7 +434,9 @@ endif
>
> $(2)_BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
> $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
> -$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
> +$(2)_DL_DIR = $$(if $$($(2)_SAME_SOURCE_AS), \
> + $$(DL_DIR)/$$($$(call UPPERCASE,$$($(2)_SAME_SOURCE_AS))_RAWNAME), \
> + $$(DL_DIR)/$$($(2)_RAWNAME))
> $(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME)
>
> ifndef $(2)_SUBDIR
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 05/13] packages: use new $($PKG)_DL_DIR) variable
2017-07-04 16:22 ` [Buildroot] [PATCH 05/13] packages: use new $($PKG)_DL_DIR) variable Maxime Hadjinlian
2017-07-22 21:39 ` Thomas Petazzoni
@ 2017-10-23 20:00 ` Arnout Vandecappelle
1 sibling, 0 replies; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-10-23 20:00 UTC (permalink / raw)
To: buildroot
On 04-07-17 18:22, Maxime Hadjinlian wrote:
> Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the
> transition into a new directory structure for DL_DIR.
>
> This commit has been generated with the following scripts:
>
> for i in package/*/*.mk; do
> if ! grep -q "\$(DL_DIR)" ${i}; then
> continue
> fi
> pkg_name="$(basename $(dirname ${i}))"
> raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
> pkg_dl_dir="${raw_pkg_name}_DL_DIR"
> sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}
> done
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
[snip]
> 48 files changed, 52 insertions(+), 52 deletions(-)
This patch could be a lot smaller if we first had a patch that adds support for
unzip to suitable-extractor...
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR)
2017-07-04 16:22 ` [Buildroot] [PATCH 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Maxime Hadjinlian
@ 2017-10-23 20:04 ` Arnout Vandecappelle
0 siblings, 0 replies; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-10-23 20:04 UTC (permalink / raw)
To: buildroot
On 04-07-17 18:22, Maxime Hadjinlian wrote:
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> Config.in | 3 +++
> docs/manual/adding-packages-generic.txt | 6 +++---
> 2 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/Config.in b/Config.in
> index 72ceadfde6..0ad43679f3 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -195,6 +195,9 @@ config BR2_DL_DIR
> If the Linux shell environment has defined the BR2_DL_DIR
> environment variable, then this overrides this configuration
> item.
> + The directory is organized with a subdirectory for each package.
> + Each package has its own $(LIBFOO_DL_DIR) variable that can be used
> + to find the correct path.
>
> The default is $(TOPDIR)/dl
>
> diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
> index 12cf1605d6..6e3b503f6b 100644
> --- a/docs/manual/adding-packages-generic.txt
> +++ b/docs/manual/adding-packages-generic.txt
> @@ -263,7 +263,7 @@ information is (assuming the package name is +libfoo+) :
> the file using this URL. Otherwise, Buildroot will assume the file
> to be downloaded is located at +LIBFOO_SITE+. Buildroot will not do
> anything with those additional files, except download them: it will
> - be up to the package recipe to use them from +$(DL_DIR)+.
> + be up to the package recipe to use them from +$(LIBFOO_DL_DIR)+.
>
> * +LIBFOO_SITE_METHOD+ determines the method used to fetch or copy the
> package source code. In many cases, Buildroot guesses the method
> @@ -537,8 +537,8 @@ In the action definitions, you can use the following variables:
> * +$(@D)+, which contains the directory in which the package source
> code has been uncompressed.
>
> -* +$(DL_DIR)+ contains the path to the directory where all the downloads made
> - by Buildroot are stored.
> +* +$(LIBFOO_DL_DIR)+ contains the path to the directory where all the downloads
> + made by Buildroot are stored.
It is no longer all the downloads, it is the downloads of this package.
Also, we'll need to update docs/manual/download-location.txt. And we'll need to
say something in docs/manual/migrating.txt and a big warning in the changelog to
explain that people have to migrate their download dir. I believe it should be
possible to use the old DL_DIR as a PRIMARY_SITE somehow, so that should be
explained.
Regards,
Arnout
>
> * +$(TARGET_CC)+, +$(TARGET_LD)+, etc. to get the target
> cross-compilation utilities
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 12/13] download: add flock call before dl-wrapper
2017-07-04 16:22 ` [Buildroot] [PATCH 12/13] download: add flock call before dl-wrapper Maxime Hadjinlian
@ 2017-10-23 20:17 ` Arnout Vandecappelle
0 siblings, 0 replies; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-10-23 20:17 UTC (permalink / raw)
To: buildroot
On 04-07-17 18:22, Maxime Hadjinlian wrote:
> In order to introduce the cache mechanisms, we need to have a lock on
> the $(LIBFOO_DL_DIR), otherwise it would be impossible to do parallel
> download (a shared DL_DIR for two buildroot instances).
Note that this is also needed for top-level parallel build.
>
> To make sure the directory exists, the mkdir call has been removed from
> the dl-wrapper and put in the infrastructure.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com> ---
> package/pkg-download.mk | 4 +++-
> support/download/dl-wrapper | 1 -
> 2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 47671e21a7..c67f1ba4e1 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -19,6 +19,7 @@ SSH := $(call qstrip,$(BR2_SSH))
> export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
>
> DL_WRAPPER = support/download/dl-wrapper
> +FLOCK = flock $($(PKG)_DL_DIR)/
IMO adding a variable for this makes things less clear.
Also, I believe it is possible that DOWNLOAD gets called outside of package
context. Therefore, it should be $(if $(PKG),$($(PKG)_DL_DIR),$(DL_DIR))
Regards,
Arnout
>
> # DL_DIR may have been set already from the environment
> ifeq ($(origin DL_DIR),undefined)
> @@ -91,7 +92,8 @@ endif
>
> define DOWNLOAD
> $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
> - $(EXTRA_ENV) $(DL_WRAPPER) \
> + $(Q)mkdir -p $($(PKG)_DL_DIR)/
> + $(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
> -c $($(PKG)_DL_VERSION) \
> -f $(notdir $(1)) \
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index f14a60c173..50c14a2e16 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -50,7 +50,6 @@ main() {
> if [ -z "${output}" ]; then
> error "no output specified, use -o\n"
> fi
> - mkdir -p "$(dirname "${output}")"
>
> # If the output file already exists and:
> # - there's no .hash file: do not download it again and exit promptly
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 10/13] pkg-generic: introduce _SAME_SOURCE_AS
2017-10-23 19:55 ` Arnout Vandecappelle
@ 2017-10-23 23:09 ` Yann E. MORIN
0 siblings, 0 replies; 41+ messages in thread
From: Yann E. MORIN @ 2017-10-23 23:09 UTC (permalink / raw)
To: buildroot
Arnout, Maxime, All,
On 2017-10-23 21:55 +0200, Arnout Vandecappelle spake thusly:
>
>
> On 04-07-17 18:22, Maxime Hadjinlian wrote:
> > This per package variable can be used to specify that a package shares
> > the same sources as another package.
> >
> > The use case here is for example, you have linux-headers and linux, you
> > don't want to download twice the kernel (because of the introduction of
> > the subdirs in DL_DIR).
> >
> > So you want to specify in linux-headers, this variable, so it will use
> > the linux's source.
>
> That patch at least should be part of this series. We *really* don't want to
> download linux twice.
>
> However, Yann is reviving the package out-of-tree build series, which IIRC will
> already do something similar but then reusing the source directory rather than
> the source tarball. Therefore, this patch may no longer be needed. Well, on
> condition that that series gets applied :-)
Yes I'm on it, but this is probably an at-least multi-week effort, if
not months, so don't hold your breath.
Besides, my work (based on Thomas' previous work) will not account for
packages that share the same source tree (and thus the same download)
except when they are a target and host variants.
I.e. linux and linux-headers are two different packages, so my series
will not try to address this use case, at least not for now.
So I think that this patch is still needed. The worst that may happen is
that it today serves to share the download, and that tomorrow it may
change to nmean also sharing the same source directory. Hence, naming
this variable _SAME_SOURCE_AS is future-proof! ;-)
Regards,
Yann E. MORIN.
> Regards,
> Arnout
>
> >
> > Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> > ---
> > package/pkg-generic.mk | 4 +++-
> > 1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> > index 7364551471..a8e0686068 100644
> > --- a/package/pkg-generic.mk
> > +++ b/package/pkg-generic.mk
> > @@ -434,7 +434,9 @@ endif
> >
> > $(2)_BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
> > $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
> > -$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
> > +$(2)_DL_DIR = $$(if $$($(2)_SAME_SOURCE_AS), \
> > + $$(DL_DIR)/$$($$(call UPPERCASE,$$($(2)_SAME_SOURCE_AS))_RAWNAME), \
> > + $$(DL_DIR)/$$($(2)_RAWNAME))
> > $(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME)
> >
> > ifndef $(2)_SUBDIR
> >
>
> --
> Arnout Vandecappelle arnout at mind be
> Senior Embedded Software Architect +32-16-286500
> Essensium/Mind http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature
2017-10-23 19:03 ` Maxime Hadjinlian
@ 2017-10-25 20:02 ` Peter Seiderer
0 siblings, 0 replies; 41+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:02 UTC (permalink / raw)
To: buildroot
Hello Maxime, Arnout,
On Mon, 23 Oct 2017 21:03:21 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
> On Mon, Oct 23, 2017 at 8:42 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> >
> >
> > On 23-10-17 20:24, Maxime Hadjinlian wrote:
> >> Hi Peter,
> >>
> >> On Tue, Oct 17, 2017 at 9:56 PM, Peter Seiderer <ps.report@gmx.net> wrote:
> >>> Hello Maxime,
> >>>
> >>> On Sun, 30 Jul 2017 18:32:27 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
> >>>
> > [snip]
> >>>> It's fixed on my branch on github for now.
> >>> Any updates on this one? Mind to resend an actual version (would like to
> >>> see this feature integrated into buildroot, tired of waiting for hours
> >>> in case RaspberryPi kernel/firmware/userland git versions are updated)?
> >> I think the most "up-to-date" is my branch on Github
> >> (https://github.com/maximeh/buildroot/tree/git_cache).
> >> I don't really think it's ready for a review, there's still many bugs
> >> in there, at least that's what I remember.
> >> What would really help me is, if you could test this and tell me what
> >> breaks so I could look it up and try to fix this, this would be a huge
> >> help !
> >> If you come up with patches, that's really nice too ;).
> >>
> >> I won't make any promises on the time I can spend on this, we all know
> >> how this goes.. but that being said, I am still really motivated by
> >> this feature, I really need it too.
> >
> > Maxime, correct me if I'm wrong, but you'd be happy if Peter does a respin of
> > the series, right?
> Oh sure, for me, the important part is that it's merged, who gets to
> do is not. Since I clearly can't make the time to make it happen; if
> someone can respin this, then please do it :). At least the topic is
> out there :).
> >
> > It would be nice if we could already apply the first few patches, just to keep
> > things going.
> Agreed.
I will send out a v2 patch series of Maximes patches from github (with
a change log added), then hope to address some of Arnouts review comments (and
do some more testing) and will send out a v3 series the next days...
Regards,
Peter
> >
> > Regards,
> > Arnout
> >
> > --
> > Arnout Vandecappelle arnout at mind be
> > Senior Embedded Software Architect +32-16-286500
> > Essensium/Mind http://www.mind.be
> > G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
> > LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> > GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 09/13] pkg-download: support new subdir for mirrors
2017-10-23 19:49 ` Arnout Vandecappelle
@ 2017-10-27 18:14 ` Peter Seiderer
0 siblings, 0 replies; 41+ messages in thread
From: Peter Seiderer @ 2017-10-27 18:14 UTC (permalink / raw)
To: buildroot
Hello Arnout,
On Mon, 23 Oct 2017 21:49:39 +0200, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 04-07-17 18:22, Maxime Hadjinlian wrote:
> > Since we introduced subdirectory to the DL_DIR, we need to support them
> subdirectories
>
> > in the PRIMARY and BACKUP mirror as they evolve to the new tree
> > structure.
> >
> > We check first the new URI (with the subdir), and in case of failure, we
> > go check without.
> > By checking both URIs, we ensure that old mirror are usable.
> mirrors
>
> >
> > Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.co> ---
> > package/pkg-download.mk | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> > index 9a455dc79b..47671e21a7 100644
> > --- a/package/pkg-download.mk
> > +++ b/package/pkg-download.mk
> > @@ -75,6 +75,7 @@ export BR_NO_CHECK_HASH_FOR =
> >
> > ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
> > DOWNLOAD_URIS += \
> > + -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))/$(notdir $($(PKG)_DL_DIR)) \
>
> I was thinking, it is a bit silly to first calculate that path and then
> applying notdir to it, but I guess this is more robust to a change of PKG_DL_DIR.
>
> There is just one thing to check: I think that this DOWNLOAD_URIS may also be
> used outside the context of a package, in which case $(PKG) is not defined. It's
> not terrible because then we just go to PRIMARY_SITE without subdir, but it's a
> pity to do things twice. So maybe better
>
> $(if $(PKG),$(call getschemeplusuri,$(BR2_PRIMARY_SITE))/$(notdir $($(PKG)_DL_DIR))) \
>
> Oh yeah, completely easy to read :-)
The original line changed in Maxims latest patchset to:
-u $(call getschemeplusuri,$(BR2_PRIMARY_SITE)/$(notdir $($(PKG)_DL_DIR)),urlencode) \
would the following line be the correct improved change?
-u $(if $(PKG),$(call getschemeplusuri,$(BR2_PRIMARY_SITE)/$(notdir $($(PKG)_DL_DIR)),urlencode)) \
Regards,
Peter
>
> Regards,
> Arnout
>
> > -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
> > endif
> >
> > @@ -83,6 +84,7 @@ DOWNLOAD_URIS += \
> > -u $($(PKG)_SITE_METHOD)+$(dir $(1))
> > ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
> > DOWNLOAD_URIS += \
> > + -u $(call getschemeplusuri,$(BR2_BACKUP_SITE))/$(notdir $($(PKG)_DL_DIR)) \
> > -u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
> > endif
> > endif
> >
>
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 13/13] download: git: introduce cache feature
2017-07-04 16:22 ` [Buildroot] [PATCH 13/13] download: git: introduce cache feature Maxime Hadjinlian
@ 2017-10-27 18:28 ` Peter Seiderer
2017-10-28 16:55 ` Arnout Vandecappelle
0 siblings, 1 reply; 41+ messages in thread
From: Peter Seiderer @ 2017-10-27 18:28 UTC (permalink / raw)
To: buildroot
Hello Maxime,
On Tue, 4 Jul 2017 18:22:11 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
> Now we keep the git clone that we download and generates our tarball
> from there.
For my Raspberry testcase with 'BR2_LINUX_KERNEL_CUSTOM_REPO_URL="https://github.com/raspberrypi/linux.git"'
the cloned git repository is downloaded as 'dl/linux/git'.
This will lead to collisions e.g. for 'https://github.com/hardkernel/linux.git'.
I think a better approach would be to store the git clone under the complete
source url (escaped), e.g. 'github-com-raspberrypi-linux-git' or
'github_com_raspberrypi_linux.git'?
Regards,
Peter
> The main goal here is that if you change the version of a package (say
> Linux), instead of cloning all over again, you will simply 'git fetch'
> from the repo the missing objects, then generates the tarball again.
>
> This should speed the 'source' part of the build significantly.
>
> The drawback is that the DL_DIR will grow much larger; but time is more
> important than disk space nowadays.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> support/download/git | 69 ++++++++++++++++++++++++++++++----------------------
> 1 file changed, 40 insertions(+), 29 deletions(-)
>
> diff --git a/support/download/git b/support/download/git
> index a49e448e60..834345b53a 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -39,28 +39,34 @@ _git() {
> eval ${GIT} "${@}"
> }
>
> -# Try a shallow clone, since it is faster than a full clone - but that only
> -# works if the version is a ref (tag or branch). Before trying to do a shallow
> -# clone we check if ${cset} is in the list provided by git ls-remote. If not
> -# we fall back on a full clone.
> -#
> -# Messages for the type of clone used are provided to ease debugging in case of
> -# problems
> -git_done=0
> -if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
> - printf "Doing shallow clone\n"
> - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
> - git_done=1
> - else
> - printf "Shallow clone failed, falling back to doing a full clone\n"
> +# We want to check if a cache of the git clone of this repo already exists.
> +git_cache="${BR2_DL_DIR}/${basename%%-*}/git"
> +
> +# If the cache directory already exists, don't try to clone.
> +if [ ! -d "${git_cache}" ]; then
> + # Try a shallow clone, since it is faster than a full clone - but that
> + # only works if the versionis a ref (tag or branch). Before trying to do a
> + # shallow clone we check if ${cset} is in the list provided by git
> + # ls-remote. If not we fall back on a full clone.
> + #
> + # Messages for the type of clone used are provided to ease debugging in
> + # case of problems
> + git_done=0
> + if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
> + printf "Doing shallow clone\n"
> + if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${git_cache}'"; then
> + git_done=1
> + else
> + printf "Shallow clone failed, falling back to doing a full clone\n"
> + fi
> + fi
> + if [ ${git_done} -eq 0 ]; then
> + printf "Doing full clone\n"
> + _git clone ${verbose} "${@}" "'${uri}'" "'${git_cache}'"
> fi
> -fi
> -if [ ${git_done} -eq 0 ]; then
> - printf "Doing full clone\n"
> - _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
> fi
>
> -pushd "${basename}" >/dev/null
> +pushd "${git_cache}" >/dev/null
>
> # Try to get the special refs exposed by some forges (pull-requests for
> # github, changes for gerrit...). There is no easy way to know whether
> @@ -86,20 +92,25 @@ if [ ${recurse} -eq 1 ]; then
> _git submodule update --init --recursive
> fi
>
> -# We do not want the .git dir; we keep other .git files, in case they
> -# are the only files in their directory.
> +# Generate the archive, sort with the C locale so that it is reproducible
> +# We do not want the .git dir; we keep other .git
> +# files, in case they are the only files in their directory.
> # The .git dir would generate non reproducible tarballs as it depends on
> # the state of the remote server. It also would generate large tarballs
> # (gigabytes for some linux trees) when a full clone took place.
> -rm -rf .git
> +find . -not -type d \
> + -and -not -path "./.git/*" >"${BR2_DL_DIR}/${basename}.list"
> +LC_ALL=C sort <"${BR2_DL_DIR}/${basename}.list" >"${BR2_DL_DIR}/${basename}.list.sorted"
>
> -popd >/dev/null
> -
> -# Generate the archive, sort with the C locale so that it is reproducible
> -find "${basename}" -not -type d >"${basename}.list"
> -LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
> # Create GNU-format tarballs, since that's the format of the tarballs on
> # sources.buildroot.org and used in the *.hash files
> -tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
> - -T "${basename}.list.sorted" >"${output}.tar"
> +tar cf - --transform="s/^\./${basename}/" \
> + --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
> + -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}.tar"
> gzip -n <"${output}.tar" >"${output}"
> +tar tf "${output}"
> +
> +rm -f "${BR2_DL_DIR}/${basename}.list"
> +rm -f "${BR2_DL_DIR}/${basename}.list.sorted"
> +
> +popd >/dev/null
^ permalink raw reply [flat|nested] 41+ messages in thread
* [Buildroot] [PATCH 13/13] download: git: introduce cache feature
2017-10-27 18:28 ` Peter Seiderer
@ 2017-10-28 16:55 ` Arnout Vandecappelle
0 siblings, 0 replies; 41+ messages in thread
From: Arnout Vandecappelle @ 2017-10-28 16:55 UTC (permalink / raw)
To: buildroot
On 27-10-17 20:28, Peter Seiderer wrote:
> Hello Maxime,
>
> On Tue, 4 Jul 2017 18:22:11 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
>
>> Now we keep the git clone that we download and generates our tarball
>> from there.
>
> For my Raspberry testcase with 'BR2_LINUX_KERNEL_CUSTOM_REPO_URL="https://github.com/raspberrypi/linux.git"'
> the cloned git repository is downloaded as 'dl/linux/git'.
>
> This will lead to collisions e.g. for 'https://github.com/hardkernel/linux.git'.
>
> I think a better approach would be to store the git clone under the complete
> source url (escaped), e.g. 'github-com-raspberrypi-linux-git' or
> 'github_com_raspberrypi_linux.git'?
No, that's exactly what we want. When the full history has to be downloaded,
90% of the objects in a linux tree are the same and can be reused, even for
wildly different vendor trees. And even if they are completely different
packages that happen to have the same name, then it is still not a problem that
they are the same.
The only possible problem is that there may be conflicting tags with the same
name and different content. Therefore, it is essential that we still do a fetch
every time, even if the tag that we're looking for is already there, to be sure
that we are looking at the right tag.
Regards,
Arnout
>
> Regards,
> Peter
>
>> The main goal here is that if you change the version of a package (say
>> Linux), instead of cloning all over again, you will simply 'git fetch'
>> from the repo the missing objects, then generates the tarball again.
>>
>> This should speed the 'source' part of the build significantly.
>>
>> The drawback is that the DL_DIR will grow much larger; but time is more
>> important than disk space nowadays.
>>
>> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
>> ---
>> support/download/git | 69 ++++++++++++++++++++++++++++++----------------------
>> 1 file changed, 40 insertions(+), 29 deletions(-)
>>
>> diff --git a/support/download/git b/support/download/git
>> index a49e448e60..834345b53a 100755
>> --- a/support/download/git
>> +++ b/support/download/git
>> @@ -39,28 +39,34 @@ _git() {
>> eval ${GIT} "${@}"
>> }
>>
>> -# Try a shallow clone, since it is faster than a full clone - but that only
>> -# works if the version is a ref (tag or branch). Before trying to do a shallow
>> -# clone we check if ${cset} is in the list provided by git ls-remote. If not
>> -# we fall back on a full clone.
>> -#
>> -# Messages for the type of clone used are provided to ease debugging in case of
>> -# problems
>> -git_done=0
>> -if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
>> - printf "Doing shallow clone\n"
>> - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
>> - git_done=1
>> - else
>> - printf "Shallow clone failed, falling back to doing a full clone\n"
>> +# We want to check if a cache of the git clone of this repo already exists.
>> +git_cache="${BR2_DL_DIR}/${basename%%-*}/git"
>> +
>> +# If the cache directory already exists, don't try to clone.
>> +if [ ! -d "${git_cache}" ]; then
>> + # Try a shallow clone, since it is faster than a full clone - but that
>> + # only works if the versionis a ref (tag or branch). Before trying to do a
>> + # shallow clone we check if ${cset} is in the list provided by git
>> + # ls-remote. If not we fall back on a full clone.
>> + #
>> + # Messages for the type of clone used are provided to ease debugging in
>> + # case of problems
>> + git_done=0
>> + if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
>> + printf "Doing shallow clone\n"
>> + if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${git_cache}'"; then
>> + git_done=1
>> + else
>> + printf "Shallow clone failed, falling back to doing a full clone\n"
>> + fi
>> + fi
>> + if [ ${git_done} -eq 0 ]; then
>> + printf "Doing full clone\n"
>> + _git clone ${verbose} "${@}" "'${uri}'" "'${git_cache}'"
>> fi
>> -fi
>> -if [ ${git_done} -eq 0 ]; then
>> - printf "Doing full clone\n"
>> - _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
>> fi
>>
>> -pushd "${basename}" >/dev/null
>> +pushd "${git_cache}" >/dev/null
>>
>> # Try to get the special refs exposed by some forges (pull-requests for
>> # github, changes for gerrit...). There is no easy way to know whether
>> @@ -86,20 +92,25 @@ if [ ${recurse} -eq 1 ]; then
>> _git submodule update --init --recursive
>> fi
>>
>> -# We do not want the .git dir; we keep other .git files, in case they
>> -# are the only files in their directory.
>> +# Generate the archive, sort with the C locale so that it is reproducible
>> +# We do not want the .git dir; we keep other .git
>> +# files, in case they are the only files in their directory.
>> # The .git dir would generate non reproducible tarballs as it depends on
>> # the state of the remote server. It also would generate large tarballs
>> # (gigabytes for some linux trees) when a full clone took place.
>> -rm -rf .git
>> +find . -not -type d \
>> + -and -not -path "./.git/*" >"${BR2_DL_DIR}/${basename}.list"
>> +LC_ALL=C sort <"${BR2_DL_DIR}/${basename}.list" >"${BR2_DL_DIR}/${basename}.list.sorted"
>>
>> -popd >/dev/null
>> -
>> -# Generate the archive, sort with the C locale so that it is reproducible
>> -find "${basename}" -not -type d >"${basename}.list"
>> -LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
>> # Create GNU-format tarballs, since that's the format of the tarballs on
>> # sources.buildroot.org and used in the *.hash files
>> -tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
>> - -T "${basename}.list.sorted" >"${output}.tar"
>> +tar cf - --transform="s/^\./${basename}/" \
>> + --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
>> + -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}.tar"
>> gzip -n <"${output}.tar" >"${output}"
>> +tar tf "${output}"
>> +
>> +rm -f "${BR2_DL_DIR}/${basename}.list"
>> +rm -f "${BR2_DL_DIR}/${basename}.list.sorted"
>> +
>> +popd >/dev/null
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 41+ messages in thread
end of thread, other threads:[~2017-10-28 16:55 UTC | newest]
Thread overview: 41+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-04 16:21 [Buildroot] [PATCH 00/13] New DL_DIR organisation; git cache feature Maxime Hadjinlian
2017-07-04 16:21 ` [Buildroot] [PATCH 01/13] pkg-{download, generic}: remove source-check Maxime Hadjinlian
2017-07-04 17:37 ` Thomas Petazzoni
2017-07-04 16:22 ` [Buildroot] [PATCH 02/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
2017-07-04 23:09 ` Arnout Vandecappelle
2017-07-04 23:14 ` Yann E. MORIN
2017-07-05 0:38 ` Arnout Vandecappelle
2017-07-05 7:09 ` Yann E. MORIN
2017-07-04 16:22 ` [Buildroot] [PATCH 03/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
2017-07-05 0:32 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 04/13] pkg-generic: make PKG_DL_DIR equal to DL_DIR Maxime Hadjinlian
2017-07-05 8:30 ` Arnout Vandecappelle
2017-07-22 21:37 ` Thomas Petazzoni
2017-07-04 16:22 ` [Buildroot] [PATCH 05/13] packages: use new $($PKG)_DL_DIR) variable Maxime Hadjinlian
2017-07-22 21:39 ` Thomas Petazzoni
2017-07-22 21:54 ` Yann E. MORIN
2017-10-23 20:00 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 06/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Maxime Hadjinlian
2017-07-04 16:22 ` [Buildroot] [PATCH 07/13] support/download: make sure the download folder is created Maxime Hadjinlian
2017-07-04 16:22 ` [Buildroot] [PATCH 08/13] pkg-generic: add a subdirectory to the DL_DIR Maxime Hadjinlian
2017-10-23 19:42 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 09/13] pkg-download: support new subdir for mirrors Maxime Hadjinlian
2017-10-23 19:49 ` Arnout Vandecappelle
2017-10-27 18:14 ` Peter Seiderer
2017-07-04 16:22 ` [Buildroot] [PATCH 10/13] pkg-generic: introduce _SAME_SOURCE_AS Maxime Hadjinlian
2017-10-23 19:55 ` Arnout Vandecappelle
2017-10-23 23:09 ` Yann E. MORIN
2017-07-04 16:22 ` [Buildroot] [PATCH 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Maxime Hadjinlian
2017-10-23 20:04 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 12/13] download: add flock call before dl-wrapper Maxime Hadjinlian
2017-10-23 20:17 ` Arnout Vandecappelle
2017-07-04 16:22 ` [Buildroot] [PATCH 13/13] download: git: introduce cache feature Maxime Hadjinlian
2017-10-27 18:28 ` Peter Seiderer
2017-10-28 16:55 ` Arnout Vandecappelle
2017-07-27 22:37 ` [Buildroot] [PATCH 00/13] New DL_DIR organisation; git " Peter Seiderer
2017-07-30 16:32 ` Maxime Hadjinlian
2017-10-17 19:56 ` Peter Seiderer
2017-10-23 18:24 ` Maxime Hadjinlian
2017-10-23 18:42 ` Arnout Vandecappelle
2017-10-23 19:03 ` Maxime Hadjinlian
2017-10-25 20:02 ` Peter Seiderer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.