* [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options
@ 2018-04-02 8:14 Maxime Hadjinlian
2018-04-02 8:14 ` [Buildroot] [v4 02/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
` (14 more replies)
0 siblings, 15 replies; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
From: "Yann E. MORIN" <yann.morin.1998@free.fr>
Currently all download helpers accepts the local output file, the remote
locations, the changesets and so on... as positional arguments.
This was well and nice when that's was all we needed.
But then we added an option to quiesce their verbosity, and that was
shoehorned with a trivial getopts, still keeping all the existing
positional arguments as... positional arguments.
Adding yet more options while keeping positional arguments will not be
very easy, even if we do not envision any new option in the foreseeable
future (but 640K ought to be enough for everyone, remember? ;-) ).
Change all helpers to accept a set of generic options (-q for quiet and
-o for the output file) as well as helper-specific options (like -r for
the repository, -c for a changeset...).
Maxime:
Changed -R to -r for recurse (only for the git backend)
Changed -r to -u for URI (for all backend)
Change -R to -c for cset (for CVS and SVN backend)
Add the export of the BR_BACKEND_DL_GETOPTS so all the backend wrapper
can use the same option easily
Now all the backends use the same common options.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
v1 -> v2:
- Fix comment in bzr backend
- Remove 'R' from BR_BACKEND_DL_GETOPTS
- check-hash shouldn't have had its options changed since it's not a
download backend (Arnout)
---
package/pkg-download.mk | 38 +++++++++++++++++++-------------------
support/download/bzr | 25 ++++++++++++++-----------
support/download/cp | 17 +++++++++--------
support/download/cvs | 34 +++++++++++++++++++---------------
support/download/dl-wrapper | 7 ++++++-
support/download/git | 33 +++++++++++++++++----------------
support/download/hg | 25 ++++++++++++++-----------
support/download/scp | 19 ++++++++++---------
support/download/svn | 25 ++++++++++++++-----------
support/download/wget | 17 +++++++++--------
10 files changed, 131 insertions(+), 109 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 6f15deba28..a410dce1ee 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -77,9 +77,9 @@ define DOWNLOAD_GIT
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_BASENAME_RAW) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_BASENAME_RAW) \
$($(PKG)_DL_OPTS)
endef
@@ -88,9 +88,9 @@ define DOWNLOAD_BZR
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_BASENAME_RAW) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_BASENAME_RAW) \
$($(PKG)_DL_OPTS)
endef
@@ -99,10 +99,10 @@ define DOWNLOAD_CVS
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAWNAME) \
- $($(PKG)_BASENAME_RAW) \
+ -u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
+ -c $($(PKG)_DL_VERSION) \
+ -N $($(PKG)_RAWNAME) \
+ -n $($(PKG)_BASENAME_RAW) \
$($(PKG)_DL_OPTS)
endef
@@ -111,9 +111,9 @@ define DOWNLOAD_SVN
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_BASENAME_RAW) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_BASENAME_RAW) \
$($(PKG)_DL_OPTS)
endef
@@ -126,7 +126,7 @@ define DOWNLOAD_SCP
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- '$(call stripurischeme,$(call qstrip,$(1)))' \
+ -u '$(call stripurischeme,$(call qstrip,$(1)))' \
$($(PKG)_DL_OPTS)
endef
@@ -135,9 +135,9 @@ define DOWNLOAD_HG
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_BASENAME_RAW) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_BASENAME_RAW) \
$($(PKG)_DL_OPTS)
endef
@@ -147,7 +147,7 @@ define DOWNLOAD_WGET
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- '$(call qstrip,$(1))' \
+ -u '$(call qstrip,$(1))' \
$($(PKG)_DL_OPTS)
endef
@@ -157,7 +157,7 @@ define DOWNLOAD_LOCALFILES
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- $(call stripurischeme,$(call qstrip,$(1))) \
+ -u $(call stripurischeme,$(call qstrip,$(1))) \
$($(PKG)_DL_OPTS)
endef
diff --git a/support/download/bzr b/support/download/bzr
index 75b7b415c1..5289a421cd 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -5,28 +5,31 @@ set -e
# Download helper for bzr, to be called from the download wrapper script
#
-# Call it as:
-# .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
+# Options:
+# -q Be quiet
+# -o FILE Generate archive in FILE.
+# -u URI Clone from repository at URI.
+# -c CSET Use changeset (or revision) CSET.
+# -n NAME Use basename NAME.
#
# Environment:
# BZR : the bzr command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) cset="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-rev="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -51,5 +54,5 @@ if [ ${bzr_version} -ge ${bzr_min_version} ]; then
fi
_bzr export ${verbose} --root="'${basename}/'" --format=tgz \
- ${timestamp_opt} - "${@}" "'${repo}'" -r "'${rev}'" \
+ ${timestamp_opt} - "${@}" "'${uri}'" -r "'${cset}'" \
>"${output}"
diff --git a/support/download/cp b/support/download/cp
index 0ee1f3ba82..52fe2de83d 100755
--- a/support/download/cp
+++ b/support/download/cp
@@ -5,8 +5,10 @@ set -e
# Download helper for cp, to be called from the download wrapper script
#
-# Call it as:
-# .../cp [-q] OUT_FILE SRC_FILE
+# Options:
+# -q Be quiet.
+# -o FILE Copy to file FILE.
+# -u FILE Copy from file FILE.
#
# Environment:
# LOCALFILES: the cp command to call
@@ -17,18 +19,17 @@ set -e
# Make 'cp' verbose by default, so it behaves a bit like the others.
verbose=-v
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=;;
+ o) output="${OPTARG}";;
+ u) source="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-source="${2}"
-
-shift 2 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
diff --git a/support/download/cvs b/support/download/cvs
index 50050ab1c9..69d5c71f28 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -5,28 +5,32 @@ set -e
# Download helper for cvs, to be called from the download wrapper script
#
-# Call it as:
-# .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
+# Options:
+# -q Be quiet
+# -o FILE Generate archive in FILE.
+# -u URI Checkout from repository at URI.
+# -c REV Use revision REV.
+# -N RAWNAME Use rawname (aka module name) RAWNAME.
+# -n NAME Use basename NAME.
#
# Environment:
# CVS : the cvs command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-Q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) rev="${OPTARG}";;
+ N) rawname="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-rev="${3}"
-rawname="${4}"
-basename="${5}"
-
-shift 5 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -42,14 +46,14 @@ else
select="-r"
fi
-# The absence of an initial : on ${repo} means access method undefined
-if [[ ! "${repo}" =~ ^: ]]; then
+# The absence of an initial : on ${uri} means access method undefined
+if [[ ! "${uri}" =~ ^: ]]; then
# defaults to anonymous pserver
- repo=":pserver:anonymous@${repo}"
+ uri=":pserver:anonymous@${uri}"
fi
export TZ=UTC
-_cvs ${verbose} -z3 -d"'${repo}'" \
+_cvs ${verbose} -z3 -d"'${uri}'" \
co "${@}" -d "'${basename}'" ${select} "'${rev}'" -P "'${rawname}'"
tar czf "${output}" "${basename}"
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index b018819833..abc51f637a 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -19,6 +19,8 @@
# We want to catch any unexpected failure, and exit immediately.
set -e
+export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:q"
+
main() {
local OPT OPTARG
local backend output hfile recurse quiet rc
@@ -83,7 +85,10 @@ main() {
# If the backend fails, we can just remove the temporary directory to
# remove all the cruft it may have left behind. Then we just exit in
# error too.
- if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
+ if ! "${OLDPWD}/support/download/${backend}" \
+ ${quiet} ${recurse} \
+ -o "${tmpf}" "${@}"
+ then
rm -rf "${tmpd}"
exit 1
fi
diff --git a/support/download/git b/support/download/git
index f590ff6494..58a2c6ad9d 100755
--- a/support/download/git
+++ b/support/download/git
@@ -5,32 +5,33 @@ set -e
# Download helper for git, to be called from the download wrapper script
#
-# Call it as:
-# .../git [-q] [-r] OUT_FILE REPO_URL CSET BASENAME
-#
-# -q Be quiet.
-# -r Clone and archive sub-modules.
+# Options:
+# -q Be quiet.
+# -r Clone and archive sub-modules.
+# -o FILE Generate archive in FILE.
+# -u URI Clone from repository at URI.
+# -c CSET Use changeset CSET.
+# -n NAME Use basename NAME.
#
# Environment:
# GIT : the git command to call
verbose=
recurse=0
-while getopts :qr OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q; exec >/dev/null;;
r) recurse=1;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) cset="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-cset="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -46,9 +47,9 @@ _git() {
# Messages for the type of clone used are provided to ease debugging in case of
# problems
git_done=0
-if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
+if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
printf "Doing shallow clone\n"
- if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
+ if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
git_done=1
else
printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -56,7 +57,7 @@ if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
fi
if [ ${git_done} -eq 0 ]; then
printf "Doing full clone\n"
- _git clone ${verbose} "${@}" "'${repo}'" "'${basename}'"
+ _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
fi
pushd "${basename}" >/dev/null
diff --git a/support/download/hg b/support/download/hg
index 3af01690b3..efb515fca5 100755
--- a/support/download/hg
+++ b/support/download/hg
@@ -5,27 +5,30 @@ set -e
# Download helper for hg, to be called from the download wrapper script
#
-# Call it as:
-# .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
+# Options:
+# -q Be quiet.
+# -o FILE Generate archive in FILE.
+# -u URI Clone from repository at URI.
+# -c CSET Use changeset (or revision) CSET.
+# -n NAME Use basename NAME.
#
# Environment:
# HG : the hg command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) cset="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-cset="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -33,7 +36,7 @@ _hg() {
eval ${HG} "${@}"
}
-_hg clone ${verbose} "${@}" --noupdate "'${repo}'" "'${basename}'"
+_hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'"
_hg archive ${verbose} --repository "'${basename}'" --type tgz \
--prefix "'${basename}'" --rev "'${cset}'" \
diff --git a/support/download/scp b/support/download/scp
index 825fd41c64..8ecf2f4b22 100755
--- a/support/download/scp
+++ b/support/download/scp
@@ -5,25 +5,26 @@ set -e
# Download helper for scp, to be called from the download wrapper script
#
-# Call it as:
-# .../scp [-q] OUT_FILE SRC_URL
+# Options:
+# -q Be quiet.
+# -o FILE Copy to local file FILE.
+# -u FILE Copy from remote file FILE.
#
# Environment:
# SCP : the scp command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-url="${2}"
-
-shift 2 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -31,4 +32,4 @@ _scp() {
eval ${SCP} "${@}"
}
-_scp ${verbose} "${@}" "'${url}'" "'${output}'"
+_scp ${verbose} "${@}" "'${uri}'" "'${output}'"
diff --git a/support/download/svn b/support/download/svn
index 77abf3d02d..542b25c0a2 100755
--- a/support/download/svn
+++ b/support/download/svn
@@ -5,27 +5,30 @@ set -e
# Download helper for svn, to be called from the download wrapper script
#
-# Call it as:
-# .../svn [-q] OUT_FILE REPO_URL REV BASNAME
+# Options:
+# -q Be quiet.
+# -o FILE Generate archive in FILE.
+# -u URI Checkout from repository at URI.
+# -c REV Use revision REV.
+# -n NAME Use basename NAME.
#
# Environment:
# SVN : the svn command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) rev="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-rev="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -33,6 +36,6 @@ _svn() {
eval ${SVN} "${@}"
}
-_svn export ${verbose} "${@}" "'${repo}@${rev}'" "'${basename}'"
+_svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
tar czf "${output}" "${basename}"
diff --git a/support/download/wget b/support/download/wget
index 768de904c3..fece6663ca 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -5,25 +5,26 @@ set -e
# Download helper for wget, to be called from the download wrapper script
#
-# Call it as:
-# .../wget [-q] OUT_FILE URL
+# Options:
+# -q Be quiet.
+# -o FILE Save into file FILE.
+# -u URL Download file at URL.
#
# Environment:
# WGET : the wget command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) url="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-url="${2}"
-
-shift 2 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 02/13] download: put most of the infra in dl-wrapper
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 10:23 ` Yann E. MORIN
2018-04-02 12:18 ` Peter Korsgaard
2018-04-02 8:14 ` [Buildroot] [v4 03/13] packages: use new $($PKG)_DL_DIR) variable Maxime Hadjinlian
` (13 subsequent siblings)
14 siblings, 2 replies; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
The goal here is to simplify the infrastructure by putting most of the
code in the dl-wrapper as it's easier to implement and to read.
Most of the functions were common already, this patch finalizes it by
making the pkg-download.mk pass all the parameters needed to the
dl-wrapper which in turns will pass everything to every backend.
The backend will then cherry-pick what it needs from these arguments
and act accordingly.
It eases the transition to the addition of a sub directory per package
in the DL_DIR, and later on, a git cache.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
---
v1 -> v2:
- Rename cp backend to file (Arnout)
- Don't use BR_BACKEND_DL_GETOPTS for dl-wrapper (Arnout)
- Add "urlencode" to scheme passed to the dl-wrapper to support the
fact that we need to urlencode the filename when using PRIMARY and
BACKUP mirror (some files are named toto.c?v=1.0) (Arnout)
- Fix uristripscheme replaced by bash'ism (Arnout)
- Add the check hash into the loop, exit with error only if all the
download+check failed. (Arnout)
---
package/pkg-download.mk | 170 ++++++++------------------------------------
support/download/cvs | 2 +-
support/download/dl-wrapper | 139 +++++++++++++++++++++++-------------
support/download/wget | 10 ++-
4 files changed, 129 insertions(+), 192 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index a410dce1ee..2ee745fddd 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -42,6 +42,8 @@ DL_DIR := $(shell mkdir -p $(DL_DIR) && cd $(DL_DIR) >/dev/null && pwd)
#
# geturischeme: http
geturischeme = $(firstword $(subst ://, ,$(call qstrip,$(1))))
+# getschemeplusuri: git|parameter+http://example.com
+getschemeplusuri = $(call geturischeme,$(1))$(if $(2),\|$(2))+$(1)
# stripurischeme: www.example.com/dir/file
stripurischeme = $(lastword $(subst ://, ,$(call qstrip,$(1))))
# domain: www.example.com
@@ -61,152 +63,42 @@ github = https://github.com/$(1)/$(2)/archive/$(3)
export BR_NO_CHECK_HASH_FOR =
################################################################################
-# The DOWNLOAD_* helpers are in charge of getting a working copy
-# of the source repository for their corresponding SCM,
-# checking out the requested version / commit / tag, and create an
-# archive out of it. DOWNLOAD_SCP uses scp to obtain a remote file with
-# ssh authentication. DOWNLOAD_WGET is the normal wget-based download
-# mechanism.
-#
-################################################################################
-
-define DOWNLOAD_GIT
- $(EXTRA_ENV) $(DL_WRAPPER) -b git \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(if $($(PKG)_GIT_SUBMODULES),-r) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_BASENAME_RAW) \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_BZR
- $(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_BASENAME_RAW) \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_CVS
- $(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
- -c $($(PKG)_DL_VERSION) \
- -N $($(PKG)_RAWNAME) \
- -n $($(PKG)_BASENAME_RAW) \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_SVN
- $(EXTRA_ENV) $(DL_WRAPPER) -b svn \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_BASENAME_RAW) \
- $($(PKG)_DL_OPTS)
-endef
-
-# SCP URIs should be of the form scp://[user@]host:filepath
-# Note that filepath is relative to the user's home directory, so you may want
-# to prepend the path with a slash: scp://[user@]host:/absolutepath
-define DOWNLOAD_SCP
- $(EXTRA_ENV) $(DL_WRAPPER) -b scp \
- -o $(DL_DIR)/$(2) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u '$(call stripurischeme,$(call qstrip,$(1)))' \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_HG
- $(EXTRA_ENV) $(DL_WRAPPER) -b hg \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_BASENAME_RAW) \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_WGET
- $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
- -o $(DL_DIR)/$(2) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u '$(call qstrip,$(1))' \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_LOCALFILES
- $(EXTRA_ENV) $(DL_WRAPPER) -b cp \
- -o $(DL_DIR)/$(2) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u $(call stripurischeme,$(call qstrip,$(1))) \
- $($(PKG)_DL_OPTS)
-endef
-
-################################################################################
-# DOWNLOAD -- Download helper. Will try to download source from:
+# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download
+# source from:
# 1) BR2_PRIMARY_SITE if enabled
# 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
# 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
#
# Argument 1 is the source location
#
-# E.G. use like this:
-# $(call DOWNLOAD,$(FOO_SITE))
-#
-# For PRIMARY and BACKUP site, any ? in the URL is replaced by %3F. A ? in
-# the URL is used to separate query arguments, but the PRIMARY and BACKUP
-# sites serve just plain files.
################################################################################
+ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
+DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE),urlencode)
+endif
+
+ifeq ($(BR2_PRIMARY_SITE_ONLY),)
+DOWNLOAD_URIS += \
+ -u $($(PKG)_SITE_METHOD)+$(dir $(1))
+ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
+DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_BACKUP_SITE),urlencode)
+endif
+endif
+
define DOWNLOAD
- $(call DOWNLOAD_INNER,$(1),$(notdir $(1)),DOWNLOAD)
-endef
-
-define DOWNLOAD_INNER
- $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
- if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
- case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \
- file) $(call $(3)_LOCALFILES,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
- scp) $(call $(3)_SCP,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
- *) $(call $(3)_WGET,$(BR2_PRIMARY_SITE)/$(subst ?,%3F,$(2)),$(2)) && exit ;; \
- esac ; \
- fi ; \
- if test "$(BR2_PRIMARY_SITE_ONLY)" = "y" ; then \
- exit 1 ; \
- fi ; \
- if test -n "$(1)" ; then \
- case "$($(PKG)_SITE_METHOD)" in \
- git) $($(3)_GIT) && exit ;; \
- svn) $($(3)_SVN) && exit ;; \
- cvs) $($(3)_CVS) && exit ;; \
- bzr) $($(3)_BZR) && exit ;; \
- file) $($(3)_LOCALFILES) && exit ;; \
- scp) $($(3)_SCP) && exit ;; \
- hg) $($(3)_HG) && exit ;; \
- *) $(call $(3)_WGET,$(1),$(2)) && exit ;; \
- esac ; \
- fi ; \
- if test -n "$(call qstrip,$(BR2_BACKUP_SITE))" ; then \
- $(call $(3)_WGET,$(BR2_BACKUP_SITE)/$(subst ?,%3F,$(2)),$(2)) && exit ; \
- fi ; \
- exit 1
+ $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
+ $(EXTRA_ENV) $(DL_WRAPPER) \
+ -c $($(PKG)_DL_VERSION) \
+ -f $(notdir $(1)) \
+ -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ -n $($(PKG)_BASENAME_RAW) \
+ -N $($(PKG)_RAWNAME) \
+ -o $(DL_DIR)/$(notdir $(1)) \
+ $(if $($(PKG)_GIT_SUBMODULES),-r) \
+ $(DOWNLOAD_URIS) \
+ $(QUIET) \
+ -- \
+ $($(PKG)_DL_OPTS)
endef
diff --git a/support/download/cvs b/support/download/cvs
index 69d5c71f28..3f77b849e4 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -21,7 +21,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-Q;;
o) output="${OPTARG}";;
- u) uri="${OPTARG}";;
+ u) uri="${OPTARG#*://}";;
c) rev="${OPTARG}";;
N) rawname="${OPTARG}";;
n) basename="${OPTARG}";;
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index abc51f637a..58cc04b4c4 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -19,31 +19,34 @@
# We want to catch any unexpected failure, and exit immediately.
set -e
-export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:q"
+export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:qf:e"
main() {
local OPT OPTARG
local backend output hfile recurse quiet rc
+ local -a uris
# Parse our options; anything after '--' is for the backend
- while getopts :hb:o:H:rq OPT; do
+ while getopts ":hc:o:n:N:H:rf:u:q" OPT; do
case "${OPT}" in
h) help; exit 0;;
- b) backend="${OPTARG}";;
+ c) cset="${OPTARG}";;
o) output="${OPTARG}";;
+ n) raw_base_name="${OPTARG}";;
+ N) base_name="${OPTARG}";;
H) hfile="${OPTARG}";;
r) recurse="-r";;
+ f) filename="${OPTARG}";;
+ u) uris+=( "${OPTARG}" );;
q) quiet="-q";;
:) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
\?) error "unknown option '%s'\n" "${OPTARG}";;
esac
done
+
# Forget our options, and keep only those for the backend
shift $((OPTIND-1))
- if [ -z "${backend}" ]; then
- error "no backend specified, use -b\n"
- fi
if [ -z "${output}" ]; then
error "no output specified, use -o\n"
fi
@@ -66,48 +69,85 @@ main() {
warn "Re-downloading '%s'...\n" "${output##*/}"
fi
- # tmpd is a temporary directory in which backends may store intermediate
- # by-products of the download.
- # tmpf is the file in which the backends should put the downloaded content.
- # tmpd is located in $(BUILD_DIR), so as not to clutter the (precious)
- # $(BR2_DL_DIR)
- # We let the backends create tmpf, so they are able to set whatever
- # permission bits they want (although we're only really interested in
- # the executable bit.)
- tmpd="$(mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX")"
- tmpf="${tmpd}/output"
+ # Look through all the uris that we were given to downoad the package
+ # source
+ download_and_check=0
+ rc=1
+ for uri in "${uris[@]}"; do
+ backend=${uri%+*}
+ case "${backend}" in
+ git|svn|cvs|bzr|file|scp|hg) ;;
+ *) backend="wget" ;;
+ esac
+ uri=${uri#*+}
- # Helpers expect to run in a directory that is *really* trashable, so
- # they are free to create whatever files and/or sub-dirs they might need.
- # Doing the 'cd' here rather than in all backends is easier.
- cd "${tmpd}"
+ urlencode=${backend#*|}
+ # urlencode must be "urlencode"
+ [ "${urlencode}" != "urlencode" ] && urlencode=""
- # If the backend fails, we can just remove the temporary directory to
- # remove all the cruft it may have left behind. Then we just exit in
- # error too.
- if ! "${OLDPWD}/support/download/${backend}" \
- ${quiet} ${recurse} \
- -o "${tmpf}" "${@}"
- then
- rm -rf "${tmpd}"
- exit 1
- fi
+ # tmpd is a temporary directory in which backends may store
+ # intermediate by-products of the download.
+ # tmpf is the file in which the backends should put the downloaded
+ # content.
+ # tmpd is located in $(BUILD_DIR), so as not to clutter the (precious)
+ # $(BR2_DL_DIR)
+ # We let the backends create tmpf, so they are able to set whatever
+ # permission bits they want (although we're only really interested in
+ # the executable bit.)
+ tmpd="$(mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX")"
+ tmpf="${tmpd}/output"
- # cd back to free the temp-dir, so we can remove it later
- cd "${OLDPWD}"
+ # Helpers expect to run in a directory that is *really* trashable, so
+ # they are free to create whatever files and/or sub-dirs they might need.
+ # Doing the 'cd' here rather than in all backends is easier.
+ cd "${tmpd}"
- # Check if the downloaded file is sane, and matches the stored hashes
- # for that file
- if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
- rc=0
- else
- if [ ${?} -ne 3 ]; then
+ # If the backend fails, we can just remove the content of the temporary
+ # directory to remove all the cruft it may have left behind, and tries
+ # the next URI until it succeeds. Once out of URI to tries, we need to
+ # cleanup and exit.
+ if ! "${OLDPWD}/support/download/${backend}" \
+ $([ -n "${urlencode}" ] && printf %s '-e') \
+ -c "${cset}" \
+ -n "${raw_base_name}" \
+ -N "${raw_name}" \
+ -f "${filename}" \
+ -u "${uri}" \
+ -o "${tmpf}" \
+ ${quiet} ${recurse} "${@}"
+ then
rm -rf "${tmpd}"
- exit 1
+ # cd back to keep path coherence
+ cd "${OLDPWD}"
+ continue
fi
- # the hash file exists and there was no hash to check the file against
- rc=1
+ # cd back to free the temp-dir, so we can remove it later
+ cd "${OLDPWD}"
+
+ # Check if the downloaded file is sane, and matches the stored hashes
+ # for that file
+ if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
+ rc=0
+ else
+ if [ ${?} -ne 3 ]; then
+ rm -rf "${tmpd}"
+ continue
+ fi
+
+ # the hash file exists and there was no hash to check the file
+ # against
+ rc=1
+ fi
+ download_and_check=1
+ break
+ done
+
+ # We tried every URI possible, none seems to work or to check against the
+ # available hash. *ABORT MISSION*
+ if [ "${download_and_check}" -eq 0 ]; then
+ rm -rf "${tmpd}"
+ exit 1
fi
# tmp_output is in the same directory as the final output, so we can
@@ -173,16 +213,13 @@ DESCRIPTION
-h This help text.
- -b BACKEND
- Wrap the specified BACKEND. Known backends are:
- bzr Bazaar
- cp Local files
- cvs Concurrent Versions System
- git Git
- hg Mercurial
- scp Secure copy
- svn Subversion
- wget HTTP download
+ -u URIs
+ The URI to get the file from, the URI must respect the format given in
+ the example.
+ You may give as many '-u URI' as you want, the script will stop at the
+ frist successful download.
+
+ Example: backend+URI; git+http://example.com or http+http://example.com
-o FILE
Store the downloaded archive in FILE.
diff --git a/support/download/wget b/support/download/wget
index fece6663ca..c69e6071aa 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -8,7 +8,9 @@ set -e
# Options:
# -q Be quiet.
# -o FILE Save into file FILE.
+# -f FILENAME The filename of the tarball to get at URL
# -u URL Download file at URL.
+# -e ENCODE Tell wget to urlencode the filename passed to it
#
# Environment:
# WGET : the wget command to call
@@ -18,7 +20,9 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
o) output="${OPTARG}";;
+ f) filename="${OPTARG}";;
u) url="${OPTARG}";;
+ e) encode="-e";;
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
@@ -32,4 +36,8 @@ _wget() {
eval ${WGET} "${@}"
}
-_wget ${verbose} "${@}" -O "'${output}'" "'${url}'"
+# Replace every '?' with '%3F' in the filename; only for the PRIMARY and BACKUP
+# mirror
+[ -n "${encode}" ] && filename=${filename//\?/%3F}
+
+_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 03/13] packages: use new $($PKG)_DL_DIR) variable
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
2018-04-02 8:14 ` [Buildroot] [v4 02/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 10:36 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 04/13] arc/xtensa: store the eXtensa overlay in the per-package DL_DIR Maxime Hadjinlian
` (12 subsequent siblings)
14 siblings, 1 reply; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the
transition into a new directory structure for DL_DIR.
This commit has been generated with the following scripts:
for i in $(find . -iname "*.mk"); do
if ! grep -q "\$(DL_DIR)" ${i}; then
continue
fi
pkg_name="$(basename $(dirname ${i}))"
[ "${pkg_name}" = "package" ] && continue
raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
pkg_dl_dir="${raw_pkg_name}_DL_DIR"
sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}
done
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
boot/at91bootstrap/at91bootstrap.mk | 2 +-
package/amd-catalyst/amd-catalyst.mk | 2 +-
package/android-tools/android-tools.mk | 2 +-
package/angularjs/angularjs.mk | 2 +-
package/bootstrap/bootstrap.mk | 4 +++-
package/cache-calibrator/cache-calibrator.mk | 2 +-
package/cargo/cargo.mk | 4 ++--
package/cracklib/cracklib.mk | 2 +-
package/cryptopp/cryptopp.mk | 2 +-
package/devmem2/devmem2.mk | 2 +-
package/dhrystone/dhrystone.mk | 2 +-
package/doom-wad/doom-wad.mk | 2 +-
package/espeak/espeak.mk | 2 +-
package/fan-ctrl/fan-ctrl.mk | 2 +-
package/freescale-imx/firmware-imx/firmware-imx.mk | 2 +-
package/freescale-imx/gpu-amd-bin-mx51/gpu-amd-bin-mx51.mk | 2 +-
package/freescale-imx/imx-codec/imx-codec.mk | 2 +-
package/freescale-imx/imx-gpu-g2d/imx-gpu-g2d.mk | 2 +-
package/freescale-imx/imx-gpu-viv/imx-gpu-viv.mk | 2 +-
package/freescale-imx/imx-parser/imx-parser.mk | 2 +-
package/freescale-imx/imx-vpu/imx-vpu.mk | 2 +-
package/freescale-imx/imx-vpuwrap/imx-vpuwrap.mk | 2 +-
package/freescale-imx/libz160/libz160.mk | 2 +-
package/gcc/gcc.mk | 2 +-
package/irrlicht/irrlicht.mk | 2 +-
package/jquery-mobile/jquery-mobile.mk | 2 +-
package/jquery-sparkline/jquery-sparkline.mk | 2 +-
package/jquery-ui-themes/jquery-ui-themes.mk | 2 +-
package/jquery-ui/jquery-ui.mk | 2 +-
package/jquery-validation/jquery-validation.mk | 2 +-
package/jquery/jquery.mk | 2 +-
package/kodi/kodi.mk | 6 +++---
package/libb64/libb64.mk | 2 +-
package/libfreeimage/libfreeimage.mk | 2 +-
package/libjson/libjson.mk | 2 +-
package/libsoil/libsoil.mk | 2 +-
package/lsof/lsof.mk | 2 +-
| 2 +-
package/nmon/nmon.mk | 2 +-
package/nvidia-driver/nvidia-driver.mk | 2 +-
.../nvidia-tegra23/nvidia-tegra23-codecs/nvidia-tegra23-codecs.mk | 2 +-
package/openobex/openobex.mk | 6 ++++++
package/opentyrian-data/opentyrian-data.mk | 2 +-
package/perl/perl.mk | 2 +-
package/python-keyring/python-keyring.mk | 1 -
package/python-pytz/python-pytz.mk | 2 +-
package/python-simplegeneric/python-simplegeneric.mk | 2 +-
package/rapidxml/rapidxml.mk | 2 +-
package/rpi-wifi-firmware/rpi-wifi-firmware.mk | 2 +-
package/rust-bin/rust-bin.mk | 2 +-
package/sam-ba/sam-ba.mk | 2 +-
package/spidev_test/spidev_test.mk | 2 +-
package/tar/tar.mk | 2 +-
package/tesseract-ocr/tesseract-ocr.mk | 2 +-
package/ti-cgt-pru/ti-cgt-pru.mk | 4 ++--
package/ti-gfx/ti-gfx.mk | 4 ++--
package/ts4900-fpga/ts4900-fpga.mk | 2 +-
package/unscd/unscd.mk | 2 +-
package/urg/urg.mk | 4 +++-
package/waf/waf.mk | 2 +-
package/whetstone/whetstone.mk | 2 +-
package/wilc1000-firmware/wilc1000-firmware.mk | 2 +-
package/zynq-boot-bin/zynq-boot-bin.mk | 2 +-
63 files changed, 76 insertions(+), 67 deletions(-)
diff --git a/boot/at91bootstrap/at91bootstrap.mk b/boot/at91bootstrap/at91bootstrap.mk
index f6550588d3..c68b56b6ec 100644
--- a/boot/at91bootstrap/at91bootstrap.mk
+++ b/boot/at91bootstrap/at91bootstrap.mk
@@ -17,7 +17,7 @@ AT91BOOTSTRAP_INSTALL_IMAGES = YES
AT91BOOTSTRAP_INSTALL_TARGET = NO
define AT91BOOTSTRAP_EXTRACT_CMDS
- $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(AT91BOOTSTRAP_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR) $(AT91BOOTSTRAP_DL_DIR)/$(AT91BOOTSTRAP_SOURCE)
mv $(BUILD_DIR)/Bootstrap-v$(AT91BOOTSTRAP_VERSION)/* $(@D)
rmdir $(BUILD_DIR)/Bootstrap-v$(AT91BOOTSTRAP_VERSION)
endef
diff --git a/package/amd-catalyst/amd-catalyst.mk b/package/amd-catalyst/amd-catalyst.mk
index b9396e11fa..d864095c31 100644
--- a/package/amd-catalyst/amd-catalyst.mk
+++ b/package/amd-catalyst/amd-catalyst.mk
@@ -17,7 +17,7 @@ AMD_CATALYST_ARCH_DIR = $(@D)/arch/x86$(AMD_CATALYST_SUFFIX)
AMD_CATALYST_LIB_SUFFIX = $(if $(BR2_x86_64),64)
define AMD_CATALYST_EXTRACT_CMDS
- unzip -q $(DL_DIR)/$(AMD_CATALYST_SOURCE) -d $(@D)
+ unzip -q $(AMD_CATALYST_DL_DIR)/$(AMD_CATALYST_SOURCE) -d $(@D)
$(SHELL) $(@D)/AMD-Catalyst-$(AMD_CATALYST_VERSION)-Linux-installer-$(AMD_CATALYST_VERBOSE_VER)-x86.x86_64.run --extract $(@D)
endef
diff --git a/package/android-tools/android-tools.mk b/package/android-tools/android-tools.mk
index f6c6913a68..11e0a15a7f 100644
--- a/package/android-tools/android-tools.mk
+++ b/package/android-tools/android-tools.mk
@@ -15,7 +15,7 @@ ANDROID_TOOLS_LICENSE_FILES = debian/copyright
# Extract the Debian tarball inside the sources
define ANDROID_TOOLS_DEBIAN_EXTRACT
$(call suitable-extractor,$(notdir $(ANDROID_TOOLS_EXTRA_DOWNLOADS))) \
- $(DL_DIR)/$(notdir $(ANDROID_TOOLS_EXTRA_DOWNLOADS)) | \
+ $(ANDROID_TOOLS_DL_DIR)/$(notdir $(ANDROID_TOOLS_EXTRA_DOWNLOADS)) | \
$(TAR) -C $(@D) $(TAR_OPTIONS) -
endef
diff --git a/package/angularjs/angularjs.mk b/package/angularjs/angularjs.mk
index 60702a26d2..a941bc3828 100644
--- a/package/angularjs/angularjs.mk
+++ b/package/angularjs/angularjs.mk
@@ -12,7 +12,7 @@ ANGULARJS_LICENSE = MIT
ANGULARJS_LICENSE_FILES = angular.js
define ANGULARJS_EXTRACT_CMDS
- unzip $(DL_DIR)/$(ANGULARJS_SOURCE) -d $(@D)
+ unzip $(ANGULARJS_DL_DIR)/$(ANGULARJS_SOURCE) -d $(@D)
mv $(@D)/angular-$(ANGULARJS_VERSION)/* $(@D)
rmdir $(@D)/angular-$(ANGULARJS_VERSION)
endef
diff --git a/package/bootstrap/bootstrap.mk b/package/bootstrap/bootstrap.mk
index 61d6c826fc..50cdc322e4 100644
--- a/package/bootstrap/bootstrap.mk
+++ b/package/bootstrap/bootstrap.mk
@@ -11,7 +11,9 @@ BOOTSTRAP_LICENSE = MIT
BOOTSTRAP_LICENSE_FILES = css/bootstrap.css
define BOOTSTRAP_EXTRACT_CMDS
- $(UNZIP) $(DL_DIR)/$(BOOTSTRAP_SOURCE) -d $(@D)
+ $(UNZIP) $(BOOTSTRAP_DL_DIR)/$(BOOTSTRAP_SOURCE) -d $(@D)
+ mv $(@D)/bootstrap-$(BOOTSTRAP_VERSION)-dist/* $(@D)/
+ rmdir $(@D)/bootstrap-$(BOOTSTRAP_VERSION)-dist
endef
define BOOTSTRAP_INSTALL_TARGET_CMDS
diff --git a/package/cache-calibrator/cache-calibrator.mk b/package/cache-calibrator/cache-calibrator.mk
index 6f12d119ef..203732e4f7 100644
--- a/package/cache-calibrator/cache-calibrator.mk
+++ b/package/cache-calibrator/cache-calibrator.mk
@@ -10,7 +10,7 @@ CACHE_CALIBRATOR_LICENSE = Cache calibrator license
CACHE_CALIBRATOR_LICENSE_FILES = calibrator.c.license
define CACHE_CALIBRATOR_EXTRACT_CMDS
- cp $(DL_DIR)/$(CACHE_CALIBRATOR_SOURCE) $(@D)
+ cp $(CACHE_CALIBRATOR_DL_DIR)/$(CACHE_CALIBRATOR_SOURCE) $(@D)
endef
define CACHE_CALIBRATOR_EXTRACT_LICENSE
diff --git a/package/cargo/cargo.mk b/package/cargo/cargo.mk
index 3fd088d727..36b2718e90 100644
--- a/package/cargo/cargo.mk
+++ b/package/cargo/cargo.mk
@@ -37,7 +37,7 @@ HOST_CARGO_HOME = $(HOST_DIR)/share/cargo
define HOST_CARGO_EXTRACT_DEPS
@mkdir -p $(@D)/vendor
$(call suitable-extractor,$(CARGO_DEPS_SOURCE)) \
- $(DL_DIR)/$(CARGO_DEPS_SOURCE) | \
+ $(CARGO_DL_DIR)/$(CARGO_DEPS_SOURCE) | \
$(TAR) --strip-components=1 -C $(@D)/vendor $(TAR_OPTIONS) -
endef
@@ -46,7 +46,7 @@ HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_DEPS
define HOST_CARGO_EXTRACT_INSTALLER
@mkdir -p $(@D)/src/rust-installer
$(call suitable-extractor,$(CARGO_INSTALLER_SOURCE)) \
- $(DL_DIR)/$(CARGO_INSTALLER_SOURCE) | \
+ $(CARGO_DL_DIR)/$(CARGO_INSTALLER_SOURCE) | \
$(TAR) --strip-components=1 -C $(@D)/src/rust-installer $(TAR_OPTIONS) -
endef
diff --git a/package/cracklib/cracklib.mk b/package/cracklib/cracklib.mk
index 2fa3d3b4ea..aeee60d1e7 100644
--- a/package/cracklib/cracklib.mk
+++ b/package/cracklib/cracklib.mk
@@ -29,7 +29,7 @@ HOST_CRACKLIB_CONF_OPTS += --without-python --without-zlib
ifeq ($(BR2_PACKAGE_CRACKLIB_FULL_DICT),y)
CRACKLIB_EXTRA_DOWNLOADS = cracklib-words-$(CRACKLIB_VERSION).gz
-CRACKLIB_DICT_SOURCE = $(DL_DIR)/cracklib-words-$(CRACKLIB_VERSION).gz
+CRACKLIB_DICT_SOURCE = $(CRACKLIB_DL_DIR)/cracklib-words-$(CRACKLIB_VERSION).gz
else
CRACKLIB_DICT_SOURCE = $(@D)/dicts/cracklib-small
endif
diff --git a/package/cryptopp/cryptopp.mk b/package/cryptopp/cryptopp.mk
index d7aabf04b8..6da39b09f2 100644
--- a/package/cryptopp/cryptopp.mk
+++ b/package/cryptopp/cryptopp.mk
@@ -12,7 +12,7 @@ CRYPTOPP_LICENSE_FILES = License.txt
CRYPTOPP_INSTALL_STAGING = YES
define HOST_CRYPTOPP_EXTRACT_CMDS
- $(UNZIP) $(DL_DIR)/$(CRYPTOPP_SOURCE) -d $(@D)
+ $(UNZIP) $(CRYPTOPP_DL_DIR)/$(CRYPTOPP_SOURCE) -d $(@D)
endef
HOST_CRYPTOPP_MAKE_OPTS = \
diff --git a/package/devmem2/devmem2.mk b/package/devmem2/devmem2.mk
index eb92db6098..044e6fa596 100644
--- a/package/devmem2/devmem2.mk
+++ b/package/devmem2/devmem2.mk
@@ -11,7 +11,7 @@ DEVMEM2_LICENSE = GPL-2.0+
DEVMEM2_LICENSE_FILES = devmem2.c.license
define DEVMEM2_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(DEVMEM2_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define DEVMEM2_EXTRACT_LICENSE
diff --git a/package/dhrystone/dhrystone.mk b/package/dhrystone/dhrystone.mk
index c0bca9895d..d54c8d94e8 100644
--- a/package/dhrystone/dhrystone.mk
+++ b/package/dhrystone/dhrystone.mk
@@ -9,7 +9,7 @@ DHRYSTONE_SOURCE = dhry-c
DHRYSTONE_SITE = http://www.netlib.org/benchmark
define DHRYSTONE_EXTRACT_CMDS
- (cd $(@D) && $(SHELL) $(DL_DIR)/$($(PKG)_SOURCE))
+ (cd $(@D) && $(SHELL) $(DHRYSTONE_DL_DIR)/$($(PKG)_SOURCE))
$(Q)cp package/dhrystone/Makefile $(@D)/
endef
diff --git a/package/doom-wad/doom-wad.mk b/package/doom-wad/doom-wad.mk
index d3ac731bd5..f348fc2207 100644
--- a/package/doom-wad/doom-wad.mk
+++ b/package/doom-wad/doom-wad.mk
@@ -11,7 +11,7 @@ DOOM_WAD_SOURCE = doom$(subst .,,$(DOOM_WAD_VERSION))s.zip
DOOM_WAD_SITE = http://www.jbserver.com/downloads/games/doom/misc/shareware
define DOOM_WAD_EXTRACT_CMDS
- $(UNZIP) -p $(DL_DIR)/$($(PKG)_SOURCE) 'DOOMS_19.[12]' > \
+ $(UNZIP) -p $(DOOM_WAD_DL_DIR)/$($(PKG)_SOURCE) 'DOOMS_19.[12]' > \
$(@D)/doom-$(DOOM_WAD_VERSION).zip
$(UNZIP) -d $(@D) $(@D)/doom-$(DOOM_WAD_VERSION).zip DOOM1.WAD
endef
diff --git a/package/espeak/espeak.mk b/package/espeak/espeak.mk
index 226f17fc05..aa0b5b0716 100644
--- a/package/espeak/espeak.mk
+++ b/package/espeak/espeak.mk
@@ -21,7 +21,7 @@ ESPEAK_DEPENDENCIES = pulseaudio
endif
define ESPEAK_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(ESPEAK_SOURCE)
+ $(UNZIP) -d $(@D) $(ESPEAK_DL_DIR)/$(ESPEAK_SOURCE)
mv $(@D)/espeak-$(ESPEAK_VERSION)-source/* $(@D)
$(RM) -r $(@D)/espeak-$(ESPEAK_VERSION)-source
endef
diff --git a/package/fan-ctrl/fan-ctrl.mk b/package/fan-ctrl/fan-ctrl.mk
index 53533a8a17..81a52f1e9f 100644
--- a/package/fan-ctrl/fan-ctrl.mk
+++ b/package/fan-ctrl/fan-ctrl.mk
@@ -12,7 +12,7 @@ FAN_CTRL_LICENSE = GPL-2.0+
FAN_CTRL_LICENSE_FILES = fan-ctrl.c
define FAN_CTRL_EXTRACT_CMDS
- cp $(DL_DIR)/$(FAN_CTRL_SOURCE) $(@D)/fan-ctrl.c
+ cp $(FAN_CTRL_DL_DIR)/$(FAN_CTRL_SOURCE) $(@D)/fan-ctrl.c
endef
define FAN_CTRL_BUILD_CMDS
diff --git a/package/freescale-imx/firmware-imx/firmware-imx.mk b/package/freescale-imx/firmware-imx/firmware-imx.mk
index 630afcaa37..d5387b38da 100644
--- a/package/freescale-imx/firmware-imx/firmware-imx.mk
+++ b/package/freescale-imx/firmware-imx/firmware-imx.mk
@@ -15,7 +15,7 @@ FIRMWARE_IMX_REDISTRIBUTE = NO
FIRMWARE_IMX_BLOBS = sdma vpu
define FIRMWARE_IMX_EXTRACT_CMDS
- $(call FREESCALE_IMX_EXTRACT_HELPER,$(DL_DIR)/$(FIRMWARE_IMX_SOURCE))
+ $(call FREESCALE_IMX_EXTRACT_HELPER,$(FIRMWARE_IMX_DL_DIR)/$(FIRMWARE_IMX_SOURCE))
endef
define FIRMWARE_IMX_INSTALL_TARGET_CMDS
diff --git a/package/freescale-imx/gpu-amd-bin-mx51/gpu-amd-bin-mx51.mk b/package/freescale-imx/gpu-amd-bin-mx51/gpu-amd-bin-mx51.mk
index 44a6b4c7d2..5586073712 100644
--- a/package/freescale-imx/gpu-amd-bin-mx51/gpu-amd-bin-mx51.mk
+++ b/package/freescale-imx/gpu-amd-bin-mx51/gpu-amd-bin-mx51.mk
@@ -23,7 +23,7 @@ GPU_AMD_BIN_MX51_LICENSE_FILES = EULA
GPU_AMD_BIN_MX51_REDISTRIBUTE = NO
define GPU_AMD_BIN_MX51_EXTRACT_CMDS
- $(call FREESCALE_IMX_EXTRACT_HELPER,$(DL_DIR)/$(GPU_AMD_BIN_MX51_SOURCE))
+ $(call FREESCALE_IMX_EXTRACT_HELPER,$(GPU_AMD_BIN_MX51_DL_DIR)/$(GPU_AMD_BIN_MX51_SOURCE))
endef
# Upstream headers need to be compiled with -D_LINUX. It is more convenient
diff --git a/package/freescale-imx/imx-codec/imx-codec.mk b/package/freescale-imx/imx-codec/imx-codec.mk
index ea4d90110f..784c1fa4a9 100644
--- a/package/freescale-imx/imx-codec/imx-codec.mk
+++ b/package/freescale-imx/imx-codec/imx-codec.mk
@@ -26,7 +26,7 @@ IMX_CODEC_CONF_OPTS += --enable-vpu
endif
define IMX_CODEC_EXTRACT_CMDS
- $(call FREESCALE_IMX_EXTRACT_HELPER,$(DL_DIR)/$(IMX_CODEC_SOURCE))
+ $(call FREESCALE_IMX_EXTRACT_HELPER,$(IMX_CODEC_DL_DIR)/$(IMX_CODEC_SOURCE))
endef
# FIXME The Makefile installs both the arm9 and arm11 versions of the
diff --git a/package/freescale-imx/imx-gpu-g2d/imx-gpu-g2d.mk b/package/freescale-imx/imx-gpu-g2d/imx-gpu-g2d.mk
index 0c92b826c4..aafd70dc9d 100644
--- a/package/freescale-imx/imx-gpu-g2d/imx-gpu-g2d.mk
+++ b/package/freescale-imx/imx-gpu-g2d/imx-gpu-g2d.mk
@@ -15,7 +15,7 @@ IMX_GPU_G2D_LICENSE_FILES = EULA COPYING
IMX_GPU_G2D_REDISTRIBUTE = NO
define IMX_GPU_G2D_EXTRACT_CMDS
- $(call FREESCALE_IMX_EXTRACT_HELPER,$(DL_DIR)/$(IMX_GPU_G2D_SOURCE))
+ $(call FREESCALE_IMX_EXTRACT_HELPER,$(IMX_GPU_G2D_DL_DIR)/$(IMX_GPU_G2D_SOURCE))
endef
define IMX_GPU_G2D_INSTALL_STAGING_CMDS
diff --git a/package/freescale-imx/imx-gpu-viv/imx-gpu-viv.mk b/package/freescale-imx/imx-gpu-viv/imx-gpu-viv.mk
index a9f8b9d9bd..0dc2072984 100644
--- a/package/freescale-imx/imx-gpu-viv/imx-gpu-viv.mk
+++ b/package/freescale-imx/imx-gpu-viv/imx-gpu-viv.mk
@@ -26,7 +26,7 @@ IMX_GPU_VIV_DEPENDENCIES += xlib_libXdamage xlib_libXext xlib_libXfixes
endif
define IMX_GPU_VIV_EXTRACT_CMDS
- $(call FREESCALE_IMX_EXTRACT_HELPER,$(DL_DIR)/$(IMX_GPU_VIV_SOURCE))
+ $(call FREESCALE_IMX_EXTRACT_HELPER,$(IMX_GPU_VIV_DL_DIR)/$(IMX_GPU_VIV_SOURCE))
endef
# Instead of building, we fix up the inconsistencies that exist
diff --git a/package/freescale-imx/imx-parser/imx-parser.mk b/package/freescale-imx/imx-parser/imx-parser.mk
index d79a4444c8..11536e5024 100644
--- a/package/freescale-imx/imx-parser/imx-parser.mk
+++ b/package/freescale-imx/imx-parser/imx-parser.mk
@@ -24,7 +24,7 @@ IMX_PARSER_CONF_OPTS += --enable-fsw
endif
define IMX_PARSER_EXTRACT_CMDS
- $(call FREESCALE_IMX_EXTRACT_HELPER,$(DL_DIR)/$(IMX_PARSER_SOURCE))
+ $(call FREESCALE_IMX_EXTRACT_HELPER,$(IMX_PARSER_DL_DIR)/$(IMX_PARSER_SOURCE))
endef
# The Makefile installs several versions of the libraries, but we only
diff --git a/package/freescale-imx/imx-vpu/imx-vpu.mk b/package/freescale-imx/imx-vpu/imx-vpu.mk
index e3a1ee661d..2bcfe53eba 100644
--- a/package/freescale-imx/imx-vpu/imx-vpu.mk
+++ b/package/freescale-imx/imx-vpu/imx-vpu.mk
@@ -21,7 +21,7 @@ IMX_VPU_LICENSE_FILES = EULA COPYING
IMX_VPU_REDISTRIBUTE = NO
define IMX_VPU_EXTRACT_CMDS
- $(call FREESCALE_IMX_EXTRACT_HELPER,$(DL_DIR)/$(IMX_VPU_SOURCE))
+ $(call FREESCALE_IMX_EXTRACT_HELPER,$(IMX_VPU_DL_DIR)/$(IMX_VPU_SOURCE))
endef
define IMX_VPU_BUILD_CMDS
diff --git a/package/freescale-imx/imx-vpuwrap/imx-vpuwrap.mk b/package/freescale-imx/imx-vpuwrap/imx-vpuwrap.mk
index 7cbf7784c9..edba87279f 100644
--- a/package/freescale-imx/imx-vpuwrap/imx-vpuwrap.mk
+++ b/package/freescale-imx/imx-vpuwrap/imx-vpuwrap.mk
@@ -15,7 +15,7 @@ IMX_VPUWRAP_LICENSE_FILES = EULA COPYING
IMX_VPUWRAP_REDISTRIBUTE = NO
define IMX_VPUWRAP_EXTRACT_CMDS
- $(call FREESCALE_IMX_EXTRACT_HELPER,$(DL_DIR)/$(IMX_VPUWRAP_SOURCE))
+ $(call FREESCALE_IMX_EXTRACT_HELPER,$(IMX_VPUWRAP_DL_DIR)/$(IMX_VPUWRAP_SOURCE))
endef
$(eval $(autotools-package))
diff --git a/package/freescale-imx/libz160/libz160.mk b/package/freescale-imx/libz160/libz160.mk
index af43b1859d..5b1feeb441 100644
--- a/package/freescale-imx/libz160/libz160.mk
+++ b/package/freescale-imx/libz160/libz160.mk
@@ -15,7 +15,7 @@ LIBZ160_LICENSE_FILES = EULA
LIBZ160_REDISTRIBUTE = NO
define LIBZ160_EXTRACT_CMDS
- $(call FREESCALE_IMX_EXTRACT_HELPER,$(DL_DIR)/$(LIBZ160_SOURCE))
+ $(call FREESCALE_IMX_EXTRACT_HELPER,$(LIBZ160_DL_DIR)/$(LIBZ160_SOURCE))
endef
define LIBZ160_INSTALL_STAGING_CMDS
diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
index 27fc1e987c..f42f36a967 100644
--- a/package/gcc/gcc.mk
+++ b/package/gcc/gcc.mk
@@ -316,7 +316,7 @@ HOST_GCC_COMMON_MAKE_OPTS = \
gcc_cv_libc_provides_ssp=$(if $(BR2_TOOLCHAIN_HAS_SSP),yes,no)
ifeq ($(BR2_CCACHE),y)
-HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE)
+HOST_GCC_COMMON_CCACHE_HASH_FILES += $(GCC_DL_DIR)/$(GCC_SOURCE)
# Cfr. PATCH_BASE_DIRS in .stamp_patched, but we catch both versioned
# and unversioned patches unconditionally. Moreover, to facilitate the
diff --git a/package/irrlicht/irrlicht.mk b/package/irrlicht/irrlicht.mk
index ccd1045ca0..a1e190d98d 100644
--- a/package/irrlicht/irrlicht.mk
+++ b/package/irrlicht/irrlicht.mk
@@ -25,7 +25,7 @@ IRRLICHT_SUBDIR = source/Irrlicht
IRRLICHT_DEPENDENCIES = libgl xlib_libXxf86vm
define IRRLICHT_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(IRRLICHT_SOURCE)
+ $(UNZIP) -d $(@D) $(IRRLICHT_DL_DIR)/$(IRRLICHT_SOURCE)
mv $(@D)/irrlicht-$(IRRLICHT_VERSION)/* $(@D)
$(RM) -r $(@D)/irrlicht-$(IRRLICHT_VERSION)
endef
diff --git a/package/jquery-mobile/jquery-mobile.mk b/package/jquery-mobile/jquery-mobile.mk
index 1067952904..1171e60d0a 100644
--- a/package/jquery-mobile/jquery-mobile.mk
+++ b/package/jquery-mobile/jquery-mobile.mk
@@ -10,7 +10,7 @@ JQUERY_MOBILE_SOURCE = jquery.mobile-$(JQUERY_MOBILE_VERSION).zip
JQUERY_MOBILE_LICENSE = MIT
define JQUERY_MOBILE_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_MOBILE_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_MOBILE_DL_DIR)/$(JQUERY_MOBILE_SOURCE)
endef
JQUERY_MOBILE_INSTALLED_FILES = \
diff --git a/package/jquery-sparkline/jquery-sparkline.mk b/package/jquery-sparkline/jquery-sparkline.mk
index 165c2fa99c..683d84f4aa 100644
--- a/package/jquery-sparkline/jquery-sparkline.mk
+++ b/package/jquery-sparkline/jquery-sparkline.mk
@@ -10,7 +10,7 @@ JQUERY_SPARKLINE_SOURCE = jquery.sparkline.min.js
JQUERY_SPARKLINE_LICENSE = BSD-3-Clause
define JQUERY_SPARKLINE_EXTRACT_CMDS
- cp $(DL_DIR)/$(JQUERY_SPARKLINE_SOURCE) $(@D)
+ cp $(JQUERY_SPARKLINE_DL_DIR)/$(JQUERY_SPARKLINE_SOURCE) $(@D)
endef
define JQUERY_SPARKLINE_INSTALL_TARGET_CMDS
diff --git a/package/jquery-ui-themes/jquery-ui-themes.mk b/package/jquery-ui-themes/jquery-ui-themes.mk
index cd3dab6d87..5108389a77 100644
--- a/package/jquery-ui-themes/jquery-ui-themes.mk
+++ b/package/jquery-ui-themes/jquery-ui-themes.mk
@@ -12,7 +12,7 @@ JQUERY_UI_THEMES_LICENSE_FILES = MIT-LICENSE.txt
JQUERY_UI_THEMES_DEPENDENCIES = jquery-ui
define JQUERY_UI_THEMES_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_UI_THEMES_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_UI_THEMES_DL_DIR)/$(JQUERY_UI_THEMES_SOURCE)
mv $(@D)/jquery-ui-themes-$(JQUERY_UI_THEMES_VERSION)/* $(@D)
$(RM) -r $(@D)/jquery-ui-themes-$(JQUERY_UI_THEMES_VERSION)
endef
diff --git a/package/jquery-ui/jquery-ui.mk b/package/jquery-ui/jquery-ui.mk
index d829d6241b..9284e683f1 100644
--- a/package/jquery-ui/jquery-ui.mk
+++ b/package/jquery-ui/jquery-ui.mk
@@ -14,7 +14,7 @@ JQUERY_UI_LICENSE = MIT
JQUERY_UI_LICENSE_FILES = MIT-LICENSE.txt
define JQUERY_UI_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_UI_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_UI_DL_DIR)/$(JQUERY_UI_SOURCE)
mv $(@D)/jquery-ui-$(JQUERY_UI_VERSION)/* $(@D)
$(RM) -r $(@D)/jquery-ui-$(JQUERY_UI_VERSION)
endef
diff --git a/package/jquery-validation/jquery-validation.mk b/package/jquery-validation/jquery-validation.mk
index 10d2a92afc..a8c2897983 100644
--- a/package/jquery-validation/jquery-validation.mk
+++ b/package/jquery-validation/jquery-validation.mk
@@ -11,7 +11,7 @@ JQUERY_VALIDATION_LICENSE = MIT
JQUERY_VALIDATION_LICENSE_FILES = README.md
define JQUERY_VALIDATION_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_VALIDATION_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_VALIDATION_DL_DIR)/$(JQUERY_VALIDATION_SOURCE)
endef
define JQUERY_VALIDATION_INSTALL_TARGET_CMDS
diff --git a/package/jquery/jquery.mk b/package/jquery/jquery.mk
index 363ee09633..f75eded8b3 100644
--- a/package/jquery/jquery.mk
+++ b/package/jquery/jquery.mk
@@ -10,7 +10,7 @@ JQUERY_SOURCE = jquery-$(JQUERY_VERSION).min.js
JQUERY_LICENSE = MIT
define JQUERY_EXTRACT_CMDS
- cp $(DL_DIR)/$(JQUERY_SOURCE) $(@D)
+ cp $(JQUERY_DL_DIR)/$(JQUERY_SOURCE) $(@D)
endef
define JQUERY_INSTALL_TARGET_CMDS
diff --git a/package/kodi/kodi.mk b/package/kodi/kodi.mk
index 9d3b65ae91..f3fa0b55b0 100644
--- a/package/kodi/kodi.mk
+++ b/package/kodi/kodi.mk
@@ -69,9 +69,9 @@ KODI_CONF_OPTS += \
-DDEPENDS_PATH=$(@D) \
-DWITH_FFMPEG=$(STAGING_DIR)/usr \
-DWITH_TEXTUREPACKER=$(HOST_DIR)/bin/TexturePacker \
- -DLIBDVDCSS_URL=$(DL_DIR)/$(KODI_LIBDVDCSS_VERSION).tar.gz \
- -DLIBDVDNAV_URL=$(DL_DIR)/$(KODI_LIBDVDNAV_VERSION).tar.gz \
- -DLIBDVDREAD_URL=$(DL_DIR)/$(KODI_LIBDVDREAD_VERSION).tar.gz \
+ -DLIBDVDCSS_URL=$(KODI_DL_DIR)/$(KODI_LIBDVDCSS_VERSION).tar.gz \
+ -DLIBDVDNAV_URL=$(KODI_DL_DIR)/$(KODI_LIBDVDNAV_VERSION).tar.gz \
+ -DLIBDVDREAD_URL=$(KODI_DL_DIR)/$(KODI_LIBDVDREAD_VERSION).tar.gz \
-DENABLE_IMX=OFF
ifeq ($(BR2_ENABLE_LOCALE),)
diff --git a/package/libb64/libb64.mk b/package/libb64/libb64.mk
index 4dea9593a0..ed6d3cf4b4 100644
--- a/package/libb64/libb64.mk
+++ b/package/libb64/libb64.mk
@@ -14,7 +14,7 @@ LIBB64_INSTALL_STAGING = YES
LIBB64_INSTALL_TARGET = NO
define LIBB64_EXTRACT_CMDS
- unzip $(DL_DIR)/$(LIBB64_SOURCE) -d $(BUILD_DIR)
+ unzip $(LIBB64_DL_DIR)/$(LIBB64_SOURCE) -d $(BUILD_DIR)
endef
define LIBB64_BUILD_CMDS
diff --git a/package/libfreeimage/libfreeimage.mk b/package/libfreeimage/libfreeimage.mk
index 0ca23933a6..e0aa1f0ae8 100644
--- a/package/libfreeimage/libfreeimage.mk
+++ b/package/libfreeimage/libfreeimage.mk
@@ -12,7 +12,7 @@ LIBFREEIMAGE_LICENSE_FILES = license-gplv2.txt license-gplv3.txt license-fi.txt
LIBFREEIMAGE_INSTALL_STAGING = YES
define LIBFREEIMAGE_EXTRACT_CMDS
- $(UNZIP) $(DL_DIR)/$(LIBFREEIMAGE_SOURCE) -d $(@D)
+ $(UNZIP) $(LIBFREEIMAGE_DL_DIR)/$(LIBFREEIMAGE_SOURCE) -d $(@D)
mv $(@D)/FreeImage/* $(@D)
rmdir $(@D)/FreeImage
endef
diff --git a/package/libjson/libjson.mk b/package/libjson/libjson.mk
index 74224ba657..d04ddc40f3 100644
--- a/package/libjson/libjson.mk
+++ b/package/libjson/libjson.mk
@@ -23,7 +23,7 @@ endif
LIBJSON_MAKE_OPTS += BUILD_TYPE= CXXFLAGS="$(LIBJSON_CXXFLAGS)"
define LIBJSON_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(LIBJSON_SOURCE)
+ $(UNZIP) -d $(@D) $(LIBJSON_DL_DIR)/$(LIBJSON_SOURCE)
mv $(@D)/libjson/* $(@D)
$(RM) -r $(@D)/libjson
$(SED) '/ldconfig/d' $(@D)/makefile
diff --git a/package/libsoil/libsoil.mk b/package/libsoil/libsoil.mk
index efa67d9eee..2945edd8dd 100644
--- a/package/libsoil/libsoil.mk
+++ b/package/libsoil/libsoil.mk
@@ -14,7 +14,7 @@ LIBSOIL_LICENSE_FILES = src/stb_image_aug.c src/image_helper.c
LIBSOIL_MAKEFILE = ../projects/makefile/alternate_Makefile.txt
define LIBSOIL_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(LIBSOIL_SOURCE)
+ $(UNZIP) -d $(@D) $(LIBSOIL_DL_DIR)/$(LIBSOIL_SOURCE)
mv $(@D)/Simple\ OpenGL\ Image\ Library/* $(@D)
endef
diff --git a/package/lsof/lsof.mk b/package/lsof/lsof.mk
index e5cd4bce6e..0dc8e2de2f 100644
--- a/package/lsof/lsof.mk
+++ b/package/lsof/lsof.mk
@@ -32,7 +32,7 @@ endif
# The .tar.bz2 contains another .tar, which contains the source code.
define LSOF_EXTRACT_CMDS
- $(call suitable-extractor,$(LSOF_SOURCE)) $(DL_DIR)/$(LSOF_SOURCE) | \
+ $(call suitable-extractor,$(LSOF_SOURCE)) $(LSOF_DL_DIR)/$(LSOF_SOURCE) | \
$(TAR) -O $(TAR_OPTIONS) - lsof_$(LSOF_VERSION)/lsof_$(LSOF_VERSION)_src.tar | \
$(TAR) --strip-components=1 -C $(LSOF_DIR) $(TAR_OPTIONS) -
endef
--git a/package/musl-compat-headers/musl-compat-headers.mk b/package/musl-compat-headers/musl-compat-headers.mk
index 91f5074ef6..1cfa894879 100644
--- a/package/musl-compat-headers/musl-compat-headers.mk
+++ b/package/musl-compat-headers/musl-compat-headers.mk
@@ -20,7 +20,7 @@ MUSL_COMPAT_HEADERS_INSTALL_STAGING = YES
# Copying both headers so legal-info finds them (they are _LICENSE_FILES)
define MUSL_COMPAT_HEADERS_EXTRACT_CMDS
- $(INSTALL) -m 0644 -D $(DL_DIR)/$(notdir $(MUSL_COMPAT_HEADERS_QUEUE_H)) $(@D)/queue.h
+ $(INSTALL) -m 0644 -D $(MUSL_COMPAT_HEADERS_DL_DIR)/$(notdir $(MUSL_COMPAT_HEADERS_QUEUE_H)) $(@D)/queue.h
$(INSTALL) -m 0644 -D $(MUSL_COMPAT_HEADERS_PKGDIR)/cdefs.h $(@D)/cdefs.h
endef
diff --git a/package/nmon/nmon.mk b/package/nmon/nmon.mk
index f561d6dcc0..f7edd66a26 100644
--- a/package/nmon/nmon.mk
+++ b/package/nmon/nmon.mk
@@ -13,7 +13,7 @@ NMON_DEPENDENCIES = ncurses
NMON_CFLAGS = $(TARGET_CFLAGS) -D JFS -D GETUSER -D LARGEMEM -D DEBIAN
define NMON_EXTRACT_CMDS
- cp $(DL_DIR)/$(NMON_SOURCE) $(@D)
+ cp $(NMON_DL_DIR)/$(NMON_SOURCE) $(@D)
endef
define NMON_BUILD_CMDS
diff --git a/package/nvidia-driver/nvidia-driver.mk b/package/nvidia-driver/nvidia-driver.mk
index e56661059f..404fa881ec 100644
--- a/package/nvidia-driver/nvidia-driver.mk
+++ b/package/nvidia-driver/nvidia-driver.mk
@@ -144,7 +144,7 @@ endif # BR2_PACKAGE_NVIDIA_DRIVER_MODULE == y
# virtually everywhere, and it is fine enough to provide useful options.
# Except it can't extract into an existing (even empty) directory.
define NVIDIA_DRIVER_EXTRACT_CMDS
- $(SHELL) $(DL_DIR)/$(NVIDIA_DRIVER_SOURCE) --extract-only --target \
+ $(SHELL) $(NVIDIA_DRIVER_DL_DIR)/$(NVIDIA_DRIVER_SOURCE) --extract-only --target \
$(@D)/tmp-extract
chmod u+w -R $(@D)
mv $(@D)/tmp-extract/* $(@D)/tmp-extract/.manifest $(@D)
diff --git a/package/nvidia-tegra23/nvidia-tegra23-codecs/nvidia-tegra23-codecs.mk b/package/nvidia-tegra23/nvidia-tegra23-codecs/nvidia-tegra23-codecs.mk
index 5514643416..2885021c58 100644
--- a/package/nvidia-tegra23/nvidia-tegra23-codecs/nvidia-tegra23-codecs.mk
+++ b/package/nvidia-tegra23/nvidia-tegra23-codecs/nvidia-tegra23-codecs.mk
@@ -15,7 +15,7 @@ NVIDIA_TEGRA23_CODECS_REDISTRIBUTE = NO
define NVIDIA_TEGRA23_CODECS_EXTRACT_CMDS
$(INSTALL) -d $(@D)
$(call suitable-extractor,$(NVIDIA_TEGRA23_CODECS_SOURCE)) \
- $(DL_DIR)/$(NVIDIA_TEGRA23_CODECS_SOURCE) | \
+ $(NVIDIA_TEGRA23_CODECS_DL_DIR)/$(NVIDIA_TEGRA23_CODECS_SOURCE) | \
$(TAR) --strip-components=0 -C $(@D) $(TAR_OPTIONS) -
$(INSTALL) -d $(@D)/restricted_codecs
$(call suitable-extractor,$(@D)/restricted_codecs.tbz2) \
diff --git a/package/openobex/openobex.mk b/package/openobex/openobex.mk
index ea41f65a80..c19780110c 100644
--- a/package/openobex/openobex.mk
+++ b/package/openobex/openobex.mk
@@ -19,6 +19,12 @@ ifeq ($(BR2_PACKAGE_BLUEZ_UTILS),y)
OPENOBEX_DEPENDENCIES += bluez_utils
endif
+define OPENOBEX_EXTRACT_CMDS
+ $(UNZIP) -d $(@D) $(OPENOBEX_DL_DIR)/$(OPENOBEX_SOURCE)
+ mv $(@D)/openobex-$(OPENOBEX_VERSION)-Source/* $(@D)
+ $(RM) -r $(@D)/openobex-$(OPENOBEX_VERSION)-Source
+endef
+
ifeq ($(BR2_PACKAGE_BLUEZ5_UTILS),y)
OPENOBEX_DEPENDENCIES += bluez5_utils
endif
diff --git a/package/opentyrian-data/opentyrian-data.mk b/package/opentyrian-data/opentyrian-data.mk
index 4b879df1c3..9e38b61bfa 100644
--- a/package/opentyrian-data/opentyrian-data.mk
+++ b/package/opentyrian-data/opentyrian-data.mk
@@ -10,7 +10,7 @@ OPENTYRIAN_DATA_SOURCE = tyrian21.zip
OPENTYRIAN_DATA_LICENSE = Freeware
define OPENTYRIAN_DATA_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(OPENTYRIAN_DATA_SOURCE)
+ $(UNZIP) -d $(@D) $(OPENTYRIAN_DATA_DL_DIR)/$(OPENTYRIAN_DATA_SOURCE)
endef
define OPENTYRIAN_DATA_INSTALL_TARGET_CMDS
diff --git a/package/perl/perl.mk b/package/perl/perl.mk
index 58bf3eb6ae..c47367b9da 100644
--- a/package/perl/perl.mk
+++ b/package/perl/perl.mk
@@ -24,7 +24,7 @@ PERL_EXTRA_DOWNLOADS = $(PERL_CROSS_SITE)/$(PERL_CROSS_SOURCE)
# as a separate package. Instead, it is downloaded and extracted
# together with perl
define PERL_CROSS_EXTRACT
- $(call suitable-extractor,$(PERL_CROSS_SOURCE)) $(DL_DIR)/$(PERL_CROSS_SOURCE) | \
+ $(call suitable-extractor,$(PERL_CROSS_SOURCE)) $(PERL_DL_DIR)/$(PERL_CROSS_SOURCE) | \
$(TAR) --strip-components=1 -C $(@D) $(TAR_OPTIONS) -
endef
PERL_POST_EXTRACT_HOOKS += PERL_CROSS_EXTRACT
diff --git a/package/python-keyring/python-keyring.mk b/package/python-keyring/python-keyring.mk
index 27db2802ed..8ac18ae052 100644
--- a/package/python-keyring/python-keyring.mk
+++ b/package/python-keyring/python-keyring.mk
@@ -6,7 +6,6 @@
PYTHON_KEYRING_VERSION = 10.5.0
PYTHON_KEYRING_SOURCE = keyring-$(PYTHON_KEYRING_VERSION).tar.gz
-PYTHON_KEYRING_SITE = https://pypi.python.org/packages/42/2e/51bd1739fe335095a2174db3f2f230346762e7e572471059540146a521f6
PYTHON_KEYRING_SETUP_TYPE = setuptools
PYTHON_KEYRING_LICENSE = MIT
PYTHON_KEYRING_LICENSE_FILES = LICENSE
diff --git a/package/python-pytz/python-pytz.mk b/package/python-pytz/python-pytz.mk
index 6e130a6b2f..1d3603389b 100644
--- a/package/python-pytz/python-pytz.mk
+++ b/package/python-pytz/python-pytz.mk
@@ -12,7 +12,7 @@ PYTHON_PYTZ_LICENSE = MIT
PYTHON_PYTZ_LICENSE_FILES = LICENSE.txt
define PYTHON_PYTZ_EXTRACT_CMDS
- unzip $(DL_DIR)/$(PYTHON_PYTZ_SOURCE) -d $(@D)
+ unzip $(PYTHON_PYTZ_DL_DIR)/$(PYTHON_PYTZ_SOURCE) -d $(@D)
mv $(@D)/pytz-$(PYTHON_PYTZ_VERSION)/* $(@D)
rmdir $(@D)/pytz-$(PYTHON_PYTZ_VERSION)
endef
diff --git a/package/python-simplegeneric/python-simplegeneric.mk b/package/python-simplegeneric/python-simplegeneric.mk
index cc84320428..d3bfa69a7e 100644
--- a/package/python-simplegeneric/python-simplegeneric.mk
+++ b/package/python-simplegeneric/python-simplegeneric.mk
@@ -16,7 +16,7 @@ PYTHON_SIMPLEGENERIC_LICENSE = ZPL-2.1
PYTHON_SIMPLEGENERIC_SETUP_TYPE = setuptools
define PYTHON_SIMPLEGENERIC_EXTRACT_CMDS
- unzip $(DL_DIR)/$(PYTHON_SIMPLEGENERIC_SOURCE) -d $(@D)
+ unzip $(PYTHON_SIMPLEGENERIC_DL_DIR)/$(PYTHON_SIMPLEGENERIC_SOURCE) -d $(@D)
mv $(@D)/simplegeneric-$(PYTHON_SIMPLEGENERIC_VERSION)/* $(@D)
rmdir $(@D)/simplegeneric-$(PYTHON_SIMPLEGENERIC_VERSION)
endef
diff --git a/package/rapidxml/rapidxml.mk b/package/rapidxml/rapidxml.mk
index 2bec8fe1f2..9d034d8205 100644
--- a/package/rapidxml/rapidxml.mk
+++ b/package/rapidxml/rapidxml.mk
@@ -15,7 +15,7 @@ RAPIDXML_INSTALL_TARGET = NO
RAPIDXML_INSTALL_STAGING = YES
define RAPIDXML_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(RAPIDXML_SOURCE)
+ $(UNZIP) -d $(@D) $(RAPIDXML_DL_DIR)/$(RAPIDXML_SOURCE)
mv $(@D)/rapidxml-$(RAPIDXML_VERSION)/* $(@D)/
rmdir $(@D)/rapidxml-$(RAPIDXML_VERSION)
endef
diff --git a/package/rpi-wifi-firmware/rpi-wifi-firmware.mk b/package/rpi-wifi-firmware/rpi-wifi-firmware.mk
index 6c855a8e14..83473d5bb9 100644
--- a/package/rpi-wifi-firmware/rpi-wifi-firmware.mk
+++ b/package/rpi-wifi-firmware/rpi-wifi-firmware.mk
@@ -12,7 +12,7 @@ RPI_WIFI_FIRMWARE_SITE = https://raw.githubusercontent.com/RPi-Distro/firmware-n
RPI_WIFI_FIRMWARE_LICENSE = PROPRIETARY
define RPI_WIFI_FIRMWARE_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(RPI_WIFI_FIRMWARE_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define RPI_WIFI_FIRMWARE_INSTALL_TARGET_CMDS
diff --git a/package/rust-bin/rust-bin.mk b/package/rust-bin/rust-bin.mk
index cd5844b115..759d468304 100644
--- a/package/rust-bin/rust-bin.mk
+++ b/package/rust-bin/rust-bin.mk
@@ -22,7 +22,7 @@ HOST_RUST_BIN_LIBSTD_HOST_PREFIX = rust-std-$(RUST_BIN_VERSION)-$(RUSTC_HOST_NAM
define HOST_RUST_BIN_LIBSTD_EXTRACT
mkdir -p $(@D)/std
$(foreach f,$(HOST_RUST_BIN_EXTRA_DOWNLOADS), \
- $(call suitable-extractor,$(f)) $(DL_DIR)/$(f) | \
+ $(call suitable-extractor,$(f)) $(RUST_BIN_DL_DIR)/$(f) | \
$(TAR) -C $(@D)/std $(TAR_OPTIONS) -
)
cd $(@D)/rustc/lib/rustlib; \
diff --git a/package/sam-ba/sam-ba.mk b/package/sam-ba/sam-ba.mk
index a7941459c7..acc03a3b6b 100644
--- a/package/sam-ba/sam-ba.mk
+++ b/package/sam-ba/sam-ba.mk
@@ -13,7 +13,7 @@ SAM_BA_LICENSE_FILES = doc/license.txt tcl_lib/boards.tcl \
applets/sam4c/libraries/libchip_sam4c/include/sam4c/sam4c32e-1.h
define HOST_SAM_BA_EXTRACT_CMDS
- $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(SAM_BA_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR) $(SAM_BA_DL_DIR)/$(SAM_BA_SOURCE)
mv $(BUILD_DIR)/sam-ba_cdc_linux/* $(@D)
rmdir $(BUILD_DIR)/sam-ba_cdc_linux/
endef
diff --git a/package/spidev_test/spidev_test.mk b/package/spidev_test/spidev_test.mk
index 1d657803b4..bf8170cd2c 100644
--- a/package/spidev_test/spidev_test.mk
+++ b/package/spidev_test/spidev_test.mk
@@ -32,7 +32,7 @@ endef
SPIDEV_TEST_POST_PATCH_HOOKS += SPIDEV_ADD_LINUX_IOCTL
define SPIDEV_TEST_EXTRACT_CMDS
- cp $(DL_DIR)/$(SPIDEV_TEST_SOURCE) $(@D)/spidev_test.c
+ cp $(SPIDEV_TEST_DL_DIR)/$(SPIDEV_TEST_SOURCE) $(@D)/spidev_test.c
endef
define SPIDEV_TEST_BUILD_CMDS
diff --git a/package/tar/tar.mk b/package/tar/tar.mk
index 92b6e9eaa5..9942e77737 100644
--- a/package/tar/tar.mk
+++ b/package/tar/tar.mk
@@ -40,7 +40,7 @@ HOST_TAR_SOURCE = tar-$(TAR_VERSION).cpio.gz
define HOST_TAR_EXTRACT_CMDS
mkdir -p $(@D)
cd $(@D) && \
- $(call suitable-extractor,$(HOST_TAR_SOURCE)) $(DL_DIR)/$(HOST_TAR_SOURCE) | cpio -i --preserve-modification-time
+ $(call suitable-extractor,$(HOST_TAR_SOURCE)) $(TAR_DL_DIR)/$(HOST_TAR_SOURCE) | cpio -i --preserve-modification-time
mv $(@D)/tar-$(TAR_VERSION)/* $(@D)
rmdir $(@D)/tar-$(TAR_VERSION)
endef
diff --git a/package/tesseract-ocr/tesseract-ocr.mk b/package/tesseract-ocr/tesseract-ocr.mk
index 9e315b239a..919a4360ed 100644
--- a/package/tesseract-ocr/tesseract-ocr.mk
+++ b/package/tesseract-ocr/tesseract-ocr.mk
@@ -62,7 +62,7 @@ TESSERACT_OCR_PRE_CONFIGURE_HOOKS += TESSERACT_OCR_PRECONFIGURE
# Language data files installation
define TESSERACT_OCR_INSTALL_LANG_DATA
$(foreach langfile,$(TESSERACT_OCR_DATA_FILES), \
- $(INSTALL) -D -m 0644 $(DL_DIR)/$(langfile) \
+ $(INSTALL) -D -m 0644 $(TESSERACT_OCR_DL_DIR)/$(langfile) \
$(TARGET_DIR)/usr/share/tessdata/$(langfile)
)
endef
diff --git a/package/ti-cgt-pru/ti-cgt-pru.mk b/package/ti-cgt-pru/ti-cgt-pru.mk
index 0b9a64b3c9..df3b97bef5 100644
--- a/package/ti-cgt-pru/ti-cgt-pru.mk
+++ b/package/ti-cgt-pru/ti-cgt-pru.mk
@@ -13,8 +13,8 @@ TI_CGT_PRU_LICENSE_FILES = PRU_Code_Generation_Tools_2.2.x_manifest.html \
pru_rts_2_2_0_82167478-F8C9-49b2-82BD-12F8550770F9.spdx
define HOST_TI_CGT_PRU_EXTRACT_CMDS
- chmod +x $(DL_DIR)/$(TI_CGT_PRU_SOURCE)
- $(DL_DIR)/$(TI_CGT_PRU_SOURCE) --prefix $(@D) --mode unattended
+ chmod +x $(TI_CGT_PRU_DL_DIR)/$(TI_CGT_PRU_SOURCE)
+ $(TI_CGT_PRU_DL_DIR)/$(TI_CGT_PRU_SOURCE) --prefix $(@D) --mode unattended
mv $(@D)/ti-cgt-pru_$(TI_CGT_PRU_VERSION)/* $(@D)
rmdir $(@D)/ti-cgt-pru_$(TI_CGT_PRU_VERSION)/
endef
diff --git a/package/ti-gfx/ti-gfx.mk b/package/ti-gfx/ti-gfx.mk
index 428878a08b..9fad553780 100644
--- a/package/ti-gfx/ti-gfx.mk
+++ b/package/ti-gfx/ti-gfx.mk
@@ -94,8 +94,8 @@ TI_GFX_HDR_DIRS = OGLES2/EGL OGLES2/EWS OGLES2/GLES2 OGLES2/KHR \
OGLES/GLES bufferclass_ti/ pvr2d/ wsegl/
define TI_GFX_EXTRACT_CMDS
- chmod +x $(DL_DIR)/$(TI_GFX_SOURCE)
- printf "Y\nY\n qY\n\n" | $(DL_DIR)/$(TI_GFX_SOURCE) \
+ chmod +x $(TI_GFX_DL_DIR)/$(TI_GFX_SOURCE)
+ printf "Y\nY\n qY\n\n" | $(TI_GFX_DL_DIR)/$(TI_GFX_SOURCE) \
--prefix $(@D) \
--mode console
endef
diff --git a/package/ts4900-fpga/ts4900-fpga.mk b/package/ts4900-fpga/ts4900-fpga.mk
index ed951b8c8f..7bb62a3984 100644
--- a/package/ts4900-fpga/ts4900-fpga.mk
+++ b/package/ts4900-fpga/ts4900-fpga.mk
@@ -11,7 +11,7 @@ TS4900_FPGA_SITE = ftp://ftp.embeddedarm.com/ts-socket-macrocontrollers/ts-4900-
# https://github.com/embeddedarm/meta-ts/blob/f31860f1204b64f765a5380d3b93a2cf18234f90/recipes-extras/ts4900-fpga/ts4900-fpga.bb#L6
define TS4900_FPGA_EXTRACT_CMDS
- cp $(DL_DIR)/$(TS4900_FPGA_SOURCE) $(@D)
+ cp $(TS4900_FPGA_DL_DIR)/$(TS4900_FPGA_SOURCE) $(@D)
endef
define TS4900_FPGA_INSTALL_TARGET_CMDS
diff --git a/package/unscd/unscd.mk b/package/unscd/unscd.mk
index f0eb5d6ad6..36f2de271b 100644
--- a/package/unscd/unscd.mk
+++ b/package/unscd/unscd.mk
@@ -11,7 +11,7 @@ UNSCD_LICENSE = GPL-2.0
UNSCD_LICENSE_FILES = $(UNSCD_SOURCE)
define UNSCD_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(UNSCD_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define UNSCD_BUILD_CMDS
diff --git a/package/urg/urg.mk b/package/urg/urg.mk
index 966627fe36..1346079893 100644
--- a/package/urg/urg.mk
+++ b/package/urg/urg.mk
@@ -25,7 +25,9 @@ endif
URG_CONFIG_SCRIPTS = c_urg-config urg-config
define URG_EXTRACT_CMDS
- $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(URG_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR) $(URG_DL_DIR)/$(URG_SOURCE)
+ test -d $(URG_DIR) || \
+ mv $(BUILD_DIR)/$(subst .zip,,$(URG_SOURCE)) $(URG_DIR)
endef
$(eval $(autotools-package))
diff --git a/package/waf/waf.mk b/package/waf/waf.mk
index e7ac891b39..639c26a3b0 100644
--- a/package/waf/waf.mk
+++ b/package/waf/waf.mk
@@ -9,7 +9,7 @@ WAF_SOURCE = waf-$(WAF_VERSION)
WAF_SITE = https://waf.io/
define HOST_WAF_EXTRACT_CMDS
- $(INSTALL) -D -m 0755 $(DL_DIR)/waf-$(WAF_VERSION) $(@D)/waf
+ $(INSTALL) -D -m 0755 $(WAF_DL_DIR)/waf-$(WAF_VERSION) $(@D)/waf
endef
define HOST_WAF_INSTALL_CMDS
diff --git a/package/whetstone/whetstone.mk b/package/whetstone/whetstone.mk
index d9b45638c5..3b6ec3419b 100644
--- a/package/whetstone/whetstone.mk
+++ b/package/whetstone/whetstone.mk
@@ -9,7 +9,7 @@ WHETSTONE_SOURCE = whetstone.c
WHETSTONE_SITE = http://www.netlib.org/benchmark
define WHETSTONE_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(WHETSTONE_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define WHETSTONE_BUILD_CMDS
diff --git a/package/wilc1000-firmware/wilc1000-firmware.mk b/package/wilc1000-firmware/wilc1000-firmware.mk
index 5dcf19045b..6f504d67a0 100644
--- a/package/wilc1000-firmware/wilc1000-firmware.mk
+++ b/package/wilc1000-firmware/wilc1000-firmware.mk
@@ -11,7 +11,7 @@ WILC1000_FIRMWARE_SOURCE = v$(WILC1000_FIRMWARE_VERSION)_Firmware.zip
WILC1000_FIRMWARE_LICENSE = PROPRIETARY
define WILC1000_FIRMWARE_EXTRACT_CMDS
- $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(WILC1000_FIRMWARE_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR) $(WILC1000_FIRMWARE_DL_DIR)/$(WILC1000_FIRMWARE_SOURCE)
mv $(BUILD_DIR)/wireless-firmware-$(WILC1000_FIRMWARE_VERSION)_Firmware/* $(@D)
rmdir $(BUILD_DIR)/wireless-firmware-$(WILC1000_FIRMWARE_VERSION)_Firmware
endef
diff --git a/package/zynq-boot-bin/zynq-boot-bin.mk b/package/zynq-boot-bin/zynq-boot-bin.mk
index 3648c63654..c523e04f45 100644
--- a/package/zynq-boot-bin/zynq-boot-bin.mk
+++ b/package/zynq-boot-bin/zynq-boot-bin.mk
@@ -12,7 +12,7 @@ ZYNQ_BOOT_BIN_LICENSE = GPL-3.0+
HOST_ZYNQ_BOOT_BIN_DEPENDENCIES = host-python
define HOST_ZYNQ_BOOT_BIN_EXTRACT_CMDS
- cp $(DL_DIR)/$(ZYNQ_BOOT_BIN_SOURCE) $(@D)
+ cp $(ZYNQ_BOOT_BIN_DL_DIR)/$(ZYNQ_BOOT_BIN_SOURCE) $(@D)
endef
define HOST_ZYNQ_BOOT_BIN_INSTALL_CMDS
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 04/13] arc/xtensa: store the eXtensa overlay in the per-package DL_DIR
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
2018-04-02 8:14 ` [Buildroot] [v4 02/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
2018-04-02 8:14 ` [Buildroot] [v4 03/13] packages: use new $($PKG)_DL_DIR) variable Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 12:08 ` Thomas Petazzoni
2018-04-02 8:14 ` [Buildroot] [v4 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Maxime Hadjinlian
` (11 subsequent siblings)
14 siblings, 1 reply; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
From: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
arch/arch.mk.xtensa | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arch.mk.xtensa b/arch/arch.mk.xtensa
index 2b6cd26d03..fd410f6bfa 100644
--- a/arch/arch.mk.xtensa
+++ b/arch/arch.mk.xtensa
@@ -12,7 +12,7 @@
BR_ARCH_XTENSA_OVERLAY_FILE = $(call qstrip,$(BR2_XTENSA_OVERLAY_FILE))
ifneq ($(filter http://% https://% ftp://% scp://%,$(BR_ARCH_XTENSA_OVERLAY_FILE)),)
ARCH_XTENSA_OVERLAY_URL = $(BR_ARCH_XTENSA_OVERLAY_FILE)
-ARCH_XTENSA_OVERLAY_FILE = $(DL_DIR)/$(notdir $(BR_ARCH_XTENSA_OVERLAY_FILE))
+ARCH_XTENSA_OVERLAY_FILE = $($(PKG)_DL_DIR)/$(notdir $(BR_ARCH_XTENSA_OVERLAY_FILE))
# Do not check that file, we can't know its hash
BR_NO_CHECK_HASH_FOR += $(notdir $(ARCH_XTENSA_OVERLAY_URL))
else
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR)
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (2 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 04/13] arc/xtensa: store the eXtensa overlay in the per-package DL_DIR Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 11:06 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 06/13] support/download: make sure the download folder is created Maxime Hadjinlian
` (10 subsequent siblings)
14 siblings, 1 reply; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
Let the infrastructure use the new variable $(PKG)_DL_DIR
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
package/pkg-download.mk | 2 +-
package/pkg-generic.mk | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 2ee745fddd..151b2e5423 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -95,7 +95,7 @@ define DOWNLOAD
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
-n $($(PKG)_BASENAME_RAW) \
-N $($(PKG)_RAWNAME) \
- -o $(DL_DIR)/$(notdir $(1)) \
+ -o $($(PKG)_DL_DIR)/$(notdir $(1)) \
$(if $($(PKG)_GIT_SUBMODULES),-r) \
$(DOWNLOAD_URIS) \
$(QUIET) \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 25a34e6ca2..e7f4a0402d 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -613,7 +613,7 @@ $(2)_TARGET_DIRCLEAN = $$($(2)_DIR)/.stamp_dircleaned
# default extract command
$(2)_EXTRACT_CMDS ?= \
- $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$(DL_DIR)/$$($(2)_SOURCE) | \
+ $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$($(2)_DL_DIR)/$$($(2)_SOURCE) | \
$$(TAR) --strip-components=$$($(2)_STRIP_COMPONENTS) \
-C $$($(2)_DIR) \
$$(foreach x,$$($(2)_EXCLUDES),--exclude='$$(x)' ) \
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 06/13] support/download: make sure the download folder is created
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (3 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 11:08 ` Yann E. MORIN
2018-04-02 12:10 ` Thomas Petazzoni
2018-04-02 8:14 ` [Buildroot] [v4 07/13] pkg-generic: add a subdirectory to the DL_DIR Maxime Hadjinlian
` (9 subsequent siblings)
14 siblings, 2 replies; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
At the moment, it means that we make sure that BR2_DL_DIR is created, in
the future, it will make sure that BR2_DL_DIR/PKG_NAME/ is created.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-download.mk | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 151b2e5423..e88d3f9fda 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -88,7 +88,8 @@ endif
endif
define DOWNLOAD
- $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
+ $(Q)mkdir -p $($(PKG)_DL_DIR)/
+ $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1))) \
$(EXTRA_ENV) $(DL_WRAPPER) \
-c $($(PKG)_DL_VERSION) \
-f $(notdir $(1)) \
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 07/13] pkg-generic: add a subdirectory to the DL_DIR
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (4 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 06/13] support/download: make sure the download folder is created Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 8:14 ` [Buildroot] [v4 08/13] pkg-download: support new subdir for mirrors Maxime Hadjinlian
` (8 subsequent siblings)
14 siblings, 0 replies; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
With all the previous changes, we are now ready to add a subdirectory to
the DL_DIR.
The structure will now be DL_DIR/PKG_NAME/{FILE1,FILE2}
This is needed for multiple reasons:
- Avoid patches with name like SHA1.patch laying flat in DL_DIR,
which makes it hard to know to which packages they apply
- Avoid possible collisions if two releases have the same name
(e.g: v01.tar)
- Allow the possibility to handle a git cache per package in the
newly created subdirectory.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
package/pkg-generic.mk | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index e7f4a0402d..c84b1eb4a2 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -432,7 +432,7 @@ endif
$(2)_BASENAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
$(2)_BASENAME_RAW = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR = $$(DL_DIR)
+$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
$(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASENAME)
ifndef $(2)_SUBDIR
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 08/13] pkg-download: support new subdir for mirrors
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (5 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 07/13] pkg-generic: add a subdirectory to the DL_DIR Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 12:13 ` Thomas Petazzoni
2018-04-02 8:14 ` [Buildroot] [v4 09/13] pkg-generic: introduce _SAME_SOURCE_AS Maxime Hadjinlian
` (7 subsequent siblings)
14 siblings, 1 reply; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
Since we introduced subdirectory to the DL_DIR, we need to support them
in the PRIMARY and BACKUP mirror as they evolve to the new tree
structure.
We check first the new URI (with the subdir), and in case of failure, we
go check without.
By checking both URIs, we ensure that old mirror are usable.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
package/pkg-download.mk | 2 ++
1 file changed, 2 insertions(+)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index e88d3f9fda..54edbae29a 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -75,6 +75,7 @@ export BR_NO_CHECK_HASH_FOR =
ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE)/$(notdir $($(PKG)_DL_DIR)),urlencode) \
-u $(call getschemeplusuri,$(BR2_PRIMARY_SITE),urlencode)
endif
@@ -83,6 +84,7 @@ DOWNLOAD_URIS += \
-u $($(PKG)_SITE_METHOD)+$(dir $(1))
ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_BACKUP_SITE)/$(notdir $($(PKG)_DL_DIR)),urlencode) \
-u $(call getschemeplusuri,$(BR2_BACKUP_SITE),urlencode)
endif
endif
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 09/13] pkg-generic: introduce _SAME_SOURCE_AS
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (6 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 08/13] pkg-download: support new subdir for mirrors Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 11:57 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 10/13] package: share downloaded files for big packages Maxime Hadjinlian
` (6 subsequent siblings)
14 siblings, 1 reply; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
This per package variable can be used to specify that a package shares
the same sources as another package.
The use case here is for example linux-headers and linux, which share
the same sources (because they are the same upstream project), so we
don't want to download twice the kernel, nor store it multiple times
either.
Make will automatically try to help by introducing leading and trailing
spaces when replacing a line-continuation '\', so we need to call
$(strip) on the variable.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-generic.mk | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index c84b1eb4a2..a034c7ea02 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -432,7 +432,10 @@ endif
$(2)_BASENAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
$(2)_BASENAME_RAW = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
+$(2)_DL_DIR = $$(strip $$(if $$($(2)_SAME_SOURCE_AS), \
+ $$($$(call UPPERCASE,$$($(2)_SAME_SOURCE_AS))_DL_DIR), \
+ $$(DL_DIR)/$$($(2)_RAWNAME)) \
+ )
$(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASENAME)
ifndef $(2)_SUBDIR
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 10/13] package: share downloaded files for big packages
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (7 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 09/13] pkg-generic: introduce _SAME_SOURCE_AS Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 8:14 ` [Buildroot] [v4 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Maxime Hadjinlian
` (5 subsequent siblings)
14 siblings, 0 replies; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
From: "Yann E. MORIN" <yann.morin.1998@free.fr>
Theses packages are given as an example of the use of the
*_SAME_SOURCE_AS feature.
There maybe other packages that would benefit from that feature, they
would need to be added on a case by case basis.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
package/gcc/gcc-initial/gcc-initial.mk | 5 +++++
| 3 +++
| 1 +
3 files changed, 9 insertions(+)
diff --git a/package/gcc/gcc-initial/gcc-initial.mk b/package/gcc/gcc-initial/gcc-initial.mk
index c476b2faeb..07da50afa8 100644
--- a/package/gcc/gcc-initial/gcc-initial.mk
+++ b/package/gcc/gcc-initial/gcc-initial.mk
@@ -8,6 +8,11 @@ GCC_INITIAL_VERSION = $(GCC_VERSION)
GCC_INITIAL_SITE = $(GCC_SITE)
GCC_INITIAL_SOURCE = $(GCC_SOURCE)
+# We do not have a 'gcc' package per-se; we only have two incarnations,
+# gcc-initial and gcc-final. gcc-initial is just am internal step that
+# users should not care about, while gcc-final is the one they shall see.
+HOST_GCC_INITIAL_SAME_SOURCE_AS = host-gcc-final
+
HOST_GCC_INITIAL_DEPENDENCIES = $(HOST_GCC_COMMON_DEPENDENCIES)
HOST_GCC_INITIAL_EXCLUDES = $(HOST_GCC_EXCLUDES)
--git a/package/linux-headers/linux-headers.mk b/package/linux-headers/linux-headers.mk
index f1e3790608..6f4071c61a 100644
--- a/package/linux-headers/linux-headers.mk
+++ b/package/linux-headers/linux-headers.mk
@@ -82,6 +82,9 @@ endif
endif # ! BR2_KERNEL_HEADERS_AS_KERNEL
+# linux-headers really is the same as the linux package
+LINUX_HEADERS_SAME_SOURCE_AS = linux
+
LINUX_HEADERS_LICENSE = GPL-2.0
LINUX_HEADERS_LICENSE_FILES = COPYING
--git a/package/mesa3d-headers/mesa3d-headers.mk b/package/mesa3d-headers/mesa3d-headers.mk
index b48d965d8a..39a25270db 100644
--- a/package/mesa3d-headers/mesa3d-headers.mk
+++ b/package/mesa3d-headers/mesa3d-headers.mk
@@ -15,6 +15,7 @@ endif
MESA3D_HEADERS_VERSION = 18.0.0
MESA3D_HEADERS_SOURCE = mesa-$(MESA3D_HEADERS_VERSION).tar.xz
MESA3D_HEADERS_SITE = https://mesa.freedesktop.org/archive
+MESA3D_HEADERS_SAME_SOURCE_AS = mesa3d
MESA3D_HEADERS_LICENSE = MIT, SGI, Khronos
MESA3D_HEADERS_LICENSE_FILES = docs/license.html
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR)
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (8 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 10/13] package: share downloaded files for big packages Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 11:59 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 12/13] download: add flock call before dl-wrapper Maxime Hadjinlian
` (4 subsequent siblings)
14 siblings, 1 reply; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Config.in | 3 +++
docs/manual/adding-packages-generic.txt | 6 +++---
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/Config.in b/Config.in
index 71b9f60e5b..c2a1925088 100644
--- a/Config.in
+++ b/Config.in
@@ -200,6 +200,9 @@ config BR2_DL_DIR
If the Linux shell environment has defined the BR2_DL_DIR
environment variable, then this overrides this configuration
item.
+ The directory is organized with a subdirectory for each package.
+ Each package has its own $(LIBFOO_DL_DIR) variable that can be used
+ to find the correct path.
The default is $(TOPDIR)/dl
diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
index 9d1428ad40..62906d92bb 100644
--- a/docs/manual/adding-packages-generic.txt
+++ b/docs/manual/adding-packages-generic.txt
@@ -263,7 +263,7 @@ information is (assuming the package name is +libfoo+) :
the file using this URL. Otherwise, Buildroot will assume the file
to be downloaded is located at +LIBFOO_SITE+. Buildroot will not do
anything with those additional files, except download them: it will
- be up to the package recipe to use them from +$(DL_DIR)+.
+ be up to the package recipe to use them from +$(LIBFOO_DL_DIR)+.
* +LIBFOO_SITE_METHOD+ determines the method used to fetch or copy the
package source code. In many cases, Buildroot guesses the method
@@ -554,8 +554,8 @@ In the action definitions, you can use the following variables:
* +$(@D)+, which contains the directory in which the package source
code has been uncompressed.
-* +$(DL_DIR)+ contains the path to the directory where all the downloads made
- by Buildroot are stored.
+* +$(LIBFOO_DL_DIR)+ contains the path to the directory where all the downloads
+ made by Buildroot for +libfoo+ are stored in.
* +$(TARGET_CC)+, +$(TARGET_LD)+, etc. to get the target
cross-compilation utilities
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 12/13] download: add flock call before dl-wrapper
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (9 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 12:00 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 13/13] download: git: introduce cache feature Maxime Hadjinlian
` (3 subsequent siblings)
14 siblings, 1 reply; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
In order to introduce the cache mechanisms, we need to have a lock on
the $(LIBFOO_DL_DIR), otherwise it would be impossible to do parallel
download (a shared DL_DIR for two buildroot instances).
To make sure the directory exists, the mkdir call has been removed from
the dl-wrapper and put in the infrastructure.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
package/pkg-download.mk | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 54edbae29a..05842bd430 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -19,6 +19,7 @@ SSH := $(call qstrip,$(BR2_SSH))
export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
DL_WRAPPER = support/download/dl-wrapper
+FLOCK = flock $($(PKG)_DL_DIR)/
# DL_DIR may have been set already from the environment
ifeq ($(origin DL_DIR),undefined)
@@ -92,7 +93,7 @@ endif
define DOWNLOAD
$(Q)mkdir -p $($(PKG)_DL_DIR)/
$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1))) \
- $(EXTRA_ENV) $(DL_WRAPPER) \
+ $(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
-c $($(PKG)_DL_VERSION) \
-f $(notdir $(1)) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 13/13] download: git: introduce cache feature
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (10 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 12/13] download: add flock call before dl-wrapper Maxime Hadjinlian
@ 2018-04-02 8:14 ` Maxime Hadjinlian
2018-04-02 12:09 ` Yann E. MORIN
2018-04-02 8:25 ` [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Yann E. MORIN
` (2 subsequent siblings)
14 siblings, 1 reply; 30+ messages in thread
From: Maxime Hadjinlian @ 2018-04-02 8:14 UTC (permalink / raw)
To: buildroot
Now we keep the git clone that we download and generates our tarball
from there.
The main goal here is that if you change the version of a package (say
Linux), instead of cloning all over again, you will simply 'git fetch'
from the repo the missing objects, then generates the tarball again.
This should speed the 'source' part of the build significantly.
The drawback is that the DL_DIR will grow much larger; but time is more
important than disk space nowadays.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
support/download/git | 63 ++++++++++++++++++++++++++++++++--------------------
1 file changed, 39 insertions(+), 24 deletions(-)
diff --git a/support/download/git b/support/download/git
index 58a2c6ad9d..c6b0f81f13 100755
--- a/support/download/git
+++ b/support/download/git
@@ -39,29 +39,40 @@ _git() {
eval ${GIT} "${@}"
}
-# Try a shallow clone, since it is faster than a full clone - but that only
-# works if the version is a ref (tag or branch). Before trying to do a shallow
-# clone we check if ${cset} is in the list provided by git ls-remote. If not
-# we fall back on a full clone.
+# We want to check if a cache of the git clone of this repo already exists.
+git_cache="${BR2_DL_DIR}/${basename%%-*}/git"
+
+# If the cache directory already exists, don't try to clone.
+if [ ! -d "${git_cache}" ]; then
+ _git init "'${git_cache}'"
+ _git -C "'${git_cache}'" remote add origin "'${uri}'"
+fi
+
+pushd "${git_cache}" >/dev/null
+
+_git remote set-url origin "'${uri}'"
+
+# Try to fetch with limited depth, since it is faster than a full clone - but
+# that only works if the version is a ref (tag or branch). Before trying to do
+# a shallow clone we check if ${cset} is in the list provided by git ls-remote.
+# If not we fall back on a full fetch.
#
-# Messages for the type of clone used are provided to ease debugging in case of
-# problems
+# Messages for the type of clone used are provided to ease debugging in
+# case of problems
git_done=0
-if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
- printf "Doing shallow clone\n"
- if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
+if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then
+ printf "Doing fetch with limited depth\n"
+ if _git fetch "${@}" --depth 1 origin "'${cset}'"; then
git_done=1
else
- printf "Shallow clone failed, falling back to doing a full clone\n"
+ printf "Fetching ref failed, falling back to fetching all refs\n"
fi
fi
if [ ${git_done} -eq 0 ]; then
- printf "Doing full clone\n"
- _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
+ printf "Fetching all references\n"
+ _git fetch origin -t
fi
-pushd "${basename}" >/dev/null
-
# Try to get the special refs exposed by some forges (pull-requests for
# github, changes for gerrit...). There is no easy way to know whether
# the cset the user passed us is such a special ref or a tag or a sha1
@@ -86,20 +97,24 @@ if [ ${recurse} -eq 1 ]; then
_git submodule update --init --recursive
fi
-# We do not want the .git dir; we keep other .git files, in case they
-# are the only files in their directory.
+# Generate the archive, sort with the C locale so that it is reproducible
+# We do not want the .git dir; we keep other .git
+# files, in case they are the only files in their directory.
# The .git dir would generate non reproducible tarballs as it depends on
# the state of the remote server. It also would generate large tarballs
# (gigabytes for some linux trees) when a full clone took place.
-rm -rf .git
+find . -not -type d \
+ -and -not -path "./.git/*" >"${output}.list"
+LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
-popd >/dev/null
-
-# Generate the archive, sort with the C locale so that it is reproducible
-find "${basename}" -not -type d >"${basename}.list"
-LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
# Create GNU-format tarballs, since that's the format of the tarballs on
# sources.buildroot.org and used in the *.hash files
-tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
- -T "${basename}.list.sorted" >"${output}.tar"
+tar cf - --transform="s/^\.$/${basename}/" \
+ --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
+ -T "${output}.list.sorted" >"${output}.tar"
gzip -6 -n <"${output}.tar" >"${output}"
+
+rm -f "${output}.list"
+rm -f "${output}.list.sorted"
+
+popd >/dev/null
--
2.16.2
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (11 preceding siblings ...)
2018-04-02 8:14 ` [Buildroot] [v4 13/13] download: git: introduce cache feature Maxime Hadjinlian
@ 2018-04-02 8:25 ` Yann E. MORIN
2018-04-02 8:40 ` Thomas Petazzoni
2018-04-02 10:26 ` Peter Korsgaard
14 siblings, 0 replies; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 8:25 UTC (permalink / raw)
To: buildroot
Maxime, All,
On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Currently all download helpers accepts the local output file, the remote
> locations, the changesets and so on... as positional arguments.
>
> This was well and nice when that's was all we needed.
>
> But then we added an option to quiesce their verbosity, and that was
> shoehorned with a trivial getopts, still keeping all the existing
> positional arguments as... positional arguments.
>
> Adding yet more options while keeping positional arguments will not be
> very easy, even if we do not envision any new option in the foreseeable
> future (but 640K ought to be enough for everyone, remember? ;-) ).
>
> Change all helpers to accept a set of generic options (-q for quiet and
> -o for the output file) as well as helper-specific options (like -r for
> the repository, -c for a changeset...).
>
> Maxime:
> Changed -R to -r for recurse (only for the git backend)
> Changed -r to -u for URI (for all backend)
> Change -R to -c for cset (for CVS and SVN backend)
> Add the export of the BR_BACKEND_DL_GETOPTS so all the backend wrapper
> can use the same option easily
> Now all the backends use the same common options.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Even though this originated from me, Maxime has done substantial
changes, so:
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Regards,
Yann E. MORIN.
> ---
> v1 -> v2:
> - Fix comment in bzr backend
> - Remove 'R' from BR_BACKEND_DL_GETOPTS
> - check-hash shouldn't have had its options changed since it's not a
> download backend (Arnout)
> ---
> package/pkg-download.mk | 38 +++++++++++++++++++-------------------
> support/download/bzr | 25 ++++++++++++++-----------
> support/download/cp | 17 +++++++++--------
> support/download/cvs | 34 +++++++++++++++++++---------------
> support/download/dl-wrapper | 7 ++++++-
> support/download/git | 33 +++++++++++++++++----------------
> support/download/hg | 25 ++++++++++++++-----------
> support/download/scp | 19 ++++++++++---------
> support/download/svn | 25 ++++++++++++++-----------
> support/download/wget | 17 +++++++++--------
> 10 files changed, 131 insertions(+), 109 deletions(-)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 6f15deba28..a410dce1ee 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -77,9 +77,9 @@ define DOWNLOAD_GIT
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> $(QUIET) \
> -- \
> - $($(PKG)_SITE) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_BASENAME_RAW) \
> + -u $($(PKG)_SITE) \
> + -c $($(PKG)_DL_VERSION) \
> + -n $($(PKG)_BASENAME_RAW) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -88,9 +88,9 @@ define DOWNLOAD_BZR
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> $(QUIET) \
> -- \
> - $($(PKG)_SITE) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_BASENAME_RAW) \
> + -u $($(PKG)_SITE) \
> + -c $($(PKG)_DL_VERSION) \
> + -n $($(PKG)_BASENAME_RAW) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -99,10 +99,10 @@ define DOWNLOAD_CVS
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> $(QUIET) \
> -- \
> - $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_RAWNAME) \
> - $($(PKG)_BASENAME_RAW) \
> + -u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
> + -c $($(PKG)_DL_VERSION) \
> + -N $($(PKG)_RAWNAME) \
> + -n $($(PKG)_BASENAME_RAW) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -111,9 +111,9 @@ define DOWNLOAD_SVN
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> $(QUIET) \
> -- \
> - $($(PKG)_SITE) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_BASENAME_RAW) \
> + -u $($(PKG)_SITE) \
> + -c $($(PKG)_DL_VERSION) \
> + -n $($(PKG)_BASENAME_RAW) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -126,7 +126,7 @@ define DOWNLOAD_SCP
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> $(QUIET) \
> -- \
> - '$(call stripurischeme,$(call qstrip,$(1)))' \
> + -u '$(call stripurischeme,$(call qstrip,$(1)))' \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -135,9 +135,9 @@ define DOWNLOAD_HG
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> $(QUIET) \
> -- \
> - $($(PKG)_SITE) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_BASENAME_RAW) \
> + -u $($(PKG)_SITE) \
> + -c $($(PKG)_DL_VERSION) \
> + -n $($(PKG)_BASENAME_RAW) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -147,7 +147,7 @@ define DOWNLOAD_WGET
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> $(QUIET) \
> -- \
> - '$(call qstrip,$(1))' \
> + -u '$(call qstrip,$(1))' \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -157,7 +157,7 @@ define DOWNLOAD_LOCALFILES
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> $(QUIET) \
> -- \
> - $(call stripurischeme,$(call qstrip,$(1))) \
> + -u $(call stripurischeme,$(call qstrip,$(1))) \
> $($(PKG)_DL_OPTS)
> endef
>
> diff --git a/support/download/bzr b/support/download/bzr
> index 75b7b415c1..5289a421cd 100755
> --- a/support/download/bzr
> +++ b/support/download/bzr
> @@ -5,28 +5,31 @@ set -e
>
> # Download helper for bzr, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
> +# Options:
> +# -q Be quiet
> +# -o FILE Generate archive in FILE.
> +# -u URI Clone from repository at URI.
> +# -c CSET Use changeset (or revision) CSET.
> +# -n NAME Use basename NAME.
> #
> # Environment:
> # BZR : the bzr command to call
>
>
> verbose=
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-q;;
> + o) output="${OPTARG}";;
> + u) uri="${OPTARG}";;
> + c) cset="${OPTARG}";;
> + n) basename="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -repo="${2}"
> -rev="${3}"
> -basename="${4}"
> -
> -shift 4 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> @@ -51,5 +54,5 @@ if [ ${bzr_version} -ge ${bzr_min_version} ]; then
> fi
>
> _bzr export ${verbose} --root="'${basename}/'" --format=tgz \
> - ${timestamp_opt} - "${@}" "'${repo}'" -r "'${rev}'" \
> + ${timestamp_opt} - "${@}" "'${uri}'" -r "'${cset}'" \
> >"${output}"
> diff --git a/support/download/cp b/support/download/cp
> index 0ee1f3ba82..52fe2de83d 100755
> --- a/support/download/cp
> +++ b/support/download/cp
> @@ -5,8 +5,10 @@ set -e
>
> # Download helper for cp, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../cp [-q] OUT_FILE SRC_FILE
> +# Options:
> +# -q Be quiet.
> +# -o FILE Copy to file FILE.
> +# -u FILE Copy from file FILE.
> #
> # Environment:
> # LOCALFILES: the cp command to call
> @@ -17,18 +19,17 @@ set -e
> # Make 'cp' verbose by default, so it behaves a bit like the others.
> verbose=-v
>
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=;;
> + o) output="${OPTARG}";;
> + u) source="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -source="${2}"
> -
> -shift 2 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> diff --git a/support/download/cvs b/support/download/cvs
> index 50050ab1c9..69d5c71f28 100755
> --- a/support/download/cvs
> +++ b/support/download/cvs
> @@ -5,28 +5,32 @@ set -e
>
> # Download helper for cvs, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
> +# Options:
> +# -q Be quiet
> +# -o FILE Generate archive in FILE.
> +# -u URI Checkout from repository at URI.
> +# -c REV Use revision REV.
> +# -N RAWNAME Use rawname (aka module name) RAWNAME.
> +# -n NAME Use basename NAME.
> #
> # Environment:
> # CVS : the cvs command to call
>
> verbose=
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-Q;;
> + o) output="${OPTARG}";;
> + u) uri="${OPTARG}";;
> + c) rev="${OPTARG}";;
> + N) rawname="${OPTARG}";;
> + n) basename="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -repo="${2}"
> -rev="${3}"
> -rawname="${4}"
> -basename="${5}"
> -
> -shift 5 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> @@ -42,14 +46,14 @@ else
> select="-r"
> fi
>
> -# The absence of an initial : on ${repo} means access method undefined
> -if [[ ! "${repo}" =~ ^: ]]; then
> +# The absence of an initial : on ${uri} means access method undefined
> +if [[ ! "${uri}" =~ ^: ]]; then
> # defaults to anonymous pserver
> - repo=":pserver:anonymous@${repo}"
> + uri=":pserver:anonymous@${uri}"
> fi
>
> export TZ=UTC
> -_cvs ${verbose} -z3 -d"'${repo}'" \
> +_cvs ${verbose} -z3 -d"'${uri}'" \
> co "${@}" -d "'${basename}'" ${select} "'${rev}'" -P "'${rawname}'"
>
> tar czf "${output}" "${basename}"
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index b018819833..abc51f637a 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -19,6 +19,8 @@
> # We want to catch any unexpected failure, and exit immediately.
> set -e
>
> +export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:q"
> +
> main() {
> local OPT OPTARG
> local backend output hfile recurse quiet rc
> @@ -83,7 +85,10 @@ main() {
> # If the backend fails, we can just remove the temporary directory to
> # remove all the cruft it may have left behind. Then we just exit in
> # error too.
> - if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
> + if ! "${OLDPWD}/support/download/${backend}" \
> + ${quiet} ${recurse} \
> + -o "${tmpf}" "${@}"
> + then
> rm -rf "${tmpd}"
> exit 1
> fi
> diff --git a/support/download/git b/support/download/git
> index f590ff6494..58a2c6ad9d 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -5,32 +5,33 @@ set -e
>
> # Download helper for git, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../git [-q] [-r] OUT_FILE REPO_URL CSET BASENAME
> -#
> -# -q Be quiet.
> -# -r Clone and archive sub-modules.
> +# Options:
> +# -q Be quiet.
> +# -r Clone and archive sub-modules.
> +# -o FILE Generate archive in FILE.
> +# -u URI Clone from repository at URI.
> +# -c CSET Use changeset CSET.
> +# -n NAME Use basename NAME.
> #
> # Environment:
> # GIT : the git command to call
>
> verbose=
> recurse=0
> -while getopts :qr OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-q; exec >/dev/null;;
> r) recurse=1;;
> + o) output="${OPTARG}";;
> + u) uri="${OPTARG}";;
> + c) cset="${OPTARG}";;
> + n) basename="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -repo="${2}"
> -cset="${3}"
> -basename="${4}"
> -
> -shift 4 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> @@ -46,9 +47,9 @@ _git() {
> # Messages for the type of clone used are provided to ease debugging in case of
> # problems
> git_done=0
> -if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
> +if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
> printf "Doing shallow clone\n"
> - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
> + if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
> git_done=1
> else
> printf "Shallow clone failed, falling back to doing a full clone\n"
> @@ -56,7 +57,7 @@ if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
> fi
> if [ ${git_done} -eq 0 ]; then
> printf "Doing full clone\n"
> - _git clone ${verbose} "${@}" "'${repo}'" "'${basename}'"
> + _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
> fi
>
> pushd "${basename}" >/dev/null
> diff --git a/support/download/hg b/support/download/hg
> index 3af01690b3..efb515fca5 100755
> --- a/support/download/hg
> +++ b/support/download/hg
> @@ -5,27 +5,30 @@ set -e
>
> # Download helper for hg, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
> +# Options:
> +# -q Be quiet.
> +# -o FILE Generate archive in FILE.
> +# -u URI Clone from repository at URI.
> +# -c CSET Use changeset (or revision) CSET.
> +# -n NAME Use basename NAME.
> #
> # Environment:
> # HG : the hg command to call
>
> verbose=
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-q;;
> + o) output="${OPTARG}";;
> + u) uri="${OPTARG}";;
> + c) cset="${OPTARG}";;
> + n) basename="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -repo="${2}"
> -cset="${3}"
> -basename="${4}"
> -
> -shift 4 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> @@ -33,7 +36,7 @@ _hg() {
> eval ${HG} "${@}"
> }
>
> -_hg clone ${verbose} "${@}" --noupdate "'${repo}'" "'${basename}'"
> +_hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'"
>
> _hg archive ${verbose} --repository "'${basename}'" --type tgz \
> --prefix "'${basename}'" --rev "'${cset}'" \
> diff --git a/support/download/scp b/support/download/scp
> index 825fd41c64..8ecf2f4b22 100755
> --- a/support/download/scp
> +++ b/support/download/scp
> @@ -5,25 +5,26 @@ set -e
>
> # Download helper for scp, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../scp [-q] OUT_FILE SRC_URL
> +# Options:
> +# -q Be quiet.
> +# -o FILE Copy to local file FILE.
> +# -u FILE Copy from remote file FILE.
> #
> # Environment:
> # SCP : the scp command to call
>
> verbose=
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-q;;
> + o) output="${OPTARG}";;
> + u) uri="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -url="${2}"
> -
> -shift 2 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> @@ -31,4 +32,4 @@ _scp() {
> eval ${SCP} "${@}"
> }
>
> -_scp ${verbose} "${@}" "'${url}'" "'${output}'"
> +_scp ${verbose} "${@}" "'${uri}'" "'${output}'"
> diff --git a/support/download/svn b/support/download/svn
> index 77abf3d02d..542b25c0a2 100755
> --- a/support/download/svn
> +++ b/support/download/svn
> @@ -5,27 +5,30 @@ set -e
>
> # Download helper for svn, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../svn [-q] OUT_FILE REPO_URL REV BASNAME
> +# Options:
> +# -q Be quiet.
> +# -o FILE Generate archive in FILE.
> +# -u URI Checkout from repository at URI.
> +# -c REV Use revision REV.
> +# -n NAME Use basename NAME.
> #
> # Environment:
> # SVN : the svn command to call
>
> verbose=
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-q;;
> + o) output="${OPTARG}";;
> + u) uri="${OPTARG}";;
> + c) rev="${OPTARG}";;
> + n) basename="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -repo="${2}"
> -rev="${3}"
> -basename="${4}"
> -
> -shift 4 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> @@ -33,6 +36,6 @@ _svn() {
> eval ${SVN} "${@}"
> }
>
> -_svn export ${verbose} "${@}" "'${repo}@${rev}'" "'${basename}'"
> +_svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
>
> tar czf "${output}" "${basename}"
> diff --git a/support/download/wget b/support/download/wget
> index 768de904c3..fece6663ca 100755
> --- a/support/download/wget
> +++ b/support/download/wget
> @@ -5,25 +5,26 @@ set -e
>
> # Download helper for wget, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../wget [-q] OUT_FILE URL
> +# Options:
> +# -q Be quiet.
> +# -o FILE Save into file FILE.
> +# -u URL Download file at URL.
> #
> # Environment:
> # WGET : the wget command to call
>
> verbose=
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-q;;
> + o) output="${OPTARG}";;
> + u) url="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -url="${2}"
> -
> -shift 2 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> --
> 2.16.2
>
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (12 preceding siblings ...)
2018-04-02 8:25 ` [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Yann E. MORIN
@ 2018-04-02 8:40 ` Thomas Petazzoni
2018-04-02 10:26 ` Peter Korsgaard
14 siblings, 0 replies; 30+ messages in thread
From: Thomas Petazzoni @ 2018-04-02 8:40 UTC (permalink / raw)
To: buildroot
Hello,
On Mon, 2 Apr 2018 10:14:22 +0200, Maxime Hadjinlian wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Currently all download helpers accepts the local output file, the remote
> locations, the changesets and so on... as positional arguments.
>
> This was well and nice when that's was all we needed.
>
> But then we added an option to quiesce their verbosity, and that was
> shoehorned with a trivial getopts, still keeping all the existing
> positional arguments as... positional arguments.
>
> Adding yet more options while keeping positional arguments will not be
> very easy, even if we do not envision any new option in the foreseeable
> future (but 640K ought to be enough for everyone, remember? ;-) ).
>
> Change all helpers to accept a set of generic options (-q for quiet and
> -o for the output file) as well as helper-specific options (like -r for
> the repository, -c for a changeset...).
>
> Maxime:
> Changed -R to -r for recurse (only for the git backend)
> Changed -r to -u for URI (for all backend)
> Change -R to -c for cset (for CVS and SVN backend)
> Add the export of the BR_BACKEND_DL_GETOPTS so all the backend wrapper
> can use the same option easily
> Now all the backends use the same common options.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Thanks!
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 02/13] download: put most of the infra in dl-wrapper
2018-04-02 8:14 ` [Buildroot] [v4 02/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
@ 2018-04-02 10:23 ` Yann E. MORIN
2018-04-02 12:18 ` Peter Korsgaard
1 sibling, 0 replies; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 10:23 UTC (permalink / raw)
To: buildroot
On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> The goal here is to simplify the infrastructure by putting most of the
> code in the dl-wrapper as it's easier to implement and to read.
>
> Most of the functions were common already, this patch finalizes it by
> making the pkg-download.mk pass all the parameters needed to the
> dl-wrapper which in turns will pass everything to every backend.
>
> The backend will then cherry-pick what it needs from these arguments
> and act accordingly.
>
> It eases the transition to the addition of a sub directory per package
> in the DL_DIR, and later on, a git cache.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
> Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
But see a few comments below...
> ---
> v1 -> v2:
> - Rename cp backend to file (Arnout)
> - Don't use BR_BACKEND_DL_GETOPTS for dl-wrapper (Arnout)
> - Add "urlencode" to scheme passed to the dl-wrapper to support the
> fact that we need to urlencode the filename when using PRIMARY and
> BACKUP mirror (some files are named toto.c?v=1.0) (Arnout)
> - Fix uristripscheme replaced by bash'ism (Arnout)
> - Add the check hash into the loop, exit with error only if all the
> download+check failed. (Arnout)
> ---
> package/pkg-download.mk | 170 ++++++++------------------------------------
> support/download/cvs | 2 +-
> support/download/dl-wrapper | 139 +++++++++++++++++++++++-------------
> support/download/wget | 10 ++-
> 4 files changed, 129 insertions(+), 192 deletions(-)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index a410dce1ee..2ee745fddd 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -42,6 +42,8 @@ DL_DIR := $(shell mkdir -p $(DL_DIR) && cd $(DL_DIR) >/dev/null && pwd)
> #
> # geturischeme: http
> geturischeme = $(firstword $(subst ://, ,$(call qstrip,$(1))))
> +# getschemeplusuri: git|parameter+http://example.com
> +getschemeplusuri = $(call geturischeme,$(1))$(if $(2),\|$(2))+$(1)
The encoding is a bit weird... :-/
But it is need because we need to pass a single option:
- the site method
- options to the backend (e.g. urlencode)
- the uri without the basename
However, this is needed only for primary/backup mirrors:
[--SNIP--]
> +ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
> +DOWNLOAD_URIS += \
> + -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE),urlencode)
> +endif
> +
> +ifeq ($(BR2_PRIMARY_SITE_ONLY),)
> +DOWNLOAD_URIS += \
> + -u $($(PKG)_SITE_METHOD)+$(dir $(1))
> +ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
> +DOWNLOAD_URIS += \
> + -u $(call getschemeplusuri,$(BR2_BACKUP_SITE),urlencode)
> +endif
> +endif
However, we only pass the base URIs (i.e. without the trailing basename)
to the primary and backend. So we can't urlencode the basename at that
point... Only the backend will be able to...
But as discussed IRL, this is because:
- _SITE and _SOURCE are separate at the package level
- the generic-package macro will aggregate the _SITE and _SOURCE
together to form a complete URI
- then , the DOWNLOAD macro will split it again
- some backends may want to re-aggregate them, while other backends
do not.
As a conclusion: reworking all this mess and complexity is a different
topic, which will take a while to sort out...
[--SNIP--]
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index abc51f637a..58cc04b4c4 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
[--SNIP--]
> @@ -66,48 +69,85 @@ main() {
> warn "Re-downloading '%s'...\n" "${output##*/}"
> fi
>
> - # tmpd is a temporary directory in which backends may store intermediate
> - # by-products of the download.
> - # tmpf is the file in which the backends should put the downloaded content.
> - # tmpd is located in $(BUILD_DIR), so as not to clutter the (precious)
> - # $(BR2_DL_DIR)
> - # We let the backends create tmpf, so they are able to set whatever
> - # permission bits they want (although we're only really interested in
> - # the executable bit.)
> - tmpd="$(mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX")"
> - tmpf="${tmpd}/output"
> + # Look through all the uris that we were given to downoad the package
*download (missing ell 'l')
[--SNIP--]
> - if [ ${?} -ne 3 ]; then
> + # If the backend fails, we can just remove the content of the temporary
> + # directory to remove all the cruft it may have left behind, and tries
> + # the next URI until it succeeds. Once out of URI to tries, we need to
> + # cleanup and exit.
> + if ! "${OLDPWD}/support/download/${backend}" \
> + $([ -n "${urlencode}" ] && printf %s '-e') \
> + -c "${cset}" \
> + -n "${raw_base_name}" \
> + -N "${raw_name}" \
> + -f "${filename}" \
> + -u "${uri}" \
> + -o "${tmpf}" \
> + ${quiet} ${recurse} "${@}"
> + then
> rm -rf "${tmpd}"
> - exit 1
> + # cd back to keep path coherence
> + cd "${OLDPWD}"
cd before rm-rf. ;-)
Regards,
Yann E. MORIN.
> + continue
> fi
>
> - # the hash file exists and there was no hash to check the file against
> - rc=1
> + # cd back to free the temp-dir, so we can remove it later
> + cd "${OLDPWD}"
> +
> + # Check if the downloaded file is sane, and matches the stored hashes
> + # for that file
> + if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
> + rc=0
> + else
> + if [ ${?} -ne 3 ]; then
> + rm -rf "${tmpd}"
> + continue
> + fi
> +
> + # the hash file exists and there was no hash to check the file
> + # against
> + rc=1
> + fi
> + download_and_check=1
> + break
> + done
> +
> + # We tried every URI possible, none seems to work or to check against the
> + # available hash. *ABORT MISSION*
> + if [ "${download_and_check}" -eq 0 ]; then
> + rm -rf "${tmpd}"
> + exit 1
> fi
>
> # tmp_output is in the same directory as the final output, so we can
> @@ -173,16 +213,13 @@ DESCRIPTION
>
> -h This help text.
>
> - -b BACKEND
> - Wrap the specified BACKEND. Known backends are:
> - bzr Bazaar
> - cp Local files
> - cvs Concurrent Versions System
> - git Git
> - hg Mercurial
> - scp Secure copy
> - svn Subversion
> - wget HTTP download
> + -u URIs
> + The URI to get the file from, the URI must respect the format given in
> + the example.
> + You may give as many '-u URI' as you want, the script will stop at the
> + frist successful download.
> +
> + Example: backend+URI; git+http://example.com or http+http://example.com
>
> -o FILE
> Store the downloaded archive in FILE.
> diff --git a/support/download/wget b/support/download/wget
> index fece6663ca..c69e6071aa 100755
> --- a/support/download/wget
> +++ b/support/download/wget
> @@ -8,7 +8,9 @@ set -e
> # Options:
> # -q Be quiet.
> # -o FILE Save into file FILE.
> +# -f FILENAME The filename of the tarball to get at URL
> # -u URL Download file at URL.
> +# -e ENCODE Tell wget to urlencode the filename passed to it
> #
> # Environment:
> # WGET : the wget command to call
> @@ -18,7 +20,9 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-q;;
> o) output="${OPTARG}";;
> + f) filename="${OPTARG}";;
> u) url="${OPTARG}";;
> + e) encode="-e";;
> :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> @@ -32,4 +36,8 @@ _wget() {
> eval ${WGET} "${@}"
> }
>
> -_wget ${verbose} "${@}" -O "'${output}'" "'${url}'"
> +# Replace every '?' with '%3F' in the filename; only for the PRIMARY and BACKUP
> +# mirror
> +[ -n "${encode}" ] && filename=${filename//\?/%3F}
> +
> +_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
> --
> 2.16.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
` (13 preceding siblings ...)
2018-04-02 8:40 ` Thomas Petazzoni
@ 2018-04-02 10:26 ` Peter Korsgaard
14 siblings, 0 replies; 30+ messages in thread
From: Peter Korsgaard @ 2018-04-02 10:26 UTC (permalink / raw)
To: buildroot
>>>>> "Maxime" == Maxime Hadjinlian <maxime.hadjinlian@gmail.com> writes:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Currently all download helpers accepts the local output file, the remote
> locations, the changesets and so on... as positional arguments.
> This was well and nice when that's was all we needed.
> But then we added an option to quiesce their verbosity, and that was
> shoehorned with a trivial getopts, still keeping all the existing
> positional arguments as... positional arguments.
> Adding yet more options while keeping positional arguments will not be
> very easy, even if we do not envision any new option in the foreseeable
> future (but 640K ought to be enough for everyone, remember? ;-) ).
> Change all helpers to accept a set of generic options (-q for quiet and
> -o for the output file) as well as helper-specific options (like -r for
> the repository, -c for a changeset...).
> Maxime:
> Changed -R to -r for recurse (only for the git backend)
> Changed -r to -u for URI (for all backend)
> Change -R to -c for cset (for CVS and SVN backend)
> Add the export of the BR_BACKEND_DL_GETOPTS so all the backend wrapper
> can use the same option easily
> Now all the backends use the same common options.
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> v1 -> v2:
> - Fix comment in bzr backend
> - Remove 'R' from BR_BACKEND_DL_GETOPTS
> - check-hash shouldn't have had its options changed since it's not a
> download backend (Arnout)
Committed, thanks.
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 03/13] packages: use new $($PKG)_DL_DIR) variable
2018-04-02 8:14 ` [Buildroot] [v4 03/13] packages: use new $($PKG)_DL_DIR) variable Maxime Hadjinlian
@ 2018-04-02 10:36 ` Yann E. MORIN
0 siblings, 0 replies; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 10:36 UTC (permalink / raw)
To: buildroot
Maxime, All,
title: the variable is not 'new', it had always existed...
On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the
> transition into a new directory structure for DL_DIR.
>
> This commit has been generated with the following scripts:
>
> for i in $(find . -iname "*.mk"); do
> if ! grep -q "\$(DL_DIR)" ${i}; then
> continue
> fi
> pkg_name="$(basename $(dirname ${i}))"
> [ "${pkg_name}" = "package" ] && continue
> raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
> pkg_dl_dir="${raw_pkg_name}_DL_DIR"
> sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}
> done
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
[--SNIP--]
> diff --git a/package/bootstrap/bootstrap.mk b/package/bootstrap/bootstrap.mk
> index 61d6c826fc..50cdc322e4 100644
> --- a/package/bootstrap/bootstrap.mk
> +++ b/package/bootstrap/bootstrap.mk
> @@ -11,7 +11,9 @@ BOOTSTRAP_LICENSE = MIT
> BOOTSTRAP_LICENSE_FILES = css/bootstrap.css
>
> define BOOTSTRAP_EXTRACT_CMDS
> - $(UNZIP) $(DL_DIR)/$(BOOTSTRAP_SOURCE) -d $(@D)
> + $(UNZIP) $(BOOTSTRAP_DL_DIR)/$(BOOTSTRAP_SOURCE) -d $(@D)
> + mv $(@D)/bootstrap-$(BOOTSTRAP_VERSION)-dist/* $(@D)/
> + rmdir $(@D)/bootstrap-$(BOOTSTRAP_VERSION)-dist
You incorrectly re-add lines that were removed in 39c181d5a99
(bootstrap: bump version to 4.0.0)...
[--SNIP--]
> diff --git a/package/openobex/openobex.mk b/package/openobex/openobex.mk
> index ea41f65a80..c19780110c 100644
> --- a/package/openobex/openobex.mk
> +++ b/package/openobex/openobex.mk
> @@ -19,6 +19,12 @@ ifeq ($(BR2_PACKAGE_BLUEZ_UTILS),y)
> OPENOBEX_DEPENDENCIES += bluez_utils
> endif
>
> +define OPENOBEX_EXTRACT_CMDS
> + $(UNZIP) -d $(@D) $(OPENOBEX_DL_DIR)/$(OPENOBEX_SOURCE)
> + mv $(@D)/openobex-$(OPENOBEX_VERSION)-Source/* $(@D)
> + $(RM) -r $(@D)/openobex-$(OPENOBEX_VERSION)-Source
> +endef
openobex no longer uses a ziop file, since commit 49a9fb0f13
(package/openobex: bump version to 1.7.2)
[--SNIP--]
> diff --git a/package/urg/urg.mk b/package/urg/urg.mk
> index 966627fe36..1346079893 100644
> --- a/package/urg/urg.mk
> +++ b/package/urg/urg.mk
> @@ -25,7 +25,9 @@ endif
> URG_CONFIG_SCRIPTS = c_urg-config urg-config
>
> define URG_EXTRACT_CMDS
> - $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(URG_SOURCE)
> + $(UNZIP) -d $(BUILD_DIR) $(URG_DL_DIR)/$(URG_SOURCE)
> + test -d $(URG_DIR) || \
> + mv $(BUILD_DIR)/$(subst .zip,,$(URG_SOURCE)) $(URG_DIR)
You are re-adding lines that got removed in 9e943e85228 (package/urg:
fix extraction commands).
Regards,
Yann E. MORIN.
> endef
>
> $(eval $(autotools-package))
> diff --git a/package/waf/waf.mk b/package/waf/waf.mk
> index e7ac891b39..639c26a3b0 100644
> --- a/package/waf/waf.mk
> +++ b/package/waf/waf.mk
> @@ -9,7 +9,7 @@ WAF_SOURCE = waf-$(WAF_VERSION)
> WAF_SITE = https://waf.io/
>
> define HOST_WAF_EXTRACT_CMDS
> - $(INSTALL) -D -m 0755 $(DL_DIR)/waf-$(WAF_VERSION) $(@D)/waf
> + $(INSTALL) -D -m 0755 $(WAF_DL_DIR)/waf-$(WAF_VERSION) $(@D)/waf
> endef
>
> define HOST_WAF_INSTALL_CMDS
> diff --git a/package/whetstone/whetstone.mk b/package/whetstone/whetstone.mk
> index d9b45638c5..3b6ec3419b 100644
> --- a/package/whetstone/whetstone.mk
> +++ b/package/whetstone/whetstone.mk
> @@ -9,7 +9,7 @@ WHETSTONE_SOURCE = whetstone.c
> WHETSTONE_SITE = http://www.netlib.org/benchmark
>
> define WHETSTONE_EXTRACT_CMDS
> - cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
> + cp $(WHETSTONE_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
> endef
>
> define WHETSTONE_BUILD_CMDS
> diff --git a/package/wilc1000-firmware/wilc1000-firmware.mk b/package/wilc1000-firmware/wilc1000-firmware.mk
> index 5dcf19045b..6f504d67a0 100644
> --- a/package/wilc1000-firmware/wilc1000-firmware.mk
> +++ b/package/wilc1000-firmware/wilc1000-firmware.mk
> @@ -11,7 +11,7 @@ WILC1000_FIRMWARE_SOURCE = v$(WILC1000_FIRMWARE_VERSION)_Firmware.zip
> WILC1000_FIRMWARE_LICENSE = PROPRIETARY
>
> define WILC1000_FIRMWARE_EXTRACT_CMDS
> - $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(WILC1000_FIRMWARE_SOURCE)
> + $(UNZIP) -d $(BUILD_DIR) $(WILC1000_FIRMWARE_DL_DIR)/$(WILC1000_FIRMWARE_SOURCE)
> mv $(BUILD_DIR)/wireless-firmware-$(WILC1000_FIRMWARE_VERSION)_Firmware/* $(@D)
> rmdir $(BUILD_DIR)/wireless-firmware-$(WILC1000_FIRMWARE_VERSION)_Firmware
> endef
> diff --git a/package/zynq-boot-bin/zynq-boot-bin.mk b/package/zynq-boot-bin/zynq-boot-bin.mk
> index 3648c63654..c523e04f45 100644
> --- a/package/zynq-boot-bin/zynq-boot-bin.mk
> +++ b/package/zynq-boot-bin/zynq-boot-bin.mk
> @@ -12,7 +12,7 @@ ZYNQ_BOOT_BIN_LICENSE = GPL-3.0+
> HOST_ZYNQ_BOOT_BIN_DEPENDENCIES = host-python
>
> define HOST_ZYNQ_BOOT_BIN_EXTRACT_CMDS
> - cp $(DL_DIR)/$(ZYNQ_BOOT_BIN_SOURCE) $(@D)
> + cp $(ZYNQ_BOOT_BIN_DL_DIR)/$(ZYNQ_BOOT_BIN_SOURCE) $(@D)
> endef
>
> define HOST_ZYNQ_BOOT_BIN_INSTALL_CMDS
> --
> 2.16.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR)
2018-04-02 8:14 ` [Buildroot] [v4 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Maxime Hadjinlian
@ 2018-04-02 11:06 ` Yann E. MORIN
2018-04-02 12:09 ` Thomas Petazzoni
0 siblings, 1 reply; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 11:06 UTC (permalink / raw)
To: buildroot
Maxime, All,
On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> Let the infrastructure use the new variable $(PKG)_DL_DIR
There were a few remaining other instances that needed changing, we've
done that IRL together.
Regards,
Yann E. MORIN.
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
> package/pkg-download.mk | 2 +-
> package/pkg-generic.mk | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 2ee745fddd..151b2e5423 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -95,7 +95,7 @@ define DOWNLOAD
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> -n $($(PKG)_BASENAME_RAW) \
> -N $($(PKG)_RAWNAME) \
> - -o $(DL_DIR)/$(notdir $(1)) \
> + -o $($(PKG)_DL_DIR)/$(notdir $(1)) \
> $(if $($(PKG)_GIT_SUBMODULES),-r) \
> $(DOWNLOAD_URIS) \
> $(QUIET) \
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 25a34e6ca2..e7f4a0402d 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -613,7 +613,7 @@ $(2)_TARGET_DIRCLEAN = $$($(2)_DIR)/.stamp_dircleaned
>
> # default extract command
> $(2)_EXTRACT_CMDS ?= \
> - $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$(DL_DIR)/$$($(2)_SOURCE) | \
> + $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$($(2)_DL_DIR)/$$($(2)_SOURCE) | \
> $$(TAR) --strip-components=$$($(2)_STRIP_COMPONENTS) \
> -C $$($(2)_DIR) \
> $$(foreach x,$$($(2)_EXCLUDES),--exclude='$$(x)' ) \
> --
> 2.16.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 06/13] support/download: make sure the download folder is created
2018-04-02 8:14 ` [Buildroot] [v4 06/13] support/download: make sure the download folder is created Maxime Hadjinlian
@ 2018-04-02 11:08 ` Yann E. MORIN
2018-04-02 12:10 ` Thomas Petazzoni
1 sibling, 0 replies; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 11:08 UTC (permalink / raw)
To: buildroot
Maxime, All,
On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> At the moment, it means that we make sure that BR2_DL_DIR is created, in
> the future, it will make sure that BR2_DL_DIR/PKG_NAME/ is created.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> package/pkg-download.mk | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 151b2e5423..e88d3f9fda 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -88,7 +88,8 @@ endif
> endif
>
> define DOWNLOAD
> - $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
> + $(Q)mkdir -p $($(PKG)_DL_DIR)/
> + $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1))) \
This extra semi-colon should not be added in the first place, in patch
2/13.
Regards,
Yann E. MORIN.
> $(EXTRA_ENV) $(DL_WRAPPER) \
> -c $($(PKG)_DL_VERSION) \
> -f $(notdir $(1)) \
> --
> 2.16.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 09/13] pkg-generic: introduce _SAME_SOURCE_AS
2018-04-02 8:14 ` [Buildroot] [v4 09/13] pkg-generic: introduce _SAME_SOURCE_AS Maxime Hadjinlian
@ 2018-04-02 11:57 ` Yann E. MORIN
0 siblings, 0 replies; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 11:57 UTC (permalink / raw)
To: buildroot
Maxime, All,
On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> This per package variable can be used to specify that a package shares
> the same sources as another package.
>
> The use case here is for example linux-headers and linux, which share
> the same sources (because they are the same upstream project), so we
> don't want to download twice the kernel, nor store it multiple times
> either.
>
> Make will automatically try to help by introducing leading and trailing
> spaces when replacing a line-continuation '\', so we need to call
> $(strip) on the variable.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Regards,
Yann E. MORIN.
> ---
> package/pkg-generic.mk | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index c84b1eb4a2..a034c7ea02 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -432,7 +432,10 @@ endif
>
> $(2)_BASENAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
> $(2)_BASENAME_RAW = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
> -$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
> +$(2)_DL_DIR = $$(strip $$(if $$($(2)_SAME_SOURCE_AS), \
> + $$($$(call UPPERCASE,$$($(2)_SAME_SOURCE_AS))_DL_DIR), \
> + $$(DL_DIR)/$$($(2)_RAWNAME)) \
> + )
> $(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASENAME)
>
> ifndef $(2)_SUBDIR
> --
> 2.16.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR)
2018-04-02 8:14 ` [Buildroot] [v4 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Maxime Hadjinlian
@ 2018-04-02 11:59 ` Yann E. MORIN
0 siblings, 0 replies; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 11:59 UTC (permalink / raw)
To: buildroot
Maxime, All,
On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Regards,
Yann E. MORIN.
> ---
> Config.in | 3 +++
> docs/manual/adding-packages-generic.txt | 6 +++---
> 2 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/Config.in b/Config.in
> index 71b9f60e5b..c2a1925088 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -200,6 +200,9 @@ config BR2_DL_DIR
> If the Linux shell environment has defined the BR2_DL_DIR
> environment variable, then this overrides this configuration
> item.
> + The directory is organized with a subdirectory for each package.
> + Each package has its own $(LIBFOO_DL_DIR) variable that can be used
> + to find the correct path.
>
> The default is $(TOPDIR)/dl
>
> diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
> index 9d1428ad40..62906d92bb 100644
> --- a/docs/manual/adding-packages-generic.txt
> +++ b/docs/manual/adding-packages-generic.txt
> @@ -263,7 +263,7 @@ information is (assuming the package name is +libfoo+) :
> the file using this URL. Otherwise, Buildroot will assume the file
> to be downloaded is located at +LIBFOO_SITE+. Buildroot will not do
> anything with those additional files, except download them: it will
> - be up to the package recipe to use them from +$(DL_DIR)+.
> + be up to the package recipe to use them from +$(LIBFOO_DL_DIR)+.
>
> * +LIBFOO_SITE_METHOD+ determines the method used to fetch or copy the
> package source code. In many cases, Buildroot guesses the method
> @@ -554,8 +554,8 @@ In the action definitions, you can use the following variables:
> * +$(@D)+, which contains the directory in which the package source
> code has been uncompressed.
>
> -* +$(DL_DIR)+ contains the path to the directory where all the downloads made
> - by Buildroot are stored.
> +* +$(LIBFOO_DL_DIR)+ contains the path to the directory where all the downloads
> + made by Buildroot for +libfoo+ are stored in.
>
> * +$(TARGET_CC)+, +$(TARGET_LD)+, etc. to get the target
> cross-compilation utilities
> --
> 2.16.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 12/13] download: add flock call before dl-wrapper
2018-04-02 8:14 ` [Buildroot] [v4 12/13] download: add flock call before dl-wrapper Maxime Hadjinlian
@ 2018-04-02 12:00 ` Yann E. MORIN
0 siblings, 0 replies; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 12:00 UTC (permalink / raw)
To: buildroot
Maxime, All,
On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> In order to introduce the cache mechanisms, we need to have a lock on
> the $(LIBFOO_DL_DIR), otherwise it would be impossible to do parallel
> download (a shared DL_DIR for two buildroot instances).
>
> To make sure the directory exists, the mkdir call has been removed from
> the dl-wrapper and put in the infrastructure.
This sentence now is no longer needed, bevause the move jas already been
done in a previous patch...
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Regards,
Yann E. MORIN.
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> package/pkg-download.mk | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 54edbae29a..05842bd430 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -19,6 +19,7 @@ SSH := $(call qstrip,$(BR2_SSH))
> export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
>
> DL_WRAPPER = support/download/dl-wrapper
> +FLOCK = flock $($(PKG)_DL_DIR)/
>
> # DL_DIR may have been set already from the environment
> ifeq ($(origin DL_DIR),undefined)
> @@ -92,7 +93,7 @@ endif
> define DOWNLOAD
> $(Q)mkdir -p $($(PKG)_DL_DIR)/
> $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1))) \
> - $(EXTRA_ENV) $(DL_WRAPPER) \
> + $(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
> -c $($(PKG)_DL_VERSION) \
> -f $(notdir $(1)) \
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> --
> 2.16.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 04/13] arc/xtensa: store the eXtensa overlay in the per-package DL_DIR
2018-04-02 8:14 ` [Buildroot] [v4 04/13] arc/xtensa: store the eXtensa overlay in the per-package DL_DIR Maxime Hadjinlian
@ 2018-04-02 12:08 ` Thomas Petazzoni
0 siblings, 0 replies; 30+ messages in thread
From: Thomas Petazzoni @ 2018-04-02 12:08 UTC (permalink / raw)
To: buildroot
Hello,
On Mon, 2 Apr 2018 10:14:25 +0200, Maxime Hadjinlian wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
The architecture is not eXtensa but Xtensa (i.e the commit title is
wrong).
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR)
2018-04-02 11:06 ` Yann E. MORIN
@ 2018-04-02 12:09 ` Thomas Petazzoni
0 siblings, 0 replies; 30+ messages in thread
From: Thomas Petazzoni @ 2018-04-02 12:09 UTC (permalink / raw)
To: buildroot
Hello,
On Mon, 2 Apr 2018 13:06:57 +0200, Yann E. MORIN wrote:
> Maxime, All,
>
> On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> > Let the infrastructure use the new variable $(PKG)_DL_DIR
>
> There were a few remaining other instances that needed changing, we've
> done that IRL together.
And also $($(PKG)_DL_DIR) is not "new".
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 13/13] download: git: introduce cache feature
2018-04-02 8:14 ` [Buildroot] [v4 13/13] download: git: introduce cache feature Maxime Hadjinlian
@ 2018-04-02 12:09 ` Yann E. MORIN
0 siblings, 0 replies; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 12:09 UTC (permalink / raw)
To: buildroot
On 2018-04-02 10:14 +0200, Maxime Hadjinlian spake thusly:
> Now we keep the git clone that we download and generates our tarball
> from there.
> The main goal here is that if you change the version of a package (say
> Linux), instead of cloning all over again, you will simply 'git fetch'
> from the repo the missing objects, then generates the tarball again.
>
> This should speed the 'source' part of the build significantly.
>
> The drawback is that the DL_DIR will grow much larger; but time is more
> important than disk space nowadays.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> support/download/git | 63 ++++++++++++++++++++++++++++++++--------------------
> 1 file changed, 39 insertions(+), 24 deletions(-)
>
> diff --git a/support/download/git b/support/download/git
> index 58a2c6ad9d..c6b0f81f13 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -39,29 +39,40 @@ _git() {
> eval ${GIT} "${@}"
> }
>
> -# Try a shallow clone, since it is faster than a full clone - but that only
> -# works if the version is a ref (tag or branch). Before trying to do a shallow
> -# clone we check if ${cset} is in the list provided by git ls-remote. If not
> -# we fall back on a full clone.
> +# We want to check if a cache of the git clone of this repo already exists.
Comment is now incorrect. Waht about:
# Location of the local git cache
> +git_cache="${BR2_DL_DIR}/${basename%%-*}/git"
> +
> +# If the cache directory already exists, don't try to clone.
Comment is now incorrect. What about:
# If there is no local git cache yet, initialise an empty
# git tree, it will be filled later via git fetch
> +if [ ! -d "${git_cache}" ]; then
> + _git init "'${git_cache}'"
> + _git -C "'${git_cache}'" remote add origin "'${uri}'"
> +fi
> +
> +pushd "${git_cache}" >/dev/null
> +
> +_git remote set-url origin "'${uri}'"
> +
> +# Try to fetch with limited depth, since it is faster than a full clone - but
> +# that only works if the version is a ref (tag or branch). Before trying to do
> +# a shallow clone we check if ${cset} is in the list provided by git ls-remote.
> +# If not we fall back on a full fetch.
* fallback _to_
(even if that was like that before, take the opportunity to fix it.)
> #
> -# Messages for the type of clone used are provided to ease debugging in case of
> -# problems
> +# Messages for the type of clone used are provided to ease debugging in
> +# case of problems
> git_done=0
> -if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
> - printf "Doing shallow clone\n"
> - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
> +if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then
> + printf "Doing fetch with limited depth\n"
* Doing a shallow fetch
> + if _git fetch "${@}" --depth 1 origin "'${cset}'"; then
> git_done=1
> else
> - printf "Shallow clone failed, falling back to doing a full clone\n"
> + printf "Fetching ref failed, falling back to fetching all refs\n"
To be in-line with the prevbious message, what about:
Shallow fetch failed, falling back to...
> fi
> fi
> if [ ${git_done} -eq 0 ]; then
> - printf "Doing full clone\n"
> - _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
> + printf "Fetching all references\n"
> + _git fetch origin -t
> fi
>
> -pushd "${basename}" >/dev/null
> -
> # Try to get the special refs exposed by some forges (pull-requests for
> # github, changes for gerrit...). There is no easy way to know whether
> # the cset the user passed us is such a special ref or a tag or a sha1
> @@ -86,20 +97,24 @@ if [ ${recurse} -eq 1 ]; then
> _git submodule update --init --recursive
> fi
>
> -# We do not want the .git dir; we keep other .git files, in case they
> -# are the only files in their directory.
> +# Generate the archive, sort with the C locale so that it is reproducible
> +# We do not want the .git dir; we keep other .git
> +# files, in case they are the only files in their directory.
Bad wrapping, line 2 is too short...
Regards,
Yann E. MORIN.
> # The .git dir would generate non reproducible tarballs as it depends on
> # the state of the remote server. It also would generate large tarballs
> # (gigabytes for some linux trees) when a full clone took place.
> -rm -rf .git
> +find . -not -type d \
> + -and -not -path "./.git/*" >"${output}.list"
> +LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
>
> -popd >/dev/null
> -
> -# Generate the archive, sort with the C locale so that it is reproducible
> -find "${basename}" -not -type d >"${basename}.list"
> -LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
> # Create GNU-format tarballs, since that's the format of the tarballs on
> # sources.buildroot.org and used in the *.hash files
> -tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
> - -T "${basename}.list.sorted" >"${output}.tar"
> +tar cf - --transform="s/^\.$/${basename}/" \
> + --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
> + -T "${output}.list.sorted" >"${output}.tar"
> gzip -6 -n <"${output}.tar" >"${output}"
> +
> +rm -f "${output}.list"
> +rm -f "${output}.list.sorted"
> +
> +popd >/dev/null
> --
> 2.16.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 06/13] support/download: make sure the download folder is created
2018-04-02 8:14 ` [Buildroot] [v4 06/13] support/download: make sure the download folder is created Maxime Hadjinlian
2018-04-02 11:08 ` Yann E. MORIN
@ 2018-04-02 12:10 ` Thomas Petazzoni
1 sibling, 0 replies; 30+ messages in thread
From: Thomas Petazzoni @ 2018-04-02 12:10 UTC (permalink / raw)
To: buildroot
Hello,
On Mon, 2 Apr 2018 10:14:27 +0200, Maxime Hadjinlian wrote:
> At the moment, it means that we make sure that BR2_DL_DIR is created, in
> the future, it will make sure that BR2_DL_DIR/PKG_NAME/ is created.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> package/pkg-download.mk | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 151b2e5423..e88d3f9fda 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -88,7 +88,8 @@ endif
> endif
>
> define DOWNLOAD
> - $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
> + $(Q)mkdir -p $($(PKG)_DL_DIR)/
Why is the trailing slash needed ?
> + $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1))) \
Why is this semi-colon removed ? Is this related to creating the
download folder ?
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 08/13] pkg-download: support new subdir for mirrors
2018-04-02 8:14 ` [Buildroot] [v4 08/13] pkg-download: support new subdir for mirrors Maxime Hadjinlian
@ 2018-04-02 12:13 ` Thomas Petazzoni
2018-04-02 12:29 ` Yann E. MORIN
0 siblings, 1 reply; 30+ messages in thread
From: Thomas Petazzoni @ 2018-04-02 12:13 UTC (permalink / raw)
To: buildroot
Hello,
On Mon, 2 Apr 2018 10:14:29 +0200, Maxime Hadjinlian wrote:
> Since we introduced subdirectory to the DL_DIR, we need to support them
> in the PRIMARY and BACKUP mirror as they evolve to the new tree
> structure.
>
> We check first the new URI (with the subdir), and in case of failure, we
> go check without.
> By checking both URIs, we ensure that old mirror are usable.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Shouldn't this be squashed into the previous patch ?
Indeed, as soon as PATCH 07/13 is merged, the sources.buildroot.net
mirror will start storing new tarballs in sources.buildroot.net/<pkg>/
and not directly in sources.buildroot.net/. So we should at the same
time start storing tarballs in the per-package sub-folder *and* start
querying primary/backup site inside their sub-folder.
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 02/13] download: put most of the infra in dl-wrapper
2018-04-02 8:14 ` [Buildroot] [v4 02/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
2018-04-02 10:23 ` Yann E. MORIN
@ 2018-04-02 12:18 ` Peter Korsgaard
1 sibling, 0 replies; 30+ messages in thread
From: Peter Korsgaard @ 2018-04-02 12:18 UTC (permalink / raw)
To: buildroot
>>>>> "Maxime" == Maxime Hadjinlian <maxime.hadjinlian@gmail.com> writes:
> The goal here is to simplify the infrastructure by putting most of the
> code in the dl-wrapper as it's easier to implement and to read.
> Most of the functions were common already, this patch finalizes it by
> making the pkg-download.mk pass all the parameters needed to the
> dl-wrapper which in turns will pass everything to every backend.
s/in turns/in turn/
> + # If the backend fails, we can just remove the content of the temporary
> + # directory to remove all the cruft it may have left behind, and tries
s/tries/try/ (both lines).
Committed with these minor fixes + what Yann pointed out, thanks.
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Buildroot] [v4 08/13] pkg-download: support new subdir for mirrors
2018-04-02 12:13 ` Thomas Petazzoni
@ 2018-04-02 12:29 ` Yann E. MORIN
0 siblings, 0 replies; 30+ messages in thread
From: Yann E. MORIN @ 2018-04-02 12:29 UTC (permalink / raw)
To: buildroot
Thomas, Maxime, All,
On 2018-04-02 14:13 +0200, Thomas Petazzoni spake thusly:
> On Mon, 2 Apr 2018 10:14:29 +0200, Maxime Hadjinlian wrote:
> > Since we introduced subdirectory to the DL_DIR, we need to support them
> > in the PRIMARY and BACKUP mirror as they evolve to the new tree
> > structure.
> >
> > We check first the new URI (with the subdir), and in case of failure, we
> > go check without.
> > By checking both URIs, we ensure that old mirror are usable.
> >
> > Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> > Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Shouldn't this be squashed into the previous patch ?
Or rather, the order should be inverted: this patch first, then the
previous one.
That way, we start with looking in sub-dirs (we do not yet exist), then
fallback to plain dir. Then sub-dirs are introduced and pushed to the
mirror.
Regards,
Yann E. MORIN.
> Indeed, as soon as PATCH 07/13 is merged, the sources.buildroot.net
> mirror will start storing new tarballs in sources.buildroot.net/<pkg>/
> and not directly in sources.buildroot.net/. So we should at the same
> time start storing tarballs in the per-package sub-folder *and* start
> querying primary/backup site inside their sub-folder.
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
> Embedded Linux and Kernel engineering
> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2018-04-02 12:29 UTC | newest]
Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-02 8:14 [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Maxime Hadjinlian
2018-04-02 8:14 ` [Buildroot] [v4 02/13] download: put most of the infra in dl-wrapper Maxime Hadjinlian
2018-04-02 10:23 ` Yann E. MORIN
2018-04-02 12:18 ` Peter Korsgaard
2018-04-02 8:14 ` [Buildroot] [v4 03/13] packages: use new $($PKG)_DL_DIR) variable Maxime Hadjinlian
2018-04-02 10:36 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 04/13] arc/xtensa: store the eXtensa overlay in the per-package DL_DIR Maxime Hadjinlian
2018-04-02 12:08 ` Thomas Petazzoni
2018-04-02 8:14 ` [Buildroot] [v4 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Maxime Hadjinlian
2018-04-02 11:06 ` Yann E. MORIN
2018-04-02 12:09 ` Thomas Petazzoni
2018-04-02 8:14 ` [Buildroot] [v4 06/13] support/download: make sure the download folder is created Maxime Hadjinlian
2018-04-02 11:08 ` Yann E. MORIN
2018-04-02 12:10 ` Thomas Petazzoni
2018-04-02 8:14 ` [Buildroot] [v4 07/13] pkg-generic: add a subdirectory to the DL_DIR Maxime Hadjinlian
2018-04-02 8:14 ` [Buildroot] [v4 08/13] pkg-download: support new subdir for mirrors Maxime Hadjinlian
2018-04-02 12:13 ` Thomas Petazzoni
2018-04-02 12:29 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 09/13] pkg-generic: introduce _SAME_SOURCE_AS Maxime Hadjinlian
2018-04-02 11:57 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 10/13] package: share downloaded files for big packages Maxime Hadjinlian
2018-04-02 8:14 ` [Buildroot] [v4 11/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Maxime Hadjinlian
2018-04-02 11:59 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 12/13] download: add flock call before dl-wrapper Maxime Hadjinlian
2018-04-02 12:00 ` Yann E. MORIN
2018-04-02 8:14 ` [Buildroot] [v4 13/13] download: git: introduce cache feature Maxime Hadjinlian
2018-04-02 12:09 ` Yann E. MORIN
2018-04-02 8:25 ` [Buildroot] [v4 01/13] core/pkg-download: change all helpers to use common options Yann E. MORIN
2018-04-02 8:40 ` Thomas Petazzoni
2018-04-02 10:26 ` Peter Korsgaard
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.