* [Buildroot] [PATCH 1/3 v10] support/download: support -q in all download backends
2015-01-02 15:53 [Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent) Yann E. MORIN
@ 2015-01-02 15:53 ` Yann E. MORIN
2015-01-02 15:53 ` [Buildroot] [PATCH 2/3 v10] pkg-download: silence downloads if make is silent Yann E. MORIN
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Yann E. MORIN @ 2015-01-02 15:53 UTC (permalink / raw)
To: buildroot
Add an option flag to all backends, as well as the check-hash script, so
as to silence download helpers when the user wants a silent build.
Additionaly, make the default be verbose.
Inspired by Fabio's patch on git/svn.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
---
Note: cvs, cp and scp are not tested, because we do not have in-tree
packages using those transport methods.
---
support/download/bzr | 23 ++++++++++++++++-------
support/download/check-hash | 8 ++++++++
support/download/cp | 20 +++++++++++++++-----
support/download/cvs | 23 +++++++++++++++--------
support/download/git | 24 ++++++++++++++++--------
support/download/hg | 24 ++++++++++++++++--------
support/download/scp | 20 +++++++++++++++-----
support/download/svn | 22 +++++++++++++++-------
support/download/wget | 20 +++++++++++++++-----
9 files changed, 131 insertions(+), 53 deletions(-)
diff --git a/support/download/bzr b/support/download/bzr
index c157ca8..5819ac7 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -4,17 +4,26 @@
set -e
# Download helper for bzr, to be called from the download wrapper script
-# Expected arguments:
-# $1: output file
-# $2: bzr repo
-# $3: bzr revision
-# $4: basename
-# And this environment:
+#
+# Call it as:
+# .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
+#
+# Environemnt:
# BZR : the bzr command to call
+
+verbose=-v
+while getopts :q OPT; do
+ case "${OPT}" in
+ q) verbose=-q;;
+ \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+ esac
+done
+shift $((OPTIND-1))
+
output="${1}"
repo="${2}"
rev="${3}"
basename="${4}"
-${BZR} export --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
+${BZR} export ${verbose} --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
diff --git a/support/download/check-hash b/support/download/check-hash
index b59fd2a..4c07274 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -10,6 +10,14 @@ set -e
# saved as, to be able to match it to the corresponding hashes
# in the .hash file
+while getopts :q OPT; do
+ case "${OPT}" in
+ q) exec >/dev/null;;
+ \?) exit 1;;
+ esac
+done
+shift $((OPTIND-1))
+
h_file="${1}"
file="${2}"
base="${3}"
diff --git a/support/download/cp b/support/download/cp
index 463fc38..132cad7 100755
--- a/support/download/cp
+++ b/support/download/cp
@@ -4,13 +4,23 @@
set -e
# Download helper for cp, to be called from the download wrapper script
-# Expected arguments:
-# $1: output file
-# $2: source file
-# And this environment:
+#
+# Call it as:
+# .../cp [-q] OUT_FILE SRC_FILE
+#
+# Environment:
# LOCALFILES: the cp command to call
+verbose=-v
+while getopts :q OPT; do
+ case "${OPT}" in
+ q) verbose=;;
+ \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+ esac
+done
+shift $((OPTIND-1))
+
output="${1}"
source="${2}"
-${LOCALFILES} "${source}" "${output}"
+${LOCALFILES} ${verbose} "${source}" "${output}"
diff --git a/support/download/cvs b/support/download/cvs
index 56a11c2..2c3a666 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -4,22 +4,29 @@
set -e
# Download helper for cvs, to be called from the download wrapper script
-# Expected arguments:
-# $1: output file
-# $2: cvs repo
-# $3: cvs revision
-# $4: package's name (eg. foobar)
-# $5: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+# .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
+#
+# Environment:
# CVS : the cvs command to call
+verbose=
+while getopts :q OPT; do
+ case "${OPT}" in
+ q) verbose=-Q;;
+ \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+ esac
+done
+shift $((OPTIND-1))
+
output="${1}"
repo="${2}"
rev="${3}"
rawname="${4}"
basename="${5}"
-${CVS} -z3 -d":pserver:anonymous@${repo}" \
+${CVS} ${verbose} -z3 -d":pserver:anonymous@${repo}" \
co -d "${basename}" -r ":${rev}" -P "${rawname}"
tar czf "${output}" "${basename}"
diff --git a/support/download/git b/support/download/git
index 5d36ca4..7f1801f 100755
--- a/support/download/git
+++ b/support/download/git
@@ -4,14 +4,22 @@
set -e
# Download helper for git, to be called from the download wrapper script
-# Expected arguments:
-# $1: output file
-# $2: git repo
-# $3: git cset
-# $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+# .../git [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
# GIT : the git command to call
+verbose=-v
+while getopts :q OPT; do
+ case "${OPT}" in
+ q) verbose=-q;;
+ \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+ esac
+done
+shift $((OPTIND-1))
+
output="${1}"
repo="${2}"
cset="${3}"
@@ -22,7 +30,7 @@ basename="${4}"
git_done=0
if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
printf "Doing shallow clone\n"
- if ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
+ if ${GIT} clone ${verbose} --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
git_done=1
else
printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -30,7 +38,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
fi
if [ ${git_done} -eq 0 ]; then
printf "Doing full clone\n"
- ${GIT} clone --bare "${repo}" "${basename}"
+ ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
fi
GIT_DIR="${basename}" \
diff --git a/support/download/hg b/support/download/hg
index 66bd2ed..f6880fa 100755
--- a/support/download/hg
+++ b/support/download/hg
@@ -4,21 +4,29 @@
set -e
# Download helper for hg, to be called from the download wrapper script
-# Expected arguments:
-# $1: output file
-# $2: hg repo
-# $3: hg cset
-# $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+# .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
# HG : the hg command to call
+verbose=-v
+while getopts :q OPT; do
+ case "${OPT}" in
+ q) verbose=-q;;
+ \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+ esac
+done
+shift $((OPTIND-1))
+
output="${1}"
repo="${2}"
cset="${3}"
basename="${4}"
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
+${HG} clone ${verbose} --noupdate --rev "${cset}" "${repo}" "${basename}"
-${HG} archive --repository "${basename}" --type tgz \
+${HG} archive ${verbose} --repository "${basename}" --type tgz \
--prefix "${basename}" --rev "${cset}" \
"${output}"
diff --git a/support/download/scp b/support/download/scp
index f3e92f3..3d6876f 100755
--- a/support/download/scp
+++ b/support/download/scp
@@ -4,13 +4,23 @@
set -e
# Download helper for scp, to be called from the download wrapper script
-# Expected arguments:
-# $1: output file
-# $2: URL
-# And this environment:
+#
+# Call it as:
+# .../scp [-q] OUT_FILE SRC_URL
+#
+# Environment:
# SCP : the scp command to call
+verbose=-v
+while getopts :q OPT; do
+ case "${OPT}" in
+ q) verbose=-q;;
+ \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+ esac
+done
+shift $((OPTIND-1))
+
output="${1}"
url="${2}"
-${SCP} "${url}" "${output}"
+${SCP} ${verbose} "${url}" "${output}"
diff --git a/support/download/svn b/support/download/svn
index a960f7d..558bca0 100755
--- a/support/download/svn
+++ b/support/download/svn
@@ -4,19 +4,27 @@
set -e
# Download helper for svn, to be called from the download wrapper script
-# Expected arguments:
-# $1: output file
-# $2: svn repo
-# $3: svn revision
-# $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+# .../svn [-q] OUT_FILE REPO_URL REV BASNAME
+#
+# Environment:
# SVN : the svn command to call
+verbose=
+while getopts :q OPT; do
+ case "${OPT}" in
+ q) verbose=-q;;
+ \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+ esac
+done
+shift $((OPTIND-1))
+
output="${1}"
repo="${2}"
rev="${3}"
basename="${4}"
-${SVN} export "${repo}@${rev}" "${basename}"
+${SVN} export ${verbose} "${repo}@${rev}" "${basename}"
tar czf "${output}" "${basename}"
diff --git a/support/download/wget b/support/download/wget
index 6b73726..7eb21a4 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -4,13 +4,23 @@
set -e
# Download helper for wget, to be called from the download wrapper script
-# Expected arguments:
-# $1: output file
-# $2: URL
-# And this environment:
+#
+# Call it as:
+# .../wget [-q] OUT_FILE URL
+#
+# Environment:
# WGET : the wget command to call
+verbose=-v
+while getopts :q OPT; do
+ case "${OPT}" in
+ q) verbose=-q;;
+ \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+ esac
+done
+shift $((OPTIND-1))
+
output="${1}"
url="${2}"
-${WGET} -O "${output}" "${url}"
+${WGET} ${verbose} -O "${output}" "${url}"
--
1.9.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [Buildroot] [PATCH 2/3 v10] pkg-download: silence downloads if make is silent
2015-01-02 15:53 [Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent) Yann E. MORIN
2015-01-02 15:53 ` [Buildroot] [PATCH 1/3 v10] support/download: support -q in all download backends Yann E. MORIN
@ 2015-01-02 15:53 ` Yann E. MORIN
2015-01-02 15:53 ` [Buildroot] [PATCH 3/3 v10] package/pkg-download: do not hard-code QUIET Yann E. MORIN
2015-01-04 17:58 ` [Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent) Thomas Petazzoni
3 siblings, 0 replies; 5+ messages in thread
From: Yann E. MORIN @ 2015-01-02 15:53 UTC (permalink / raw)
To: buildroot
From: Fabio Porcedda <fabio.porcedda@gmail.com>
If doing a silent build (make -s -> QUIET=-q), silence all downloads,
by passing the -q flag downward to backends as well as to check-hash.
Change a printf to use the trace functions.
Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
package/pkg-download.mk | 8 ++++++++
support/download/dl-wrapper | 13 +++++++------
2 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index ba72fc1..9c782fe 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -88,6 +88,7 @@ define DOWNLOAD_GIT
$(EXTRA_ENV) $(DL_WRAPPER) -b git \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ $(QUIET) \
-- \
$($(PKG)_SITE) \
$($(PKG)_DL_VERSION) \
@@ -109,6 +110,7 @@ define DOWNLOAD_BZR
$(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ $(QUIET) \
-- \
$($(PKG)_SITE) \
$($(PKG)_DL_VERSION) \
@@ -127,6 +129,7 @@ define DOWNLOAD_CVS
$(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ $(QUIET) \
-- \
$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
$($(PKG)_DL_VERSION) \
@@ -147,6 +150,7 @@ define DOWNLOAD_SVN
$(EXTRA_ENV) $(DL_WRAPPER) -b svn \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ $(QUIET) \
-- \
$($(PKG)_SITE) \
$($(PKG)_DL_VERSION) \
@@ -168,6 +172,7 @@ define DOWNLOAD_SCP
$(EXTRA_ENV) $(DL_WRAPPER) -b scp \
-o $(DL_DIR)/$(2) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ $(QUIET) \
-- \
'$(call stripurischeme,$(call qstrip,$(1)))'
endef
@@ -185,6 +190,7 @@ define DOWNLOAD_HG
$(EXTRA_ENV) $(DL_WRAPPER) -b hg \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ $(QUIET) \
-- \
$($(PKG)_SITE) \
$($(PKG)_DL_VERSION) \
@@ -206,6 +212,7 @@ define DOWNLOAD_WGET
$(EXTRA_ENV) $(DL_WRAPPER) -b wget \
-o $(DL_DIR)/$(2) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ $(QUIET) \
-- \
'$(call qstrip,$(1))'
endef
@@ -222,6 +229,7 @@ define DOWNLOAD_LOCALFILES
$(EXTRA_ENV) $(DL_WRAPPER) -b cp \
-o $(DL_DIR)/$(2) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ $(QUIET) \
-- \
$(call stripurischeme,$(call qstrip,$(1)))
endef
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index cced8f6..3b30840 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -21,15 +21,16 @@ set -e
main() {
local OPT OPTARG
- local backend output hfile
+ local backend output hfile quiet
# Parse our options; anything after '--' is for the backend
- while getopts :hb:o:H: OPT; do
+ while getopts :hb:o:H:q OPT; do
case "${OPT}" in
h) help; exit 0;;
b) backend="${OPTARG}";;
o) output="${OPTARG}";;
H) hfile="${OPTARG}";;
+ q) quiet="-q";;
:) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
\?) error "unknown option '%s'\n" "${OPTARG}";;
esac
@@ -49,11 +50,11 @@ main() {
# If the output file already exists, do not download it again
if [ -e "${output}" ]; then
- if support/download/check-hash "${hfile}" "${output}" "${output##*/}"; then
+ if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
exit 0
fi
rm -f "${output}"
- printf "Re-downloading '%s'...\n" "${output##*/}"
+ warn "Re-downloading '%s'...\n" "${output##*/}"
fi
# tmpd is a temporary directory in which backends may store intermediate
@@ -75,7 +76,7 @@ main() {
# If the backend fails, we can just remove the temporary directory to
# remove all the cruft it may have left behind. Then we just exit in
# error too.
- if ! "${OLDPWD}/support/download/${backend}" "${tmpf}" "${@}"; then
+ if ! "${OLDPWD}/support/download/${backend}" ${quiet} "${tmpf}" "${@}"; then
rm -rf "${tmpd}"
exit 1
fi
@@ -85,7 +86,7 @@ main() {
# Check if the downloaded file is sane, and matches the stored hashes
# for that file
- if ! support/download/check-hash "${hfile}" "${tmpf}" "${output##*/}"; then
+ if ! support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
rm -rf "${tmpd}"
exit 1
fi
--
1.9.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [Buildroot] [PATCH 3/3 v10] package/pkg-download: do not hard-code QUIET
2015-01-02 15:53 [Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent) Yann E. MORIN
2015-01-02 15:53 ` [Buildroot] [PATCH 1/3 v10] support/download: support -q in all download backends Yann E. MORIN
2015-01-02 15:53 ` [Buildroot] [PATCH 2/3 v10] pkg-download: silence downloads if make is silent Yann E. MORIN
@ 2015-01-02 15:53 ` Yann E. MORIN
2015-01-04 17:58 ` [Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent) Thomas Petazzoni
3 siblings, 0 replies; 5+ messages in thread
From: Yann E. MORIN @ 2015-01-02 15:53 UTC (permalink / raw)
To: buildroot
Do not hard-code QUIET in our download commands, since it is handled in
the backends.
Suggested by Fabio.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
---
package/pkg-download.mk | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 9c782fe..5e74519 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -8,14 +8,14 @@
################################################################################
# Download method commands
-export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
+export WGET := $(call qstrip,$(BR2_WGET))
export SVN := $(call qstrip,$(BR2_SVN))
export CVS := $(call qstrip,$(BR2_CVS))
export BZR := $(call qstrip,$(BR2_BZR))
export GIT := $(call qstrip,$(BR2_GIT))
-export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
-export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
-SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
+export HG := $(call qstrip,$(BR2_HG))
+export SCP := $(call qstrip,$(BR2_SCP))
+SSH := $(call qstrip,$(BR2_SSH))
export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
DL_WRAPPER = support/download/dl-wrapper
--
1.9.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent)
2015-01-02 15:53 [Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent) Yann E. MORIN
` (2 preceding siblings ...)
2015-01-02 15:53 ` [Buildroot] [PATCH 3/3 v10] package/pkg-download: do not hard-code QUIET Yann E. MORIN
@ 2015-01-04 17:58 ` Thomas Petazzoni
3 siblings, 0 replies; 5+ messages in thread
From: Thomas Petazzoni @ 2015-01-04 17:58 UTC (permalink / raw)
To: buildroot
Dear Yann E. MORIN,
On Fri, 2 Jan 2015 16:53:37 +0100, Yann E. MORIN wrote:
> Silent builds are also nice when top-level parallel make is being used
> to reduce the output clutter.
>
> Some downloaders, like git or wget, insist on reporting progress on
> stderr instead of stdout, so redirecting stdout is not enough, we need
> to pass the quiet information all the way down to the beackends, so
> they know they have to be silent. Sigh... :-(
Series applied, after fixing a minor typo in the first patch. Thanks to
you and Fabio for working on this!
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 5+ messages in thread