[Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent)

[Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent)

[Yann E. MORIN]

Hello All!

This patch set improves silent builds.

Silent builds are also nice when top-level parallel make is being used
to reduce the output clutter.

Some downloaders, like git or wget, insist on reporting progress on
stderr instead of stdout, so redirecting stdout is not enough, we need
to pass the quiet information all the way down to the beackends, so
they know they have to be silent. Sigh... :-(


Changes v9 -> v10:
  - do not redirect stdout at all  (Thomas)
  - on unknown options, print an error message in backends  (Fabio)

Changes v8 -> v9:
  - propagate to all download backends  (Fabio)

Changes v7 -> v8:
  - simplify by just consigning stdout to oblivion  (Yann)


Regards,
Yann E. MORIN.


The following changes since commit 81cbf0bf84bb62d1b40f9ea4e30296b5ef495556:

  libevent: add optional dependency on OpenSSL (2015-01-02 16:37:48 +0100)

are available in the git repository at:

  git://git.busybox.net/~ymorin/git/buildroot yem/fabio/silent

for you to fetch changes up to 9b12978224418fe368d83e1ca7988619a18e9b74:

  package/pkg-download: do not hard-code QUIET (2015-01-02 16:51:53 +0100)

----------------------------------------------------------------
Fabio Porcedda (1):
      pkg-download: silence downloads if make is silent

Yann E. MORIN (2):
      support/download: support -q in all download backends
      package/pkg-download: do not hard-code QUIET

 package/pkg-download.mk     | 16 ++++++++++++----
 support/download/bzr        | 23 ++++++++++++++++-------
 support/download/check-hash |  8 ++++++++
 support/download/cp         | 20 +++++++++++++++-----
 support/download/cvs        | 23 +++++++++++++++--------
 support/download/dl-wrapper | 13 +++++++------
 support/download/git        | 24 ++++++++++++++++--------
 support/download/hg         | 24 ++++++++++++++++--------
 support/download/scp        | 20 +++++++++++++++-----
 support/download/svn        | 22 +++++++++++++++-------
 support/download/wget       | 20 +++++++++++++++-----
 11 files changed, 150 insertions(+), 63 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/3 v10] support/download: support -q in all download backends

[Yann E. MORIN]

Add an option flag to all backends, as well as the check-hash script, so
as to silence download helpers when the user wants a silent build.

Additionaly, make the default be verbose.

Inspired by Fabio's patch on git/svn.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>

---
Note: cvs, cp and scp are not tested, because we do not have in-tree
packages using those transport methods.
---
 support/download/bzr        | 23 ++++++++++++++++-------
 support/download/check-hash |  8 ++++++++
 support/download/cp         | 20 +++++++++++++++-----
 support/download/cvs        | 23 +++++++++++++++--------
 support/download/git        | 24 ++++++++++++++++--------
 support/download/hg         | 24 ++++++++++++++++--------
 support/download/scp        | 20 +++++++++++++++-----
 support/download/svn        | 22 +++++++++++++++-------
 support/download/wget       | 20 +++++++++++++++-----
 9 files changed, 131 insertions(+), 53 deletions(-)

diff --git a/support/download/bzr b/support/download/bzr
index c157ca8..5819ac7 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -4,17 +4,26 @@
 set -e
 
 # Download helper for bzr, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: bzr repo
-#   $3: bzr revision
-#   $4: basename
-# And this environment:
+#
+# Call it as:
+#   .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
+#
+# Environemnt:
 #   BZR      : the bzr command to call
 
+
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${BZR} export --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
+${BZR} export ${verbose} --root="${basename}/" --format=tgz "${output}" "${repo}" -r "${rev}"
diff --git a/support/download/check-hash b/support/download/check-hash
index b59fd2a..4c07274 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -10,6 +10,14 @@ set -e
 #       saved as, to be able to match it to the corresponding hashes
 #       in the .hash file
 
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  exec >/dev/null;;
+    \?) exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 h_file="${1}"
 file="${2}"
 base="${3}"
diff --git a/support/download/cp b/support/download/cp
index 463fc38..132cad7 100755
--- a/support/download/cp
+++ b/support/download/cp
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for cp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: source file
-# And this environment:
+#
+# Call it as:
+#   .../cp [-q] OUT_FILE SRC_FILE
+#
+# Environment:
 #   LOCALFILES: the cp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 source="${2}"
 
-${LOCALFILES} "${source}" "${output}"
+${LOCALFILES} ${verbose} "${source}" "${output}"
diff --git a/support/download/cvs b/support/download/cvs
index 56a11c2..2c3a666 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -4,22 +4,29 @@
 set -e
 
 # Download helper for cvs, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: cvs repo
-#   $3: cvs revision
-#   $4: package's name (eg. foobar)
-#   $5: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
+#
+# Environment:
 #   CVS      : the cvs command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-Q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 rawname="${4}"
 basename="${5}"
 
-${CVS} -z3 -d":pserver:anonymous@${repo}" \
+${CVS} ${verbose} -z3 -d":pserver:anonymous@${repo}" \
        co -d "${basename}" -r ":${rev}" -P "${rawname}"
 
 tar czf "${output}" "${basename}"
diff --git a/support/download/git b/support/download/git
index 5d36ca4..7f1801f 100755
--- a/support/download/git
+++ b/support/download/git
@@ -4,14 +4,22 @@
 set -e
 
 # Download helper for git, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: git repo
-#   $3: git cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../git [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   GIT      : the git command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"
@@ -22,7 +30,7 @@ basename="${4}"
 git_done=0
 if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
     printf "Doing shallow clone\n"
-    if ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
+    if ${GIT} clone ${verbose} --depth 1 -b "${cset}" --bare "${repo}" "${basename}"; then
         git_done=1
     else
         printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -30,7 +38,7 @@ if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
 fi
 if [ ${git_done} -eq 0 ]; then
     printf "Doing full clone\n"
-    ${GIT} clone --bare "${repo}" "${basename}"
+    ${GIT} clone ${verbose} --bare "${repo}" "${basename}"
 fi
 
 GIT_DIR="${basename}" \
diff --git a/support/download/hg b/support/download/hg
index 66bd2ed..f6880fa 100755
--- a/support/download/hg
+++ b/support/download/hg
@@ -4,21 +4,29 @@
 set -e
 
 # Download helper for hg, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: hg repo
-#   $3: hg cset
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
+#
+# Environment:
 #   HG       : the hg command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 cset="${3}"
 basename="${4}"
 
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
+${HG} clone ${verbose} --noupdate --rev "${cset}" "${repo}" "${basename}"
 
-${HG} archive --repository "${basename}" --type tgz \
+${HG} archive ${verbose} --repository "${basename}" --type tgz \
               --prefix "${basename}" --rev "${cset}" \
               "${output}"
diff --git a/support/download/scp b/support/download/scp
index f3e92f3..3d6876f 100755
--- a/support/download/scp
+++ b/support/download/scp
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for scp, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../scp [-q] OUT_FILE SRC_URL
+#
+# Environment:
 #   SCP       : the scp command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${SCP} "${url}" "${output}"
+${SCP} ${verbose} "${url}" "${output}"
diff --git a/support/download/svn b/support/download/svn
index a960f7d..558bca0 100755
--- a/support/download/svn
+++ b/support/download/svn
@@ -4,19 +4,27 @@
 set -e
 
 # Download helper for svn, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: svn repo
-#   $3: svn revision
-#   $4: package's basename (eg. foobar-1.2.3)
-# And this environment:
+#
+# Call it as:
+#   .../svn [-q] OUT_FILE REPO_URL REV BASNAME
+#
+# Environment:
 #   SVN      : the svn command to call
 
+verbose=
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 repo="${2}"
 rev="${3}"
 basename="${4}"
 
-${SVN} export "${repo}@${rev}" "${basename}"
+${SVN} export ${verbose} "${repo}@${rev}" "${basename}"
 
 tar czf "${output}" "${basename}"
diff --git a/support/download/wget b/support/download/wget
index 6b73726..7eb21a4 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -4,13 +4,23 @@
 set -e
 
 # Download helper for wget, to be called from the download wrapper script
-# Expected arguments:
-#   $1: output file
-#   $2: URL
-# And this environment:
+#
+# Call it as:
+#   .../wget [-q] OUT_FILE URL
+#
+# Environment:
 #   WGET     : the wget command to call
 
+verbose=-v
+while getopts :q OPT; do
+    case "${OPT}" in
+    q)  verbose=-q;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+shift $((OPTIND-1))
+
 output="${1}"
 url="${2}"
 
-${WGET} -O "${output}" "${url}"
+${WGET} ${verbose} -O "${output}" "${url}"
-- 
1.9.1

[Buildroot] [PATCH 2/3 v10] pkg-download: silence downloads if make is silent

[Yann E. MORIN]

From: Fabio Porcedda <fabio.porcedda@gmail.com>

If doing a silent build (make -s -> QUIET=-q), silence all downloads,
by passing the -q flag downward to backends as well as to check-hash.

Change a printf to use the trace functions.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 package/pkg-download.mk     |  8 ++++++++
 support/download/dl-wrapper | 13 +++++++------
 2 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index ba72fc1..9c782fe 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -88,6 +88,7 @@ define DOWNLOAD_GIT
 	$(EXTRA_ENV) $(DL_WRAPPER) -b git \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \
@@ -109,6 +110,7 @@ define DOWNLOAD_BZR
 	$(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \
@@ -127,6 +129,7 @@ define DOWNLOAD_CVS
 	$(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
 		$($(PKG)_DL_VERSION) \
@@ -147,6 +150,7 @@ define DOWNLOAD_SVN
 	$(EXTRA_ENV) $(DL_WRAPPER) -b svn \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \
@@ -168,6 +172,7 @@ define DOWNLOAD_SCP
 	$(EXTRA_ENV) $(DL_WRAPPER) -b scp \
 		-o $(DL_DIR)/$(2) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		'$(call stripurischeme,$(call qstrip,$(1)))'
 endef
@@ -185,6 +190,7 @@ define DOWNLOAD_HG
 	$(EXTRA_ENV) $(DL_WRAPPER) -b hg \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
 		$($(PKG)_DL_VERSION) \
@@ -206,6 +212,7 @@ define DOWNLOAD_WGET
 	$(EXTRA_ENV) $(DL_WRAPPER) -b wget \
 		-o $(DL_DIR)/$(2) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		'$(call qstrip,$(1))'
 endef
@@ -222,6 +229,7 @@ define DOWNLOAD_LOCALFILES
 	$(EXTRA_ENV) $(DL_WRAPPER) -b cp \
 		-o $(DL_DIR)/$(2) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		$(QUIET) \
 		-- \
 		$(call stripurischeme,$(call qstrip,$(1)))
 endef
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index cced8f6..3b30840 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -21,15 +21,16 @@ set -e
 
 main() {
     local OPT OPTARG
-    local backend output hfile
+    local backend output hfile quiet
 
     # Parse our options; anything after '--' is for the backend
-    while getopts :hb:o:H: OPT; do
+    while getopts :hb:o:H:q OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
         b)  backend="${OPTARG}";;
         o)  output="${OPTARG}";;
         H)  hfile="${OPTARG}";;
+        q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
         esac
@@ -49,11 +50,11 @@ main() {
 
     # If the output file already exists, do not download it again
     if [ -e "${output}" ]; then
-        if support/download/check-hash "${hfile}" "${output}" "${output##*/}"; then
+        if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
             exit 0
         fi
         rm -f "${output}"
-        printf "Re-downloading '%s'...\n" "${output##*/}"
+        warn "Re-downloading '%s'...\n" "${output##*/}"
     fi
 
     # tmpd is a temporary directory in which backends may store intermediate
@@ -75,7 +76,7 @@ main() {
     # If the backend fails, we can just remove the temporary directory to
     # remove all the cruft it may have left behind. Then we just exit in
     # error too.
-    if ! "${OLDPWD}/support/download/${backend}" "${tmpf}" "${@}"; then
+    if ! "${OLDPWD}/support/download/${backend}" ${quiet} "${tmpf}" "${@}"; then
         rm -rf "${tmpd}"
         exit 1
     fi
@@ -85,7 +86,7 @@ main() {
 
     # Check if the downloaded file is sane, and matches the stored hashes
     # for that file
-    if ! support/download/check-hash "${hfile}" "${tmpf}" "${output##*/}"; then
+    if ! support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
         rm -rf "${tmpd}"
         exit 1
     fi
-- 
1.9.1

[Buildroot] [PATCH 3/3 v10] package/pkg-download: do not hard-code QUIET

[Yann E. MORIN]

Do not hard-code QUIET in our download commands, since it is handled in
the backends.

Suggested by Fabio.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-download.mk | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 9c782fe..5e74519 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -8,14 +8,14 @@
 ################################################################################
 
 # Download method commands
-export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
+export WGET := $(call qstrip,$(BR2_WGET))
 export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
 export BZR := $(call qstrip,$(BR2_BZR))
 export GIT := $(call qstrip,$(BR2_GIT))
-export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
-export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
-SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
+export HG := $(call qstrip,$(BR2_HG))
+export SCP := $(call qstrip,$(BR2_SCP))
+SSH := $(call qstrip,$(BR2_SSH))
 export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 DL_WRAPPER = support/download/dl-wrapper
-- 
1.9.1

[Buildroot] [PATCH 0/3 v10] Improve silent builds (branch yem/fabio/silent)

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Fri,  2 Jan 2015 16:53:37 +0100, Yann E. MORIN wrote:

> Silent builds are also nice when top-level parallel make is being used
> to reduce the output clutter.
> 
> Some downloaders, like git or wget, insist on reporting progress on
> stderr instead of stdout, so redirecting stdout is not enough, we need
> to pass the quiet information all the way down to the beackends, so
> they know they have to be silent. Sigh... :-(

Series applied, after fixing a minor typo in the first patch. Thanks to
you and Fabio for working on this!

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com