[Buildroot] [PATCH v8 00/14] Relocatable SDK/Build machine leaks: RPATH fixing

[Buildroot] [PATCH v8 00/14] Relocatable SDK/Build machine leaks: RPATH fixing

[Samuel Martin]

Hi all,


Here is yet another round of the series aiming improving the relocatability
of the SDK built by Buildroot.

This time, I reduce the series to those related to RPATH (hope a shorter
series will be easier to review...). Besides, the noticeable changes are:
- the refactoring of the O definition, as it needs to point to the
  absolute canonical path for the whole build, so that fix-rpath can do
  its job, without missing anything;
- and the refactoring of the check-host-leaks script.

For further details, check the per-commit changelog in each commit log.


Patches 1 and 2:
  Fix the O variable definition.

Patches 3 to 5:
  Add the fix-rpath script, then run it on the Buildroot host, target and
  staging trees.

Patch 6:
  Clean-up speex package WRT its RPATH hook.

Patch 7:
  Add a post-install hook to the toolchain package to render it relocatable.

Patch 8 to 10:
  Update and fix the check-host-rpath to re-use the shell helpers.

Patches 11 to 13:
  Add means to track build machine leaks into the host, target and staging
  trees. These patches will allow to identify what remains to be fixed WRT
  build machine leaks.

Patches 14:
  Update documentation about how to enable/adjust log level on script using
  the shell modules introduced in patch 8/18.


Regards,
Samuel


Samuel Martin (14):
  core: split variables definition related to in/out-of-tree build from
    O itself
  core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
    path
  support/scripts: add fix-rpath script + a bunch of helpers
  core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS
  core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
    TARGET_FINALIZE_HOOKS
  package/speex: remove no longer needed hook
  toolchain: add post-install hooks making the toolchain relocatable
  support/scripts: update check-host-rpath to use the shell helpers
  support/scripts/check-host-rpath: silent find command
  support/scripts/check-host-rpath: also check HOST_DIR/{bin,sbin}
  support/scripts: add check-host-leaks script + all needed helpers
  core: add check-leaks-in-{target,host,staging} targets
  support/scripts/check-host-leaks: add option to classify leaks
  docs/manual: document how to debug shell script

 Makefile                            | 128 ++++++++++++---
 docs/manual/debugging-buildroot.txt |  17 ++
 package/pkg-utils.mk                |   5 +
 package/speex/speex.mk              |   5 -
 support/scripts/check-host-leaks    | 181 +++++++++++++++++++++
 support/scripts/check-host-rpath    |  73 ++++-----
 support/scripts/fix-rpath           | 116 ++++++++++++++
 support/scripts/shell/log.sh        |  61 +++++++
 support/scripts/shell/patchelf.sh   | 178 +++++++++++++++++++++
 support/scripts/shell/readelf.sh    | 306 ++++++++++++++++++++++++++++++++++++
 support/scripts/shell/sdk.sh        |  75 +++++++++
 support/scripts/shell/source.sh     |  77 +++++++++
 support/scripts/shell/utils.sh      |  74 +++++++++
 toolchain/toolchain/toolchain.mk    |   5 +
 14 files changed, 1230 insertions(+), 71 deletions(-)
 create mode 100755 support/scripts/check-host-leaks
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh

--
2.8.0

[Buildroot] [PATCH v8 01/14] core: split variables definition related to in/out-of-tree build from O itself

[Samuel Martin]

This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from
the presence of the O variable in the command line.

Now, the condition used to set these variables is the value of O itself.

This change is a preparatory work since the O definition will need to
be moved around when we will make Buildroot run with absolute canonical
paths for both its root directory and the output location.
This will be addressed in a follow-up patch.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- new patch
---
 Makefile | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 0e4beb2..3d86c9b 100644
--- a/Makefile
+++ b/Makefile
@@ -110,10 +110,11 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
 ifneq ("$(origin O)", "command line")
 O := output
-CONFIG_DIR := $(TOPDIR)
-NEED_WRAPPER =
 else
 # other packages might also support Linux-style out of tree builds
 # with the O=<dir> syntax (E.G. BusyBox does). As make automatically
@@ -126,9 +127,16 @@ MAKEOVERRIDES =
 # Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
 # To really make O go away, we have to override it.
 override O := $(O)
-CONFIG_DIR := $(O)
 # we need to pass O= everywhere we call back into the toplevel makefile
 EXTRAMAKEARGS = O=$(O)
+endif
+
+# Set variables related to in-tree or out-of-tree build.
+ifeq ($(O),output)
+CONFIG_DIR := $(TOPDIR)
+NEED_WRAPPER =
+else
+CONFIG_DIR := $(O)
 NEED_WRAPPER = y
 endif
 
-- 
2.8.0

[Buildroot] [PATCH v8 02/14] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path

[Samuel Martin]

When $(CURDIR) or $(O) contain symlinks in their path, they can be
resolved differently, depending on each package build-system (whether it
uses the given paths or get the absolute canonical ones).

This will make easier tracking down host machine paths leaking into the
host, target or staging trees, the CURDIR and O variables are set to
their absolute canonical paths.

In order to recall the toplevel makefile with absolute canonical paths
for $(CURDIR) and $(O), we need to:
1- Move the O variable definition out of any if-block; so they are
   always available.
2- Compute the absolute canonical paths for $(CURDIR) and $(O) that will
   be passed to the sub-make. This is achieved using the 'realpath' make
   primitive. However, some care must be taken when manipulating O:
   - the out-of-tree makefile wrapper happens a trailing "/.", we need
     to strip this part away to not break the comparison driving the
     sub-make call;
   - according to [1,2], realpath returns an empty string in case of
     non-existing entry. So, to avoid passing an empty O= variable to
     sub-make, it is necessary to define the output directory and create
     it prior to call realpath on it (because on the first invocation,
     $(O) usually does not yet exists), hence the trick doing the mkdir
     right before calling realpath.
3- Update EXTRAMAKEARGS with the absobulte canonical $(O) and use it
   when call recalling the toplevel makefile with umask and paths
   correctly set.
4- Lastly, update the condition for setting the CONFIG_DIR and
   NEED_WRAPPER variables.

Notes:
* This change takes care of the makefile wrapper installed in $(O) to
  avoid unneeded make recursion.
* Now, only $(O) is strip away from MAKEOVERRIDES whatever the build is
  done in- or out-of-tree (i.o.w. without or with O set in the command
  line); wheares is the previous implementation, all variables set on
  the command line were stripped away only in the case of out-of-tree
  build.

[1] https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html
[2] http://man7.org/linux/man-pages/man3/realpath.3.html

Reported-by: Matthew Weber <matt@thewebers.ws>
Cc: Matthew Weber <matt@thewebers.ws>
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- keep @ at the beginning of the command (Yann)
- make ifneq condition easier to read/parsed (Yann)
- fix O definition before re-entering make (Reported by Matthew)
- use EXTRAMAKEARGS when re-entering make
- update the condition for CONFIG_DIR and NEED_WRAPPER

changes v6->v7:
- none

changes v5->v6:
- new patch
---
 Makefile | 86 +++++++++++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 58 insertions(+), 28 deletions(-)

diff --git a/Makefile b/Makefile
index 3d86c9b..a05b9e1 100644
--- a/Makefile
+++ b/Makefile
@@ -24,18 +24,68 @@
 # You shouldn't need to mess with anything beyond this point...
 #--------------------------------------------------------------
 
-# Trick for always running with a fixed umask
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
+ifneq ("$(origin O)", "command line")
+O := $(CURDIR)/output
+else
+# Other packages might also support Linux-style out of tree builds
+# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
+# forwards command line variable definitions those packages get very
+# confused. Fix this by telling make to not do so, only for O=..., but
+# keep all others (such as BR2_EXTERNAL, BR2_DL_DIR, etc).
+MAKEOVERRIDES := $(filter-out O=%,$(MAKEOVERRIDES))
+# Strangely enough O is still passed to submakes with MAKEOVERRIDES
+# (with make 3.81 atleast), the only thing that changes is the output
+# of the origin function (command line -> environment).
+# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
+# To really make O go away, we have to override it.
+override O := $(O)
+endif
+
+# Check if the current Buildroot execution meets all the pre-requisites.
+# If they are not met, Buildroot will actually do its job in a sub-make meeting
+# its pre-requisites, which are:
+#  1- Permissive enough umask:
+#       Wrong or too restrictive umask will prevent Buildroot and packages from
+#       creating files and directories.
+#  2- Absolute canonical CWD (i.e. $(CURDIR)):
+#       Otherwise, some packages will use CWD as-is, others will compute its
+#       absolute canonical path. This makes harder tracking and fixing host
+#       machine path leaks.
+#  3- Absolute canonical output location (i.e. $(O)):
+#       For the same reason as the one for CWD.
+
+# Current state:
+CUR_UMASK := $(shell umask)
+# Remove the trailing '/.' from $(O) as it can be added by the makefile wrapper
+# installed in the $(O) directory.
+O := $(patsubst %/.,%,$(O))
+
+# Buildroot requirements:
 UMASK = 0022
-ifneq ($(shell umask),$(UMASK))
+REAL_CURDIR := $(realpath $(CURDIR))
+# realpath needs the entry to exists, otherwise an empty string is returned.
+REAL_O := $(shell mkdir -p $(O) >/dev/null 2>&1)$(realpath $(O))
+
+# Make sure O= is passed (with its absolute canonical path) everywhere the
+# toplevel makefile is called back.
+EXTRAMAKEARGS := O=$(REAL_O)
+
+# Check Buildroot execution pre-requisites here.
+ifneq ($(CUR_UMASK):$(CURDIR):$(O),$(UMASK):$(REAL_CURDIR):$(REAL_O))
 .PHONY: _all $(MAKECMDGOALS)
 
 $(MAKECMDGOALS): _all
 	@:
 
 _all:
-	@umask $(UMASK) && $(MAKE) --no-print-directory $(MAKECMDGOALS)
+	@umask $(UMASK) && \
+		$(MAKE) -C $(REAL_CURDIR) --no-print-directory \
+			$(MAKECMDGOALS) $(EXTRAMAKEARGS)
 
-else # umask
+else # umask / $(CURDIR) / $(O)
 
 # This is our default rule, so must come first
 all:
@@ -110,30 +160,10 @@ comma := ,
 empty :=
 space := $(empty) $(empty)
 
-# Set O variable if not already done on the command line;
-# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
-# build by preventing it from being forwarded to sub-make calls.
-ifneq ("$(origin O)", "command line")
-O := output
-else
-# other packages might also support Linux-style out of tree builds
-# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
-# forwards command line variable definitions those packages get very
-# confused. Fix this by telling make to not do so
-MAKEOVERRIDES =
-# strangely enough O is still passed to submakes with MAKEOVERRIDES
-# (with make 3.81 atleast), the only thing that changes is the output
-# of the origin function (command line -> environment).
-# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
-# To really make O go away, we have to override it.
-override O := $(O)
-# we need to pass O= everywhere we call back into the toplevel makefile
-EXTRAMAKEARGS = O=$(O)
-endif
-
 # Set variables related to in-tree or out-of-tree build.
-ifeq ($(O),output)
-CONFIG_DIR := $(TOPDIR)
+# Here, both $(O) and $(CURDIR) are absolute canonical paths.
+ifeq ($(O),$(CURDIR)/output)
+CONFIG_DIR := $(CURDIR)
 NEED_WRAPPER =
 else
 CONFIG_DIR := $(O)
@@ -1017,4 +1047,4 @@ include docs/manual/manual.mk
 
 .PHONY: $(noconfig_targets)
 
-endif #umask
+endif #umask / $(CURDIR) / $(O)
-- 
2.8.0

[Buildroot] [PATCH v8 03/14] support/scripts: add fix-rpath script + a bunch of helpers

[Samuel Martin]

This commit introduces a fix-rpath shell-script able to scan a tree,
detect ELF files, check their RPATH and fix it in a proper way.

Along to the fix-rpath script, it also adds a bunch of shell helper
functions grouped into modules. This will help writing scripts handling
RPATH and other things, while allowing to share and reuse these
functions between scripts.

These helpers are namespaced within the filename of the module in which
they are gathered.

This change adds 6 modules:
- source.sh: provides simple helper to easily source another module, taking
  care of not sourcing again an already-sourced one;
- log.sh: provides logging helpers;
- utils.sh: provides simple functions to filter ELF files in a list;
- readelf.sh: provides functions retrieving ELF details from a file;
- patchelf.sh: provides function updating ELF files;
- sdk.sh: provides RPATH computation functions.

These 6 modules are used by the fix-rpath script.
Follow-up patches will make some scripts leveraging these modules.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- slightly rework log.error (return non-0 instead of exiting; exit
  decision is deferred to the caller)

changes v6->v7 (includes changes started during BR dev. days):
- log module:
  - debug env. var. renamed: DEBUG -> SHELL_DEBUG
  - trace now prints the message level
- patchelf module:
  - get rid of XRPATH/XORIGIN stuff (Arnout)
  - set_rpath now handles ELF files with no .dynamic section
  - cleanup and minor fix in sanitize_rpath
- readelf module:
  - move {filter,is}_elf* functions from utils to readelf modules
  - add list_sections and has_section functions (needed in
    sanitize_rpath)
- sdk modules:
  - get rid of XRPATH/XORIGIN stuff (Arnout)
  - minor functions doc. fixes
- source module:
  - move "Could not load module" error message in load_module (Arnout)
- utils module:
  - move {filter,is}_elf* functions from utils to readelf modules
  - add list_has function
- fix-rpath:
  - runs with "set -e" (exit on error)
  - improve help text
  - slight refactoring because of modules changes

changes v5->v6:
- fully rewritten in shell

changes v4->v5:
- add verbose support
- rename shrink_rpath -> clear_rpath
- add sanitize_rpath function

changes v3->v4:
- fix typos and license (Baruch)

changes v2->v3:
- no change
---
 support/scripts/fix-rpath         | 116 +++++++++++++++++++++++++
 support/scripts/shell/log.sh      |  61 +++++++++++++
 support/scripts/shell/patchelf.sh | 178 ++++++++++++++++++++++++++++++++++++++
 support/scripts/shell/readelf.sh  | 173 ++++++++++++++++++++++++++++++++++++
 support/scripts/shell/sdk.sh      |  68 +++++++++++++++
 support/scripts/shell/source.sh   |  77 +++++++++++++++++
 support/scripts/shell/utils.sh    |  60 +++++++++++++
 7 files changed, 733 insertions(+)
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh

diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
new file mode 100755
index 0000000..fb2bfeb
--- /dev/null
+++ b/support/scripts/fix-rpath
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+set -e
+
+usage() {
+  cat <<EOF >&2
+Usage:  ${0} TREE_KIND TREE_ROOT
+
+Description:
+
+        This script scans a tree and sanitize ELF files' RPATH found in there.
+
+        Sanitization behaves the same whatever the kindd of the processed tree, but
+        the resulting RPATH differs.
+
+        Sanitization action:
+        - remove RPATH pointing outside of the tree
+        - for RPATH pointing in the tree:
+          - if they point to standard location (/lib, /usr/lib): remove them
+          - otherwise: make them relative using \$ORIGIN
+
+        For the target tree:
+        - scan the whole tree for sanitization
+
+        For the staging tree :
+        - scan the whole tree for sanitization
+
+        For the host tree:
+        - skip the staging tree for sanitization
+        - add \$HOST_DIR/{lib,usr/lib} to RPATH (as relative pathes)
+
+Arguments:
+
+        TREE_KIND   Kind of tree to be processed.
+                    Allowed values: host, target, staging
+
+        TREE_ROOT   Path to the root of the tree to be scaned
+
+Environment:
+
+        HOST_READELF    readelf program to use for host ELF files
+                        (default: readelf)
+
+        HOST_READELF    readelf program to use for host ELF files
+                        (default: readelf)
+
+        PATCHELF        patchelf program to use
+                        (default: patchelf)
+EOF
+}
+
+source "${0%/*}/shell/source.sh"
+
+source.load_module readelf
+source.load_module patchelf
+
+: ${HOST_READELF:=readelf}
+: ${TARGET_READELF:=readelf}
+: ${PATCHELF:=patchelf}
+
+main() {
+    local tree="${1}"
+    local basedir="$(readlink -f "${2}")"
+
+    local find_args=( "${basedir}" )
+    local sanitize_extra_args=()
+    local readelf
+
+    case "${tree}" in
+        host)
+            # do not process the sysroot (only contains target binaries)
+            find_args+=( "-name" "sysroot" "-prune" "-o" )
+
+            # do not process the external toolchain installation directory to
+            # to avoid breaking it.
+            find_args+=( "-path" "*/opt/ext-toolchain" "-prune" "-o" )
+
+            # make sure RPATH will point to ${hostdir}/lib and ${hostdir}/usr/lib
+            sanitize_extra_args+=( "keep_lib_and_usr_lib" )
+
+            readelf="${HOST_READELF}"
+            ;;
+        staging|target)
+            readelf="${TARGET_READELF}"
+            ;;
+        *)
+            usage
+            exit 1
+            ;;
+    esac
+
+    find_args+=( "-type" "f" "-print" )
+
+    while read file ; do
+        READELF="${readelf}" PATCHELF="${PATCHELF}" \
+            patchelf.sanitize_rpath "${basedir}" "${file}" ${sanitize_extra_args[@]}
+    done < <(find ${find_args[@]} | readelf.filter_elf)
+}
+
+main ${@}
diff --git a/support/scripts/shell/log.sh b/support/scripts/shell/log.sh
new file mode 100644
index 0000000..efadb3f
--- /dev/null
+++ b/support/scripts/shell/log.sh
@@ -0,0 +1,61 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Logging helpers
+#
+# This module defines the following functions:
+#   log.trace
+#   log.debug
+#   log.info
+#   log.warn
+#   log.errorN
+#   log.error
+#
+# This module sets the following variables:
+#   my_name
+#
+# This module is sensitive to the following environment variables:
+#   DEBUG
+
+source.declare_module log
+
+# Debug level:
+# - 0 or empty: only show errors
+# - 1         : show errors and warnings
+# - 2         : show errors, warnings, and info
+# - 3         : show errors, warnings, info and debug
+: ${SHELL_DEBUG:=0}
+
+# Low level utility function
+log.trace()  {
+    local level="${1}" msg="${2}"
+    shift 2
+    printf "[%-5s] %s: ${msg}" "${level:0:5}" "${my_name}" "${@}"
+}
+
+# Public logging functions
+log.debug()  { :; }
+[ ${SHELL_DEBUG} -lt 3 ] || log.debug() { log.trace DEBUG "${@}" >&2; }
+log.info()   { :; }
+[ ${SHELL_DEBUG} -lt 2 ] || log.info()  { log.trace INFO "${@}" >&2; }
+log.warn()   { :; }
+[ ${SHELL_DEBUG} -lt 1 ] || log.warn()  { log.trace WARN "${@}" >&2; }
+log.errorN() { local ret="${1}" ; shift ; log.trace ERROR "${@}" ; return ${ret} ; }
+log.error()  { log.errorN 1 "${@}"; }
+
+# Program name
+my_name="${0##*/}"
+
diff --git a/support/scripts/shell/patchelf.sh b/support/scripts/shell/patchelf.sh
new file mode 100644
index 0000000..a035305
--- /dev/null
+++ b/support/scripts/shell/patchelf.sh
@@ -0,0 +1,178 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Patchelf helpers
+#
+# This module defines the following functions:
+#   patchelf.set_rpath
+#   patchelf.update_rpath
+#   patchelf.sanitize_rpath
+#
+# This module is sensitive to the following environment variables:
+#   PATCHELF
+#   READELF
+
+source.declare_module patchelf
+
+source.load_module log
+source.load_module sdk
+source.load_module utils
+source.load_module readelf
+
+: ${PATCHELF:=patchelf}
+
+# patchelf.set_xrpath file rpath...
+#
+# Set RPATH in $file.
+# Automatically join all RPATH with the correct separator.
+#
+# file  : ELF file path
+# rpath : RPATH element
+#
+# environment:
+#   PATCHELF: patchelf program path
+patchelf.set_rpath() {
+    local file="${1}"
+    shift
+    local rpath="$(sed -e 's/ +/:/g' <<<"${@}")"
+    # Sanity check: patchelf needs the ELF file to have a .dynamic section.
+    # So, check for it and behaves in a proper way:
+    # - returns immediatly if no .dynamic section, and RPATH is empty;
+    # - bail out if no .dynamic section and RPATH is not empty.
+    if ! readelf.has_section "${file}" '.dynamic' ; then
+        if test -z "${rpath}" ; then
+            return 0
+        else
+            local fmt="Trying to set a RPATH to a ELF file with no .dynamic section\n"
+            fmt="${fmt}\tfile : %s\n"
+            fmt="${fmt}\tRPATH: %s\n"
+            log.error "${fmt}" "${file}" "${rpath}" || return 1
+        fi
+    fi
+    "${PATCHELF}" --set-rpath "${rpath}" "${file}"
+}
+
+# patchelf.update_rpath basedir binary libdirs...
+#
+# Set RPATH in $binary computing them from the paths $libdirs (and $basedir).
+# Existing RPATH in $file will be overwritten if any.
+#
+# basedir : absolute path of the tree in which the $bindir and $libdirs must be
+# binary  : ELF file absolute path
+# libdirs : list of library location (absolute paths)
+#
+# environment:
+#   PATCHELF: patchelf program path
+patchelf.update_rpath() {
+    local basedir="${1}"
+    local binary="${2}"
+    shift 2
+    local libdirs=( ${@} )
+    log.debug "  basedir: %s\n" "${basedir}"
+    log.debug "      elf: %s\n" "${binary}"
+    log.debug "  libdirs: %s\n" "${libdirs[*]}"
+    log.info  "    rpath: %s\n" \
+        "$(sdk.compute_rpath "${basedir}" "${binary%/*}" ${libdirs[@]})"
+    patchelf.set_rpath "${binary}" \
+        "$(sdk.compute_rpath "${basedir}" "${binary%/*}" ${libdirs[@]})"
+}
+
+# patchelf.sanitize_rpath basedir binary [keep_lib_usr_lib]
+#
+# Scan $binary's RPATH, remove any of them pointing outside of $basedir.
+# If $keep_lib_usr_lib in not empty, the library directories $basedir/lib and
+# $basedir/usr/lib will be added to the RPATH.
+#
+# Note:
+#     Absolute paths is needed to correctly handle symlinks and or mount-bind in
+#     the $basedir path.
+#
+# basedir          : absolute path of the tree in which the $bindir and $libdirs
+#                    must be
+# binary           : ELF file absolute path
+# keep_lib_usr_lib : add to RPATH $basedir/lib and $basedir/usr/lib
+#
+# environment:
+#   PATCHELF: patchelf program path
+#   READELF : readelf program path
+patchelf.sanitize_rpath() {
+    local basedir="$(readlink -f "${1}")"
+    local binary="${2}"
+    local keep_lib_usr_lib="${3}"
+
+    readelf.is_elf_shared_object "${binary}" ||
+        readelf.is_elf_executable "${binary}" ||
+            return 0
+
+    local path abspath rpath
+    local libdirs=()
+
+    if test -n "${keep_lib_usr_lib}" ; then
+        libdirs+=( "${basedir}/lib" "${basedir}/usr/lib" )
+    fi
+
+    log.info "ELF: %s\n" "${binary}"
+
+    local rpaths="$(readelf.get_rpath "${binary}")"
+
+    for rpath in ${rpaths//:/ } ; do
+        # figure out if we should keep or discard the path; there are several
+        # cases to handled:
+        # - $path starts with "$ORIGIN":
+        #     The original build-system already took care of setting a relative
+        #     RPATH, resolve it and test if it is worthwhile to keep it;
+        # - $basedir/$path exists:
+        #     The original build-system already took care of setting an absolute
+        #     RPATH (absolute in the final rootfs), resolve it and test if it is
+        #     worthwhile to keep it;
+        # - $path start with $basedir:
+        #     The original build-system added some absolute RPATH (absolute on
+        #     the build machine). While this is wrong, it can still be fixed; so
+        #     test if it is worthwhile to keep it;
+        # - $path points somewhere else:
+        #     (can be anywhere: build trees, staging tree, host location,
+        #     non-existing location, etc.)
+        #     Just discard such a path.
+        if grep -q '^$ORIGIN/' <<<"${rpath}" ; then
+            path="${binary%/*}/${rpath#*ORIGIN/}"
+        elif test -e "${basedir}/${rpath}" ; then
+            path="${basedir}/${rpath}"
+        elif grep -q "^${basedir}/" <<<"$(readlink -f "${rpath}")" ; then
+            path="${rpath}"
+        else
+            log.debug "\tDROPPED [out-of-tree]: %s\n" "${rpath}"
+            continue
+        fi
+
+        abspath="$(readlink -f "${path}")"
+
+        # discard path pointing to default locations handled by ld-linux
+        if grep -qE "^${basedir}/(lib|usr/lib)$" <<<"${abspath}" ; then
+            log.debug \
+                "\tDROPPED [std libdirs]: %s (%s)\n" "${rpath}" "${abspath}"
+            continue
+        fi
+
+        log.debug "\tKEPT %s (%s)\n" "${rpath}" "${abspath}"
+
+        libdirs+=( "${abspath}" )
+
+    done
+
+    libdirs=( $(utils.list_reduce ${libdirs[@]}) )
+
+    patchelf.update_rpath "${basedir}" "${binary}" ${libdirs[@]}
+}
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
new file mode 100644
index 0000000..c8ad38b
--- /dev/null
+++ b/support/scripts/shell/readelf.sh
@@ -0,0 +1,173 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Readelf helpers
+#
+# This module defines the following functions:
+#   readelf._filter_elf_regexp
+#   readelf.filter_elf
+#   readelf.filter_elf_executable
+#   readelf.filter_elf_shared_object
+#   readelf.is_elf_executable
+#   readelf.is_elf_shared_object
+#   readelf.get_rpath
+#   readelf.list_sections
+#   readelf.has_section
+#
+# This module is sensitive to the following environment variables:
+#   READELF
+source.declare_module readelf
+
+# When calling readelf(1) program, the user's locale will be overriden with the
+# C locale, so we are sure we can reliably parse its output.
+: ${READELF:=readelf}
+
+# readelf._filter_elf_regexp filter_cmd file...
+#
+# Filters ELF files WRT the given regular extended expression.
+# This funtion can take one or several files, or read them from stdin.
+#
+# filter_cmd : filter command (usually based on grep)
+# file       : list of files to be filtered
+#
+# environment:
+#   READELF: readelf program path
+readelf._filter_elf_regexp() {
+    local regexp="${1}"
+    shift
+    local in file
+    test ${#} -gt 0 && in='printf "%s\n" "${@}"' || in='dd 2>/dev/null'
+    eval "${in}" |
+        while read file ; do
+            LC_ALL=C ${READELF} -h "${file}" 2>/dev/null |
+                grep -qE "${regexp}" ||
+                    continue
+            printf "%s\n" "${file}"
+        done
+}
+
+# readelf.filter_elf file...
+#
+# Filters ELF files; if $file is an ELF file, $file is printed, else it is
+# discarded.
+# This funtion can take one or several arguments, or read them from stdin.
+#
+# file : path of file to be filtered
+#
+# environment:
+#   READELF: readelf program path
+readelf.filter_elf() {
+    readelf._filter_elf_regexp "Class:\s+ELF" "${@}"
+}
+
+# readelf.filter_elf_shared_object file...
+#
+# Filters ELF files; if $file is an ELF file, $file is printed, else it is
+# discarded.
+# This funtion can take one or several arguments, or read them from stdin.
+#
+# file : path of file to be filtered
+#
+# environment:
+#   READELF: readelf program path
+readelf.filter_elf_shared_object() {
+    readelf._filter_elf_regexp "Type:\s+DYN\s\(Shared\sobject\sfile\)" "${@}"
+}
+
+# readelf.filter_elf_executable file...
+#
+# Filters ELF files; if $file is an ELF file, $file is printed, else it is
+# discarded.
+# This funtion can take one or several arguments, or read them from stdin.
+#
+# file : path of file to be filtered
+#
+# environment:
+#   READELF: readelf program path
+readelf.filter_elf_executable() {
+    readelf._filter_elf_regexp "Type:\s+EXEC\s\(Executable\sfile\)" "${@}"
+}
+
+# readelf.is_elf_shared_object file
+#
+# Returns 0 if $file is an ELF file, non-0 otherwise.
+#
+# file : path of file to be tested
+#
+# environment:
+#   READELF: readelf program path
+readelf.is_elf_shared_object() {
+    test "$(readelf.filter_elf_shared_object "${1}")" != ""
+}
+
+# readelf.is_elf_executable file
+#
+# Returns 0 if $file is an ELF file, non-0 otherwise.
+#
+# file : path of file to be tested
+#
+# environment:
+#   READELF: readelf program path
+readelf.is_elf_executable() {
+    test "$(readelf.filter_elf_executable "${1}")" != ""
+}
+
+# readelf.get_rpath file
+#
+# Return the unsplitted RPATH/RUNPATH of $file.
+#
+# To split the returned RPATH string and store them in an array, do:
+#
+#     paths=( $(readelf.get_rpath "${file}" | sed -e 's/:/ /g') )
+#
+# file : ELF file path
+#
+# environment:
+#   READELF: readelf program path
+readelf.get_rpath() {
+    local file="${1}"
+    LC_ALL=C "${READELF}" --dynamic "${file}" |
+        sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d ; s//\3/'
+}
+
+# readelf.list_sections file
+#
+# Returns the list of ELF sections in $file.
+#
+# file    : ELF file path
+#
+# environment:
+#   READELF: readelf program path
+readelf.list_sections() {
+    local file="${1}"
+    LC_ALL=C "${READELF}" --sections "${file}" |
+        sed -re '/^  \[ *[0-9]+\] (\S+).*/!d ; s//\1/' |
+        sort
+}
+
+# readelf.has_section file section
+#
+# Return 0 if $file has a section named $section
+#
+# file    : ELF file path
+# section : ELF section name
+#
+# environment:
+#   READELF: readelf program path
+readelf.has_section() {
+    local file="${1}" section_name="${2}"
+    readelf.list_sections "${file}" | grep -q "^${section_name}$"
+}
diff --git a/support/scripts/shell/sdk.sh b/support/scripts/shell/sdk.sh
new file mode 100644
index 0000000..b2f699c
--- /dev/null
+++ b/support/scripts/shell/sdk.sh
@@ -0,0 +1,68 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# SDK helpers
+#
+# This module defines the following functions:
+#   sdk.compute_relative_path
+#   sdk.compute_rpath
+
+source.declare_module sdk
+
+# sdk.compute_relative_path basedir path start
+#
+# Computes and prints the relative path between $start and $path within $basedir.
+#
+# basedir : absolute path of the tree in which the $path and $start must be
+# path    : destination absolute path
+# start   : origin absolute path
+sdk.compute_relative_path() {
+    local basedir="${1}"
+    local path="${2}"
+    local start="${3}"
+    # sanity checks: make sure $path and $start starts with $basedir
+    grep -q "^${basedir}" <<<"${path}" || return 1
+    grep -q "^${basedir}" <<<"${start}" || return 1
+    local i
+    local backward="${start#${basedir}}"
+    local relative=()
+    for i in ${backward//\// } ; do
+        # don't need to check for empty items they are already discarded
+        test "${i}" != '.' || continue
+        relative+=( ".." )
+    done
+    relative+=( ${path#${basedir}} )
+    sed -r -e 's:[ /]+:/:g' <<<"${relative[@]}"
+}
+
+# sdk.compute_rpath basedir bindir libdirs...
+#
+# Computes and prints the list of RPATH.
+#
+# basedir : absolute path of the tree in which the $bindir and $libdirs must be
+# bindir  : binary directory absolute path
+# libdirs : list of library directories (absolute paths)
+sdk.compute_rpath() {
+    local basedir="${1}"
+    local bindir="${2}"
+    shift 2
+    local libdirs=( ${@} )
+    local rpath=()
+    for libdir in ${libdirs[@]} ; do
+        rpath+=( "\$ORIGIN/$(sdk.compute_relative_path "${basedir}" "${libdir}" "${bindir}")" )
+    done
+    sed -e 's/ /:/g' <<<"${rpath[@]}"
+}
diff --git a/support/scripts/shell/source.sh b/support/scripts/shell/source.sh
new file mode 100644
index 0000000..70b7fac
--- /dev/null
+++ b/support/scripts/shell/source.sh
@@ -0,0 +1,77 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Source helpers
+#
+# This module defines the following functions:
+#   source.declare_module
+#   source.load_module
+#
+# This module is sensitive to the following environment variables:
+#   TOPDIR
+
+# Assuming the script sourcing this file is in support/scripts/
+: ${TOPDIR:=$(readlink -f "${0}" | sed -re 's:(/[^/]+){3}$::')}
+
+# source.declare_module module_name
+#
+# Declare a shell module.
+# Set the variable '_source_${module_name}'.
+# Should be called once per module, in the global scope.
+#
+# module_name : Module name (allowed char.: [_a-zA-Z0-9])
+source.declare_module() {
+    local module_name="${1}"
+    # use printf from bash to set the variable in the environment:
+    printf -v "_source_${module_name}" "%s" "${module_name}"
+}
+
+# source.load_module module_name
+#
+# Load the given shell module, making available all functions declared
+# in it, ensuring it is not reloaded if it already is.
+# Should be called in the global scope.
+# Need the TOPDIR environment variable.
+#
+# param module_name: Module name
+source.load_module() {
+    local module_name="${1}"
+    local loaded="loaded=\${_source_${module_name}}"
+    eval "${loaded}"
+    local module_file="${TOPDIR}/support/scripts/shell/${module_name}.sh"
+
+    if [ ! -f "${module_file}" ] ; then
+        cat <<EOF >&2
+error:  Could load module '${module_name}',
+        ${module_file} does not exists.
+
+        Maybe TOPDIR does not point to Buildroot's '\$(TOPDIR)'.
+
+        Or this script '${0##*/}' is most not installed in Buildroot's
+        '\$(TOPDIR)/support/scripts' directory.
+
+        You can fix this by:
+        - either installing '${0##*/}' in the support/scripts/ directory;
+        - or setting the TOPDIR variable in the '${0##*/}' script, before
+          sourcing anything.
+EOF
+        exit 1
+    fi
+
+    test -n "${loaded}" || source "${module_file}"
+}
+
+source.declare_module source
diff --git a/support/scripts/shell/utils.sh b/support/scripts/shell/utils.sh
new file mode 100644
index 0000000..9e9aaab
--- /dev/null
+++ b/support/scripts/shell/utils.sh
@@ -0,0 +1,60 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Miscellaneous helpers
+#
+# This module defines the following functions:
+#   utils.list_has
+#   utils.list_reduce
+
+source.declare_module utils
+
+# utils.list_has value list_items...
+#
+# Returns 0 if $list_items contains $value, returns 1 otherwise.
+#
+# value      : item to be checked if it is in the list
+# list_items : list of items
+utils.list_has() {
+    local key=$1
+    shift
+    for val in $@ ; do
+        if test "$val" = "$key" ; then
+            return 0
+        fi
+    done
+    return 1
+}
+
+# utils.list_reduce input_list
+#
+# Prints the $input_list list with duplicated items removed.
+# Order is preserved, WRT the first occurence of duplicated items.
+#
+# input_list : list of items
+utils.list_reduce() {
+    local -a lout # return list
+    local i
+
+    for i in ${@} ; do
+        if utils.list_has "${i}" ${lout[@]} ; then
+            continue
+        fi
+        lout+=( "${i}" )
+    done
+
+    echo ${lout[@]}
+}
-- 
2.8.0

[Buildroot] [PATCH v8 04/14] core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Samuel Martin]

This patch adds host-patchelf as a target-finalize dependency, and
introduces the HOST_SANITIZE_RPATH_HOOK hook to fix the ELF files'
RPATH from the HOST_DIR location (excluding the sysroot).

After running this hook, the RPATH from any host ELF files is relative to
the binary location itself.

Notes:
- we avoid to fix RPATH in the sysroot.
- we do not try to fix RPATH in the external toolchain installation
  location as they may have been built in a way, this is already correct;
  furthermore, fixing RPATH in those programs may result in breaking them.
- the whole host directory is processed because a number of
  host-package install programs that could be useful in places
  different from $(HOST_DIR)/{bin,sbin,usr/bin,usr/sbin}.
- the shared libraries are also processed in case they have a 'main'
  function.

As a step toward a fully relocatable SDK, this change allows to get the
toolchain relocatable, but not yet the whole SDK.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- none

changes v6->v7:
- remove the DEBUG env. var. specific handling in Makefile, now just
  place the variable in the Buildroot environment, to enable debug logs.
  This is documented in a follow-up patch of this series.

changes v5->v6:
- update for the new script version
- add debug mode support

changes v4->v5:
- add verbose support

changes v3->v4:
- add host-patchelf to PACKAGES instead of target-finalize (Baruch)
- add comment

changes v2->v3:
- move hook in Makefile (Baruch)
---
 Makefile | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/Makefile b/Makefile
index a05b9e1..e3574cf 100644
--- a/Makefile
+++ b/Makefile
@@ -638,6 +638,21 @@ endef
 TARGET_FINALIZE_HOOKS += PURGE_LOCALES
 endif
 
+# RPATH fixing
+# - The host hook sets RPATH in host ELF binaries, using relative paths to the
+#   library locations.
+# - The target hook sanitizes RPATH in target ELF binaries, removing paths
+#   pointing to package's build directories or the sysroot's libdirs.
+PACKAGES += host-patchelf
+
+define HOST_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=readelf \
+		$(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
+
 $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(PACKAGES)
-- 
2.8.0

[Buildroot] [PATCH v8 05/14] core: add {TARGET, STAGING}_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Samuel Martin]

This patch introduces the TARGET_SANITIZE_RPATH_HOOK and
STAGING_SANITIZE_RPATH_HOOK hooks fixing the ELF files' RPATH of
binaries from, respectively, the TARGET_DIR and the STAGING_DIR
locations.

After running this hook, the RPATH from any target ELF files from both
the target and the staging locations won't contain any occurence of the
sysroot or some build locations.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- none

changes v6->v7:
- remove the DEBUG env. var. specific handling in Makefile

changes v5->v6:
- update for the new script version
- add debug mode support

changes v4->v5:
- target hook now sanitizes the rpath (Baruch)
- add verbose support
- update comment

changes v3->v4:
- rebase
- add comment

changes v2->v3:
- move hook in Makefile (Baruch)
---
 Makefile | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/Makefile b/Makefile
index e3574cf..38d1076 100644
--- a/Makefile
+++ b/Makefile
@@ -653,6 +653,24 @@ endef
 
 TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
 
+# Function sanitizing target/staging ELF files' RPATH.
+# i.e. it removes paths pointing to the staging or build location from the ELF
+# files' RPATH.
+define TARGET_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=$(TARGET_READELF) \
+		$(TOPDIR)/support/scripts/fix-rpath target $(TARGET_DIR)
+endef
+
+define STAGING_SANITIZE_RPATH_HOOK
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=$(TARGET_READELF) \
+		$(TOPDIR)/support/scripts/fix-rpath staging $(STAGING_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += TARGET_SANITIZE_RPATH_HOOK \
+       STAGING_SANITIZE_RPATH_HOOK
+
 $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(PACKAGES)
-- 
2.8.0

[Buildroot] [PATCH v8 06/14] package/speex: remove no longer needed hook

[Samuel Martin]

Remove the LIBTOOL_FIXUP hook since RPATH are now sanitized in
target-finalize hooks.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- none

changes v6->v7:
- none

changes v5->v6:
- none

changes v4->v5:
- new patch (suggested by Baruch)
---
 package/speex/speex.mk | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/package/speex/speex.mk b/package/speex/speex.mk
index a34bfac..24782ec 100644
--- a/package/speex/speex.mk
+++ b/package/speex/speex.mk
@@ -24,11 +24,6 @@ ifeq ($(BR2_PACKAGE_SPEEX_ARM5E),y)
 SPEEX_CONF_OPTS += --enable-arm5e-asm
 endif
 
-define SPEEX_LIBTOOL_FIXUP
-	$(SED) 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' $(@D)/libtool
-	$(SED) 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' $(@D)/libtool
-endef
-
 SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP
 
 define SPEEX_BUILD_CMDS
-- 
2.8.0

[Buildroot] [PATCH v8 07/14] toolchain: add post-install hooks making the toolchain relocatable

[Samuel Martin]

This change adds the required dependency and post-target-install hooks
to render the toolchain relocatable.

Notes:
- Although the toolchain package is a virtual package, it does support
  hooks.
- This has to be post-target-install hooks because the toolchain package
  is handled as a "target" package.

Ultimately, this patch allows to generate relocatable toolchains, just
by doing:

  $ make menuconfig
  $ make toolchain

Then, this toolchain can be used/shared/deployed anywhere*.

*: on any Linux system with an ABI-compatible set of system libraries
   (usually including: glibc, libgcc, zlib, flex and libsigsegv).

Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <peter@korsgaard.com>

---
changes v7->v8:
- none

changes v6->v7:
- new patch
---
 toolchain/toolchain/toolchain.mk | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
index c22713b..618aa64 100644
--- a/toolchain/toolchain/toolchain.mk
+++ b/toolchain/toolchain/toolchain.mk
@@ -12,6 +12,11 @@ endif
 
 TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
 
+# dependency and hooks needed to make the toolchain relocatable
+TOOLCHAIN_DEPENDENCIES += host-patchelf
+TOOLCHAIN_POST_INSTALL_TARGET_HOOKS += HOST_SANITIZE_RPATH_HOOK \
+	STAGING_SANITIZE_RPATH_HOOK
+
 $(eval $(virtual-package))
 
 toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
-- 
2.8.0

[Buildroot] [PATCH v8 08/14] support/scripts: update check-host-rpath to use the shell helpers

[Samuel Martin]

Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- none

changes v6->v7:
- minor updates after shell modules shell changes

changes v5->v6:
- new patch
---
 support/scripts/check-host-rpath | 73 +++++++++++++++-------------------------
 support/scripts/shell/readelf.sh | 69 +++++++++++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+), 45 deletions(-)

diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath
index 48d69da..45bece8 100755
--- a/support/scripts/check-host-rpath
+++ b/support/scripts/check-host-rpath
@@ -1,12 +1,29 @@
 #!/usr/bin/env bash
 
+# Copyright (C) 2015 Yann E. MORIN <yann.morin.1998@free.fr>
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
 # This script scans $(HOST_DIR)/{bin,sbin} for all ELF files, and checks
 # they have an RPATH to $(HOST_DIR)/usr/lib if they need libraries from
 # there.
 
-# Override the user's locale so we are sure we can parse the output of
-# readelf(1) and file(1)
-export LC_ALL=C
+source "${0%/*}/shell/source.sh"
+
+source.load_module readelf
 
 main() {
     local pkg="${1}"
@@ -14,58 +31,24 @@ main() {
     local file ret
 
     # Remove duplicate and trailing '/' for proper match
-    hostdir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${hostdir}" )"
+    hostdir="$(sed -r -e 's:/+:/:g; s:/$::;' <<<"${hostdir}")"
 
     ret=0
     while read file; do
-        elf_needs_rpath "${file}" "${hostdir}" || continue
-        check_elf_has_rpath "${file}" "${hostdir}" && continue
+        READELF=readelf readelf.needs_rpath "${file}" "${hostdir}" || continue
+        READELF=readelf readelf.has_rpath "${file}" "${hostdir}" && continue
         if [ ${ret} -eq 0 ]; then
             ret=1
             printf "***\n"
-            printf "*** ERROR: package %s installs executables without proper RPATH:\n" "${pkg}"
+            printf \
+                "*** ERROR: package %s installs executables without proper RPATH:\n" \
+                "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <( find "${hostdir}"/usr/{bin,sbin} -type f -exec file {} + 2>/dev/null \
-              |sed -r -e '/^([^:]+):.*\<ELF\>.*\<executable\>.*/!d'                \
-                      -e 's//\1/'                                                  \
-            )
+    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print |
+                  readelf.filter_elf_executable)
 
     return ${ret}
 }
 
-elf_needs_rpath() {
-    local file="${1}"
-    local hostdir="${2}"
-    local lib
-
-    while read lib; do
-        [ -e "${hostdir}/usr/lib/${lib}" ] && return 0
-    done < <( readelf -d "${file}"                                         \
-              |sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d;' \
-                     -e 's//\1/;'                                          \
-            )
-
-    return 1
-}
-
-check_elf_has_rpath() {
-    local file="${1}"
-    local hostdir="${2}"
-    local rpath dir
-
-    while read rpath; do
-        for dir in ${rpath//:/ }; do
-            # Remove duplicate and trailing '/' for proper match
-            dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
-            [ "${dir}" = "${hostdir}/usr/lib" ] && return 0
-        done
-    done < <( readelf -d "${file}"                                              \
-              |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \
-                      -e 's//\3/;'                                              \
-            )
-
-    return 1
-}
-
 main "${@}"
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
index c8ad38b..0b7e8d2 100644
--- a/support/scripts/shell/readelf.sh
+++ b/support/scripts/shell/readelf.sh
@@ -24,11 +24,15 @@
 #   readelf.is_elf_executable
 #   readelf.is_elf_shared_object
 #   readelf.get_rpath
+#   readelf.get_neededs
+#   readelf.needs_rpath
+#   readelf.has_rpath
 #   readelf.list_sections
 #   readelf.has_section
 #
 # This module is sensitive to the following environment variables:
 #   READELF
+
 source.declare_module readelf
 
 # When calling readelf(1) program, the user's locale will be overriden with the
@@ -143,6 +147,71 @@ readelf.get_rpath() {
         sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d ; s//\3/'
 }
 
+# readelf.get_neededs file
+#
+# Return the list of the NEEDED libraries of $file.
+#
+# file : ELF file path
+#
+# environment:
+#   READELF: readelf program path
+readelf.get_neededs() {
+    local file="${1}"
+    LC_ALL=C "${READELF}" --dynamic "${file}" |
+        sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d ; s//\1/'
+}
+
+# readelf.needs_rpath file basedir
+#
+# Return 0 if $file needs to have RPATH set, 1 otherwise.
+#
+# file    : path of file to be tested
+# basedir : path of the tree in which $basedir/lib and $basedir/usr/lib are
+#           checked for belonging to RPATH
+#
+# environment:
+#   READELF: readelf program path
+readelf.needs_rpath() {
+    local file="${1}"
+    local basedir="${2}"
+    local lib
+
+    while read lib; do
+        [ -e "${basedir}/lib/${lib}" ] && return 0
+        [ -e "${basedir}/usr/lib/${lib}" ] && return 0
+    done < <(readelf.get_neededs "${file}")
+    return 1
+}
+
+# readelf.has_rpath file basedir
+#
+# Return 0 if $file has RPATH already set to $basedir/lib or $basedir/usr/lib,
+# 1 otherwise.
+#
+# file    : path of file to be tested
+# basedir : path of the tree in which $basedir/lib and $basedir/usr/lib are
+#           checked for belonging to RPATH
+#
+# environment:
+#   READELF: readelf program path
+readelf.has_rpath() {
+    local file="${1}"
+    local basedir="${2}"
+    local rpath dir
+
+    while read rpath; do
+        for dir in ${rpath//:/ }; do
+            # Remove duplicate and trailing '/' for proper match
+            dir="$(sed -r -e "s:/+:/:g; s:/$::" <<<"${dir}")"
+            [ "${dir}" = "${basedir}/lib" ] && return 0
+            [ "${dir}" = "${basedir}/usr/lib" ] && return 0
+            grep -q '^\$ORIGIN/' <<<"${dir}" && return 0
+        done
+    done < <(readelf.get_rpath "${file}")
+
+    return 1
+}
+
 # readelf.list_sections file
 #
 # Returns the list of ELF sections in $file.
-- 
2.8.0

[Buildroot] [PATCH v8 09/14] support/scripts/check-host-rpath: silent find command

[Samuel Martin]

Silent find command when HOST_DIR/usr/sbin does not exist.

Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- none

changes v6->v7:
- new patch
---
 support/scripts/check-host-rpath | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath
index 45bece8..3df70bd 100755
--- a/support/scripts/check-host-rpath
+++ b/support/scripts/check-host-rpath
@@ -45,7 +45,7 @@ main() {
                 "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print |
+    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print 2>/dev/null |
                   readelf.filter_elf_executable)
 
     return ${ret}
-- 
2.8.0

[Buildroot] [PATCH v8 10/14] support/scripts/check-host-rpath: also check HOST_DIR/{bin, sbin}

[Samuel Martin]

At least syslinux is installing stuff in HOST_DIR/sbin.

Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- none

changes v6->v7:
- new patch
---
 support/scripts/check-host-rpath | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath
index 3df70bd..022f01b 100755
--- a/support/scripts/check-host-rpath
+++ b/support/scripts/check-host-rpath
@@ -45,7 +45,7 @@ main() {
                 "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print 2>/dev/null |
+    done < <(find "${hostdir}"/{,usr/}{bin,sbin} -type f -print 2>/dev/null |
                   readelf.filter_elf_executable)
 
     return ${ret}
-- 
2.8.0

[Buildroot] [PATCH v8 11/14] support/scripts: add check-host-leaks script + all needed helpers

[Samuel Martin]

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- move sdk.check_host_leaks in the check-host-leaks script itself
  (Arnout)
- drop leak classification (Arnout)
- rework readelf.is_elf_{static_library,object} (Arnout)
- add utils.assert_absolute_canonical_path
- drop utils.guess_gnu_target_name
- rework/simplify the check-host-leaks script command line

changes v6->v7:
- {filer,is}_elf* functions moved from utils to readelf module
- update sdk.check_host_leaks

changes v5->v6:
- new patch
---
 support/scripts/check-host-leaks | 126 +++++++++++++++++++++++++++++++++++++++
 support/scripts/shell/readelf.sh |  77 ++++++++++++++++++++++--
 support/scripts/shell/sdk.sh     |   7 +++
 support/scripts/shell/utils.sh   |  14 +++++
 4 files changed, 218 insertions(+), 6 deletions(-)
 create mode 100755 support/scripts/check-host-leaks

diff --git a/support/scripts/check-host-leaks b/support/scripts/check-host-leaks
new file mode 100755
index 0000000..9f363af
--- /dev/null
+++ b/support/scripts/check-host-leaks
@@ -0,0 +1,126 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+usage() {
+  local ret="${1:-0}"
+  cat <<EOF
+
+Usage:  ${0##*/} [options] ROOT_DIR LEAK_PATH...
+
+Description:
+
+        This script scans a tree for host paths leaked into it.
+        Prints out leaks alongside with the kind of leak.
+
+Arguments:
+
+        ROOT_DIR        Path to the root of the tree to be scanned
+
+        LEAK_PATH       Paths to check for occurence. Several paths can be
+                        passed.
+                        A LEAK_PATH must be an absolute canonical path.
+                        In the Buildroot context, the LEAK_PATH are Buildroot's
+                        TOPDIR, BASE_DIR, HOST_DIR and STAGING_DIR.
+
+Options:
+
+        --exclude=PATH
+                Excludes files from ROOT_DIR starting with PATH. Obviously, the
+                excluded paths should point to some sub-location of ROOT_DIR.
+                Can be set more than once.
+                An excluded path must be an absolute canonical path.
+
+
+EOF
+  return ${ret}
+}
+
+source "${0%/*}/shell/source.sh"
+
+source.load_module log
+source.load_module utils
+source.load_module sdk
+
+main() {
+    local root_dir
+    local class_leaks
+    local excluded=()
+    local leak_paths=()
+    while test ${#} -gt 0 ; do
+        case "${1}" in
+            --exclude=*) excluded+=( "${1#*=}" )
+                ;;
+            --exclude)   shift ; excluded+=( "${1}" )
+                ;;
+            -h|--help)
+                usage
+                exit 0
+                ;;
+            *)  if test -z "${root_dir}" ; then root_dir="${1}"
+                else leak_paths+=( "${1}" )
+                fi
+                ;;
+        esac
+        shift
+    done
+
+    excluded=( $(utils.list_reduce ${excluded[@]}) )
+    leak_paths=( $(utils.list_reduce ${leak_paths[@]}) )
+
+    # sanity checks
+    if test -z "${root_dir}" ; then
+        log.error "Not enough argument: no ROOT_DIR given.\n" ||
+            usage ${?} || exit ${?}
+    fi
+    if test ${#leak_paths[@]} -lt 1 ; then
+        log.error "Not enough argument. No LEAK_PATH given.\n" ||
+            usage ${?} || exit ${?}
+    fi
+
+    local path has_non_abspath
+    for path in ${leak_paths[@]} ; do
+        utils.assert_absolute_canonical_path "${path}" || has_non_abspath=y
+    done
+    if test -n "${has_non_abspath}" ; then
+        log.error "Some LEAK_PATH are not absolute canonical paths.\n" ||
+            usage ${?} || exit ${?}
+    fi
+    has_non_abspath=
+    for path in ${excluded[@]} ; do
+        utils.assert_absolute_canonical_path "${path}" || has_non_abspath=y
+    done
+    if test -n "${has_non_abspath}" ; then
+        log.error "Some excluded path are not absolute canonical paths.\n" ||
+            usage ${?} || exit ${?}
+    fi
+
+    local re_leaks="$(sed -re 's/^/(/ ; s/$/)/ ; s/ +/|/g' <<<"${leak_paths[*]}")"
+    local re_excl="$(sed -re 's/ +/|/g' <<<"${excluded[*]}")"
+    if test -n "${re_excl}" ; then
+      re_excl="\\:^(${re_excl}):d"
+    fi
+    pushd "${root_dir}" >/dev/null
+    local f
+    grep -raEl "${re_leaks}" . |
+        sed -re "${re_excl} ; s:^\.:${root_dir}:" |
+        while read f ; do
+            printf "%s\n" "${f}"
+        done | sort
+    popd >/dev/null
+}
+main "${@}"
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
index 0b7e8d2..87599bb 100644
--- a/support/scripts/shell/readelf.sh
+++ b/support/scripts/shell/readelf.sh
@@ -17,18 +17,22 @@
 # Readelf helpers
 #
 # This module defines the following functions:
+#   readelf._match_elf_regexp
 #   readelf._filter_elf_regexp
 #   readelf.filter_elf
 #   readelf.filter_elf_executable
 #   readelf.filter_elf_shared_object
 #   readelf.is_elf_executable
 #   readelf.is_elf_shared_object
+#   readelf.is_elf_static_library
+#   readelf.is_elf_object
 #   readelf.get_rpath
 #   readelf.get_neededs
 #   readelf.needs_rpath
 #   readelf.has_rpath
 #   readelf.list_sections
 #   readelf.has_section
+#   readelf.string_section
 #
 # This module is sensitive to the following environment variables:
 #   READELF
@@ -39,12 +43,28 @@ source.declare_module readelf
 # C locale, so we are sure we can reliably parse its output.
 : ${READELF:=readelf}
 
-# readelf._filter_elf_regexp filter_cmd file...
+# readelf._match_elf_regexp regexp file
 #
-# Filters ELF files WRT the given regular extended expression.
+# Returns 0 if the ELF file matches the ELF type given in extended regular
+# expression, non-0 otherwise.
+#
+# regexp     : extended regular expression
+# file       : list of files to be filtered
+#
+# environment:
+#   READELF: readelf program path
+readelf._match_elf_regexp() {
+    log._trace_func
+    local regexp="${1}" file="${2}"
+    LC_ALL=C ${READELF} -h "${file}" 2>/dev/null | grep -qE "${regexp}"
+}
+
+# readelf._filter_elf_regexp regexp file...
+#
+# Filters ELF files WRT the given extended regular expression.
 # This funtion can take one or several files, or read them from stdin.
 #
-# filter_cmd : filter command (usually based on grep)
+# regexp     : extended regular expression
 # file       : list of files to be filtered
 #
 # environment:
@@ -56,9 +76,7 @@ readelf._filter_elf_regexp() {
     test ${#} -gt 0 && in='printf "%s\n" "${@}"' || in='dd 2>/dev/null'
     eval "${in}" |
         while read file ; do
-            LC_ALL=C ${READELF} -h "${file}" 2>/dev/null |
-                grep -qE "${regexp}" ||
-                    continue
+            readelf._match_elf_regexp "${regexp}" "${file}" || continue
             printf "%s\n" "${file}"
         done
 }
@@ -129,6 +147,39 @@ readelf.is_elf_executable() {
     test "$(readelf.filter_elf_executable "${1}")" != ""
 }
 
+# readelf.is_elf file
+#
+# Returns 0 if $file is an ELF file, non-0 otherwise.
+#
+# file : path of file to be tested
+#
+# environment:
+#   READELF: readelf program path
+readelf.is_elf() {
+    test "$(readelf.filter_elf "${1}")" != ""
+}
+
+# readelf.is_elf_static_library file
+#
+# Return 0 if $file is a Linux static libraries, i.e. an ar-archive
+# containing *.o files.
+#
+# file : path of file to be tested
+readelf.is_elf_static_library() {
+    readelf._match_elf_regexp "Type:\s+REL\s\(Relocatable\sfile\)" "${@}" &&
+        readelf._match_elf_regexp "^File:\s+\S+\)$" "${@}"
+}
+
+# readelf.is_elf_object file
+#
+# Return 0 if $file is an ELF object file, i.e. a *.o (or *.ko) file.
+#
+# file : path of file to be tested
+readelf.is_elf_object() {
+    readelf._match_elf_regexp "Type:\s+REL\s\(Relocatable\sfile\)" "${@}" &&
+        ! readelf._match_elf_regexp "^File:\s+\S+\)$" "${@}"
+}
+
 # readelf.get_rpath file
 #
 # Return the unsplitted RPATH/RUNPATH of $file.
@@ -240,3 +291,17 @@ readelf.has_section() {
     local file="${1}" section_name="${2}"
     readelf.list_sections "${file}" | grep -q "^${section_name}$"
 }
+
+# readelf.string_section file section
+#
+# Return the given $section of $file.
+#
+# file    : ELF file path
+# section : ELF section name
+#
+# environment:
+#   READELF: readelf program path
+readelf.string_section() {
+    local file="${1}" section="${2}"
+    LC_ALL=C "${READELF}" --string-dump "${section}" "${file}" 2>/dev/null
+}
diff --git a/support/scripts/shell/sdk.sh b/support/scripts/shell/sdk.sh
index b2f699c..6cb3bbe 100644
--- a/support/scripts/shell/sdk.sh
+++ b/support/scripts/shell/sdk.sh
@@ -19,9 +19,16 @@
 # This module defines the following functions:
 #   sdk.compute_relative_path
 #   sdk.compute_rpath
+#   sdk.check_host_leaks
+#
+# This module is sensitive to the following environment variables:
+#   READELF
 
 source.declare_module sdk
 
+source.load_module utils
+source.load_module readelf
+
 # sdk.compute_relative_path basedir path start
 #
 # Computes and prints the relative path between $start and $path within $basedir.
diff --git a/support/scripts/shell/utils.sh b/support/scripts/shell/utils.sh
index 9e9aaab..eac3158 100644
--- a/support/scripts/shell/utils.sh
+++ b/support/scripts/shell/utils.sh
@@ -19,6 +19,7 @@
 # This module defines the following functions:
 #   utils.list_has
 #   utils.list_reduce
+#   utils.assert_absolute_canonical_path
 
 source.declare_module utils
 
@@ -58,3 +59,16 @@ utils.list_reduce() {
 
     echo ${lout[@]}
 }
+
+# utils.assert_absolute_canonical_path path
+#
+# Returns 0 if 'path' is the absolute canonical path, returns non-0
+# otherwise.
+#
+# If the test failed, an error message will be issued to stderr.
+#
+# path : path to be tested
+utils.assert_absolute_canonical_path() {
+    test "$(readlink -f "${1}")" = "${1}" ||
+        log.error "%s is not the absolute canonical path.\n" "${1}" >&2
+}
-- 
2.8.0

[Buildroot] [PATCH v8 12/14] core: add check-leaks-in-{target, host, staging} targets

[Samuel Martin]

This new targets allow identifying what need to be fixed to fulfill the
ultimate goal of removing any trace of the host in the target filesystem
or the SDK.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- update the check-host-leak script call

changes v6->v7:
- none

changes v5->v6:
- new patch
---
 Makefile             | 15 +++++++++++++++
 package/pkg-utils.mk |  5 +++++
 2 files changed, 20 insertions(+)

diff --git a/Makefile b/Makefile
index 38d1076..8309eae 100644
--- a/Makefile
+++ b/Makefile
@@ -638,6 +638,21 @@ endef
 TARGET_FINALIZE_HOOKS += PURGE_LOCALES
 endif
 
+# Checks for build machine leaks into target filesystem, SDK
+check-leaks-in-target:
+	@$(call MESSAGE,Checking leaks in the target)
+	$(Q)$(call check-for-build-machine-leaks-in,$(TARGET_DIR))
+
+check-leaks-in-host:
+	@$(call MESSAGE,Checking leaks in the host)
+	$(Q)$(call check-for-build-machine-leaks-in,$(HOST_DIR),--exclude=$(STAGING_DIR))
+
+check-leaks-in-staging:
+	@$(call MESSAGE,Checking leaks in the staging)
+	$(Q)$(call check-for-build-machine-leaks-in,$(STAGING_DIR))
+
+check-leaks:	check-leaks-in-target check-leaks-in-host check-leaks-in-staging
+
 # RPATH fixing
 # - The host hook sets RPATH in host ELF binaries, using relative paths to the
 #   library locations.
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index f88313a..8b80028 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -150,3 +150,8 @@ define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET}
 	mkdir -p $(LICENSE_FILES_DIR_$(4))/$(1)/$(dir $(2)) && \
 	cp $(3) $(LICENSE_FILES_DIR_$(4))/$(1)/$(2)
 endef
+
+define check-for-build-machine-leaks-in
+	$(TOPDIR)/support/scripts/check-host-leaks $(1) \
+		$(TOPDIR) $(BASE_DIR) $(HOST_DIR) $(STAGING_DIR) $(2)
+endef
-- 
2.8.0

[Buildroot] [PATCH v8 13/14] support/scripts/check-host-leaks: add option to classify leaks

[Samuel Martin]

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- new patch. Move this feature in its own change-set as it seems a bit
  controversial.
- make leaks classification optional
---
 support/scripts/check-host-leaks | 59 ++++++++++++++++++++++++++++++++++++++--
 support/scripts/shell/readelf.sh |  1 -
 2 files changed, 57 insertions(+), 3 deletions(-)

diff --git a/support/scripts/check-host-leaks b/support/scripts/check-host-leaks
index 9f363af..bd92493 100755
--- a/support/scripts/check-host-leaks
+++ b/support/scripts/check-host-leaks
@@ -45,6 +45,12 @@ Options:
                 Can be set more than once.
                 An excluded path must be an absolute canonical path.
 
+        --classify-leaks
+                Enables leak classification. Categories are defined according to
+                the matched files' type or ELF kinds.
+                For ELF files, the sections containing the leaks will displayed.
+                Enabling this option substantially increases the execution time
+                of the script.
 
 EOF
   return ${ret}
@@ -56,9 +62,51 @@ source.load_module log
 source.load_module utils
 source.load_module sdk
 
+classify_leak() {
+    local f="${1}" regexp="${2}"
+    local leak
+    if test -h "${f}" ; then leak="symlink"
+    elif readelf.is_elf "${f}" ; then
+        if readelf.is_elf_executable "${f}" ; then leak="ELF/exe"
+        elif readelf.is_elf_shared_object "${f}" ; then leak="ELF/*.so"
+        elif readelf.is_elf_static_library "${f}" ; then leak="ELF/*.a"
+        elif readelf.is_elf_object "${f}" ; then
+            case "${f}" in
+                *.ko) leak="ELF/*.ko" ;;
+                *) leak="ELF/*.o" ;;
+            esac
+        else leak="ELF/?"
+        fi
+        local section
+        local sections=()
+        for section in $(readelf.list_sections "${f}") ; do
+            if readelf.string_section "${f}" "${section}" |
+                    grep -qaE "${regexp}" ; then
+                if ! utils.list_has ${section} ${sections[@]} ; then
+                    sections+=( "${section}" )
+                fi
+            fi
+        done
+        leak="${leak} [${sections[*]}]"
+    else
+        case "${f}" in
+            *-config) leak="*-config script" ;;
+            *.la) leak="*.la" ;;
+            *.pc) leak="*.pc" ;;
+            *.py) leak="*.py" ;;
+            *.pyc|*.pyo) leak="*.py[co]" ;;
+        esac
+    fi
+    if test -z "${leak}" ; then
+        leak="? [$(file -z "${f}" | sed -e 's/.*: //')]"
+    fi
+    printf "${leak}"
+}
+
+
 main() {
     local root_dir
-    local class_leaks
+    local classify_leaks
     local excluded=()
     local leak_paths=()
     while test ${#} -gt 0 ; do
@@ -67,6 +115,8 @@ main() {
                 ;;
             --exclude)   shift ; excluded+=( "${1}" )
                 ;;
+            --classify-leaks) classify_leaks=y
+                ;;
             -h|--help)
                 usage
                 exit 0
@@ -119,7 +169,12 @@ main() {
     grep -raEl "${re_leaks}" . |
         sed -re "${re_excl} ; s:^\.:${root_dir}:" |
         while read f ; do
-            printf "%s\n" "${f}"
+            if test -n "${classify_leaks}" ; then
+                local leak="$(classify_leak "${f}" "${regexp}")"
+                printf "%-70s : %-120s\n" "${leak}" "${f}"
+            else
+                printf "%s\n" "${f}"
+            fi
         done | sort
     popd >/dev/null
 }
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
index 87599bb..acb522f 100644
--- a/support/scripts/shell/readelf.sh
+++ b/support/scripts/shell/readelf.sh
@@ -54,7 +54,6 @@ source.declare_module readelf
 # environment:
 #   READELF: readelf program path
 readelf._match_elf_regexp() {
-    log._trace_func
     local regexp="${1}" file="${2}"
     LC_ALL=C ${READELF} -h "${file}" 2>/dev/null | grep -qE "${regexp}"
 }
-- 
2.8.0

[Buildroot] [PATCH v8 14/14] docs/manual: document how to debug shell script

[Samuel Martin]

The method describes here is only true for scripts using the shell
modules from support/scripts/shell/*.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v7->v8:
- put instrumentation in a subsection (Arnout)
- minor rewording

changes v6->v7:
- new patch
---
 docs/manual/debugging-buildroot.txt | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/docs/manual/debugging-buildroot.txt b/docs/manual/debugging-buildroot.txt
index f575fc4..b809c2d 100644
--- a/docs/manual/debugging-buildroot.txt
+++ b/docs/manual/debugging-buildroot.txt
@@ -5,6 +5,8 @@
 
 == Debugging Buildroot
 
+=== Running scripts before and after each build step
+
 It is possible to instrument the steps +Buildroot+ does when building
 packages. Define the variable +BR2_INSTRUMENTATION_SCRIPTS+ to contain
 the path of one or more scripts (or other executables), in a
@@ -41,3 +43,18 @@ The script has access to the following variables:
   - +BINARIES_DIR+: the place where all binary files (aka images) are
     stored
   - +BASE_DIR+: the base output directory
+
+=== Debugging shell scripts in Buildroot
+
+For shell scripts using the _shell modules_ (available in
++support/scripts/shell/+), an environment variable is available to trace
+the scripts:
+
+  - +SHELL_DEBUG+ adjustes the debug level, from 0 to 3 (only errors to
+    all debug messages)
+
+To enable these debug outputs:
+
+----
+SHELL_DEBUG=3 make
+----
-- 
2.8.0

[Buildroot] [PATCH v8 04/14] core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Thomas Petazzoni]

Hello,

On Sun, 17 Apr 2016 23:38:21 +0200, Samuel Martin wrote:

> +# RPATH fixing
> +# - The host hook sets RPATH in host ELF binaries, using relative paths to the
> +#   library locations.
> +# - The target hook sanitizes RPATH in target ELF binaries, removing paths
> +#   pointing to package's build directories or the sysroot's libdirs.

There is nothing in this patch that adds a target hook.

> +PACKAGES += host-patchelf
> +
> +define HOST_SANITIZE_RPATH_HOOK
> +	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
> +	READELF=readelf \
> +		$(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
> +endef
> +
> +TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK

I find it somewhat odd that this gets done as a "target finalize" hook
since it really has nothing to do with finalizing the target. It really
just needs to be done "at the end of the build".

We could add that as a new target, but it would complexify the
dependency chain, so maybe adding to TARGET_FINALIZE_HOOKS is the
easiest solution after all.

One tricky aspect though is that not all host packages are guaranteed
to be built by the time TARGET_FINALIZE_HOOKS is called. Indeed,
TARGET_FINALIZE_HOOKS is guaranteed to be called after all *target*
packages have been built. But if a host package is needed just to
build a filesystem image for example, it is not guaranteed that it will
be built before TARGET_FINALIZE_HOOKS is called.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 04/14] core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Samuel Martin]

On Mon, Apr 18, 2016 at 1:33 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Sun, 17 Apr 2016 23:38:21 +0200, Samuel Martin wrote:
>
>> +# RPATH fixing
>> +# - The host hook sets RPATH in host ELF binaries, using relative paths to the
>> +#   library locations.
>> +# - The target hook sanitizes RPATH in target ELF binaries, removing paths
>> +#   pointing to package's build directories or the sysroot's libdirs.
>
> There is nothing in this patch that adds a target hook.
Oops, my bad! :-s
This is for the next patch in the series.

>
>> +PACKAGES += host-patchelf
>> +
>> +define HOST_SANITIZE_RPATH_HOOK
>> +     PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
>> +     READELF=readelf \
>> +             $(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
>> +endef
>> +
>> +TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
>
> I find it somewhat odd that this gets done as a "target finalize" hook
> since it really has nothing to do with finalizing the target. It really
> just needs to be done "at the end of the build".
>
> We could add that as a new target, but it would complexify the
> dependency chain, so maybe adding to TARGET_FINALIZE_HOOKS is the
> easiest solution after all.
>
> One tricky aspect though is that not all host packages are guaranteed
> to be built by the time TARGET_FINALIZE_HOOKS is called. Indeed,
> TARGET_FINALIZE_HOOKS is guaranteed to be called after all *target*
> packages have been built. But if a host package is needed just to
> build a filesystem image for example, it is not guaranteed that it will
> be built before TARGET_FINALIZE_HOOKS is called.

Good point!
I'll check and see if I can come up with something.

Regards,


-- 
Samuel

[Buildroot] [PATCH v8 07/14] toolchain: add post-install hooks making the toolchain relocatable

[Thomas Petazzoni]

Hello,

On Sun, 17 Apr 2016 23:38:24 +0200, Samuel Martin wrote:
> This change adds the required dependency and post-target-install hooks
> to render the toolchain relocatable.
> 
> Notes:
> - Although the toolchain package is a virtual package, it does support
>   hooks.
> - This has to be post-target-install hooks because the toolchain package
>   is handled as a "target" package.
> 
> Ultimately, this patch allows to generate relocatable toolchains, just
> by doing:
> 
>   $ make menuconfig
>   $ make toolchain
> 
> Then, this toolchain can be used/shared/deployed anywhere*.
> 
> *: on any Linux system with an ABI-compatible set of system libraries
>    (usually including: glibc, libgcc, zlib, flex and libsigsegv).
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <peter@korsgaard.com>

Why is this needed? You already do such steps in the
TARGET_FINALIZE_HOOKS ?

Yes, it means after just a "make toolchain", you don't have a
relocatable toolchain. But Buildroot never guaranteed that a partial
build will yield fully  usable results.

So I would prefer to not have this patch, at least in the initial
version of the series.

Again, this stuff is complicated. Please keep things as minimal as
possible in a first step.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 06/14] package/speex: remove no longer needed hook

[Thomas Petazzoni]

Hello,

On Sun, 17 Apr 2016 23:38:23 +0200, Samuel Martin wrote:

> -define SPEEX_LIBTOOL_FIXUP
> -	$(SED) 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' $(@D)/libtool
> -	$(SED) 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' $(@D)/libtool
> -endef
> -
>  SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP

So presumably this line is also not useful anymore?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 09/14] support/scripts/check-host-rpath: silent find command

[Thomas Petazzoni]

Hello,

On Sun, 17 Apr 2016 23:38:26 +0200, Samuel Martin wrote:

> -    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print |
> +    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print 2>/dev/null |
>                    readelf.filter_elf_executable)

This patch is useless. Your patch 08/14 is doing:

-    done < <( find "${hostdir}"/usr/{bin,sbin} -type f -exec file {} + 2>/dev/null \
-              |sed -r -e '/^([^:]+):.*\<ELF\>.*\<executable\>.*/!d'                \
-                      -e 's//\1/'                                                  \
-            )
+    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print |
+                  readelf.filter_elf_executable)

So essentially the 2>/dev/null was already here, and you remove it. And
then you add it again in your patch 09/14. Simply squash both patches.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 08/14] support/scripts: update check-host-rpath to use the shell helpers

[Thomas Petazzoni]

Hello,

So the commit title says that you update check-host-rpath to use the
shell helpers...

On Sun, 17 Apr 2016 23:38:25 +0200, Samuel Martin wrote:

>  support/scripts/check-host-rpath | 73 +++++++++++++++-------------------------
>  support/scripts/shell/readelf.sh | 69 +++++++++++++++++++++++++++++++++++++

but in practice, you also add a lot of new functions to the shell
helpers.

Either this is intended and it should be mentioned in the commit log,
or it is not intended.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 10/14] support/scripts/check-host-rpath: also check HOST_DIR/{bin, sbin}

[Thomas Petazzoni]

Hello,

On Sun, 17 Apr 2016 23:38:27 +0200, Samuel Martin wrote:
> At least syslinux is installing stuff in HOST_DIR/sbin.
> 
> Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> 
> ---
> changes v7->v8:
> - none
> 
> changes v6->v7:
> - new patch
> ---
>  support/scripts/check-host-rpath | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath
> index 3df70bd..022f01b 100755
> --- a/support/scripts/check-host-rpath
> +++ b/support/scripts/check-host-rpath
> @@ -45,7 +45,7 @@ main() {
>                  "${pkg}"
>          fi
>          printf "***   %s\n" "${file}"
> -    done < <(find "${hostdir}"/usr/{bin,sbin} -type f -print 2>/dev/null |
> +    done < <(find "${hostdir}"/{,usr/}{bin,sbin} -type f -print 2>/dev/null |
>                    readelf.filter_elf_executable)

Please put this patch as PATCH 01 in your series, so that we can apply
it right now as a preparation fix. This way, you reduce the size of
your patch series, and things move forward.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 04/14] core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Arnout Vandecappelle]

On 04/18/16 13:33, Thomas Petazzoni wrote:
> Hello,
>
> On Sun, 17 Apr 2016 23:38:21 +0200, Samuel Martin wrote:
>
>> +# RPATH fixing
>> +# - The host hook sets RPATH in host ELF binaries, using relative paths to the
>> +#   library locations.
>> +# - The target hook sanitizes RPATH in target ELF binaries, removing paths
>> +#   pointing to package's build directories or the sysroot's libdirs.
>
> There is nothing in this patch that adds a target hook.
>
>> +PACKAGES += host-patchelf
>> +
>> +define HOST_SANITIZE_RPATH_HOOK
>> +	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
>> +	READELF=readelf \
>> +		$(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
>> +endef
>> +
>> +TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
>
> I find it somewhat odd that this gets done as a "target finalize" hook
> since it really has nothing to do with finalizing the target. It really
> just needs to be done "at the end of the build".
>
> We could add that as a new target, but it would complexify the
> dependency chain, so maybe adding to TARGET_FINALIZE_HOOKS is the
> easiest solution after all.
>
> One tricky aspect though is that not all host packages are guaranteed
> to be built by the time TARGET_FINALIZE_HOOKS is called. Indeed,
> TARGET_FINALIZE_HOOKS is guaranteed to be called after all *target*
> packages have been built. But if a host package is needed just to
> build a filesystem image for example, it is not guaranteed that it will
> be built before TARGET_FINALIZE_HOOKS is called.

  Actually, it is run after all explicitly selected packages have been built. So 
this is yet another reason to also add Kconfig options for host packages.

  Regards,
  Arnout

>
> Best regards,
>
> Thomas
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v8 04/14] core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Thomas Petazzoni]

Hello,

On Tue, 19 Apr 2016 01:38:19 +0200, Arnout Vandecappelle wrote:

> > One tricky aspect though is that not all host packages are guaranteed
> > to be built by the time TARGET_FINALIZE_HOOKS is called. Indeed,
> > TARGET_FINALIZE_HOOKS is guaranteed to be called after all *target*
> > packages have been built. But if a host package is needed just to
> > build a filesystem image for example, it is not guaranteed that it will
> > be built before TARGET_FINALIZE_HOOKS is called.
> 
>   Actually, it is run after all explicitly selected packages have been built. So 
> this is yet another reason to also add Kconfig options for host packages.

Well, while I do like the fact that you try to find more arguments to
add Kconfig options for host packages, I am not sure this particular
one is a good reason. Indeed, as the name TARGET_FINALIZE_HOOKS
suggests, it finalizes the _target_, so there is no point in
guaranteeing that all host packages have been built *before* the hooks
in TARGET_FINALIZE_HOOKS are executed.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com