* [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing
@ 2016-04-22 20:50 Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself Samuel Martin
` (11 more replies)
0 siblings, 12 replies; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
Hi all,
Here is yet another round of the series aiming improving the relocatability
of the SDK built by Buildroot.
Still slowly reducing the number of patch in the series, as they got merged...
For this round, just fixes from comments, nothing really noticeable, except
maybe the move of the HOST_DIR RPATH sanitation.
For further details, check the per-commit changelog in each commit log.
Patches 1 and 2:
Fix the O variable definition.
Patches 3 to 5:
Add the fix-rpath script, then run it on the Buildroot host, target and
staging trees.
Patch 6:
Clean-up speex package WRT its RPATH hook.
Patch 7:
Update and fix the check-host-rpath to re-use the shell helpers.
Patches 8 to 10:
Add means to track build machine leaks into the host, target and staging
trees. These patches will allow to identify what remains to be fixed WRT
build machine leaks.
Patches 11:
Update documentation about how to enable/adjust log level on script using
the shell modules introduced in patch 3/11.
Regards,
Samuel
Samuel Martin (11):
core: split variables definition related to in/out-of-tree build from
O itself
core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
path
support/scripts: add fix-rpath script + a bunch of helpers
core: sanitize HOST_DIR at the very end of the build
core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
TARGET_FINALIZE_HOOKS
package/speex: remove no longer needed hook
support/scripts: update check-host-rpath to use the shell helpers
support/scripts: add check-host-leaks script + all needed helpers
core: add check-leaks-in-{target,host,staging} targets
support/scripts/check-host-leaks: add option to classify leaks
docs/manual: document how to debug shell script
Makefile | 126 ++++++++++++---
docs/manual/debugging-buildroot.txt | 17 ++
package/pkg-utils.mk | 5 +
package/speex/speex.mk | 7 -
support/scripts/check-host-leaks | 181 +++++++++++++++++++++
support/scripts/check-host-rpath | 73 ++++-----
support/scripts/fix-rpath | 116 ++++++++++++++
support/scripts/shell/log.sh | 61 +++++++
support/scripts/shell/patchelf.sh | 178 +++++++++++++++++++++
support/scripts/shell/readelf.sh | 306 ++++++++++++++++++++++++++++++++++++
support/scripts/shell/sdk.sh | 75 +++++++++
support/scripts/shell/source.sh | 77 +++++++++
support/scripts/shell/utils.sh | 74 +++++++++
13 files changed, 1223 insertions(+), 73 deletions(-)
create mode 100755 support/scripts/check-host-leaks
create mode 100755 support/scripts/fix-rpath
create mode 100644 support/scripts/shell/log.sh
create mode 100644 support/scripts/shell/patchelf.sh
create mode 100644 support/scripts/shell/readelf.sh
create mode 100644 support/scripts/shell/sdk.sh
create mode 100644 support/scripts/shell/source.sh
create mode 100644 support/scripts/shell/utils.sh
--
2.8.0
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-06-26 22:27 ` Arnout Vandecappelle
2016-07-05 17:02 ` Yann E. MORIN
2016-04-22 20:50 ` [Buildroot] [PATCH v9 02/11] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path Samuel Martin
` (10 subsequent siblings)
11 siblings, 2 replies; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from
the presence of the O variable in the command line.
Now, the condition used to set these variables is the value of O itself.
This change is a preparatory work since the O definition will need to
be moved around when we will make Buildroot run with absolute canonical
paths for both its root directory and the output location.
This will be addressed in a follow-up patch.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- none
changes v7->v8:
- new patch
---
Makefile | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile
index 0e4beb2..3d86c9b 100644
--- a/Makefile
+++ b/Makefile
@@ -110,10 +110,11 @@ comma := ,
empty :=
space := $(empty) $(empty)
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
ifneq ("$(origin O)", "command line")
O := output
-CONFIG_DIR := $(TOPDIR)
-NEED_WRAPPER =
else
# other packages might also support Linux-style out of tree builds
# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
@@ -126,9 +127,16 @@ MAKEOVERRIDES =
# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
# To really make O go away, we have to override it.
override O := $(O)
-CONFIG_DIR := $(O)
# we need to pass O= everywhere we call back into the toplevel makefile
EXTRAMAKEARGS = O=$(O)
+endif
+
+# Set variables related to in-tree or out-of-tree build.
+ifeq ($(O),output)
+CONFIG_DIR := $(TOPDIR)
+NEED_WRAPPER =
+else
+CONFIG_DIR := $(O)
NEED_WRAPPER = y
endif
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 02/11] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-06-26 23:08 ` Arnout Vandecappelle
2016-04-22 20:50 ` [Buildroot] [PATCH v9 03/11] support/scripts: add fix-rpath script + a bunch of helpers Samuel Martin
` (9 subsequent siblings)
11 siblings, 1 reply; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
When $(CURDIR) or $(O) contain symlinks in their path, they can be
resolved differently, depending on each package build-system (whether it
uses the given paths or get the absolute canonical ones).
This will make easier tracking down host machine paths leaking into the
host, target or staging trees, the CURDIR and O variables are set to
their absolute canonical paths.
In order to recall the toplevel makefile with absolute canonical paths
for $(CURDIR) and $(O), we need to:
1- Move the O variable definition out of any if-block; so they are
always available.
2- Compute the absolute canonical paths for $(CURDIR) and $(O) that will
be passed to the sub-make. This is achieved using the 'realpath' make
primitive. However, some care must be taken when manipulating O:
- the out-of-tree makefile wrapper happens a trailing "/.", we need
to strip this part away to not break the comparison driving the
sub-make call;
- according to [1,2], realpath returns an empty string in case of
non-existing entry. So, to avoid passing an empty O= variable to
sub-make, it is necessary to define the output directory and create
it prior to call realpath on it (because on the first invocation,
$(O) usually does not yet exists), hence the trick doing the mkdir
right before calling realpath.
3- Update EXTRAMAKEARGS with the absobulte canonical $(O) and use it
when call recalling the toplevel makefile with umask and paths
correctly set.
4- Lastly, update the condition for setting the CONFIG_DIR and
NEED_WRAPPER variables.
Notes:
* This change takes care of the makefile wrapper installed in $(O) to
avoid unneeded make recursion.
* Now, only $(O) is strip away from MAKEOVERRIDES whatever the build is
done in- or out-of-tree (i.o.w. without or with O set in the command
line); wheares is the previous implementation, all variables set on
the command line were stripped away only in the case of out-of-tree
build.
[1] https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html
[2] http://man7.org/linux/man-pages/man3/realpath.3.html
Reported-by: Matthew Weber <matt@thewebers.ws>
Cc: Matthew Weber <matt@thewebers.ws>
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- none
changes v7->v8:
- keep @ at the beginning of the command (Yann)
- make ifneq condition easier to read/parsed (Yann)
- fix O definition before re-entering make (Reported by Matthew)
- use EXTRAMAKEARGS when re-entering make
- update the condition for CONFIG_DIR and NEED_WRAPPER
changes v6->v7:
- none
changes v5->v6:
- new patch
---
Makefile | 86 +++++++++++++++++++++++++++++++++++++++++++---------------------
1 file changed, 58 insertions(+), 28 deletions(-)
diff --git a/Makefile b/Makefile
index 3d86c9b..a05b9e1 100644
--- a/Makefile
+++ b/Makefile
@@ -24,18 +24,68 @@
# You shouldn't need to mess with anything beyond this point...
#--------------------------------------------------------------
-# Trick for always running with a fixed umask
+# Set O variable if not already done on the command line;
+# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
+# build by preventing it from being forwarded to sub-make calls.
+ifneq ("$(origin O)", "command line")
+O := $(CURDIR)/output
+else
+# Other packages might also support Linux-style out of tree builds
+# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
+# forwards command line variable definitions those packages get very
+# confused. Fix this by telling make to not do so, only for O=..., but
+# keep all others (such as BR2_EXTERNAL, BR2_DL_DIR, etc).
+MAKEOVERRIDES := $(filter-out O=%,$(MAKEOVERRIDES))
+# Strangely enough O is still passed to submakes with MAKEOVERRIDES
+# (with make 3.81 atleast), the only thing that changes is the output
+# of the origin function (command line -> environment).
+# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
+# To really make O go away, we have to override it.
+override O := $(O)
+endif
+
+# Check if the current Buildroot execution meets all the pre-requisites.
+# If they are not met, Buildroot will actually do its job in a sub-make meeting
+# its pre-requisites, which are:
+# 1- Permissive enough umask:
+# Wrong or too restrictive umask will prevent Buildroot and packages from
+# creating files and directories.
+# 2- Absolute canonical CWD (i.e. $(CURDIR)):
+# Otherwise, some packages will use CWD as-is, others will compute its
+# absolute canonical path. This makes harder tracking and fixing host
+# machine path leaks.
+# 3- Absolute canonical output location (i.e. $(O)):
+# For the same reason as the one for CWD.
+
+# Current state:
+CUR_UMASK := $(shell umask)
+# Remove the trailing '/.' from $(O) as it can be added by the makefile wrapper
+# installed in the $(O) directory.
+O := $(patsubst %/.,%,$(O))
+
+# Buildroot requirements:
UMASK = 0022
-ifneq ($(shell umask),$(UMASK))
+REAL_CURDIR := $(realpath $(CURDIR))
+# realpath needs the entry to exists, otherwise an empty string is returned.
+REAL_O := $(shell mkdir -p $(O) >/dev/null 2>&1)$(realpath $(O))
+
+# Make sure O= is passed (with its absolute canonical path) everywhere the
+# toplevel makefile is called back.
+EXTRAMAKEARGS := O=$(REAL_O)
+
+# Check Buildroot execution pre-requisites here.
+ifneq ($(CUR_UMASK):$(CURDIR):$(O),$(UMASK):$(REAL_CURDIR):$(REAL_O))
.PHONY: _all $(MAKECMDGOALS)
$(MAKECMDGOALS): _all
@:
_all:
- @umask $(UMASK) && $(MAKE) --no-print-directory $(MAKECMDGOALS)
+ @umask $(UMASK) && \
+ $(MAKE) -C $(REAL_CURDIR) --no-print-directory \
+ $(MAKECMDGOALS) $(EXTRAMAKEARGS)
-else # umask
+else # umask / $(CURDIR) / $(O)
# This is our default rule, so must come first
all:
@@ -110,30 +160,10 @@ comma := ,
empty :=
space := $(empty) $(empty)
-# Set O variable if not already done on the command line;
-# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
-# build by preventing it from being forwarded to sub-make calls.
-ifneq ("$(origin O)", "command line")
-O := output
-else
-# other packages might also support Linux-style out of tree builds
-# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
-# forwards command line variable definitions those packages get very
-# confused. Fix this by telling make to not do so
-MAKEOVERRIDES =
-# strangely enough O is still passed to submakes with MAKEOVERRIDES
-# (with make 3.81 atleast), the only thing that changes is the output
-# of the origin function (command line -> environment).
-# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
-# To really make O go away, we have to override it.
-override O := $(O)
-# we need to pass O= everywhere we call back into the toplevel makefile
-EXTRAMAKEARGS = O=$(O)
-endif
-
# Set variables related to in-tree or out-of-tree build.
-ifeq ($(O),output)
-CONFIG_DIR := $(TOPDIR)
+# Here, both $(O) and $(CURDIR) are absolute canonical paths.
+ifeq ($(O),$(CURDIR)/output)
+CONFIG_DIR := $(CURDIR)
NEED_WRAPPER =
else
CONFIG_DIR := $(O)
@@ -1017,4 +1047,4 @@ include docs/manual/manual.mk
.PHONY: $(noconfig_targets)
-endif #umask
+endif #umask / $(CURDIR) / $(O)
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 03/11] support/scripts: add fix-rpath script + a bunch of helpers
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 02/11] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-07-07 5:23 ` Benjamin Kamath
2016-04-22 20:50 ` [Buildroot] [PATCH v9 04/11] core: sanitize HOST_DIR at the very end of the build Samuel Martin
` (8 subsequent siblings)
11 siblings, 1 reply; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
This commit introduces a fix-rpath shell-script able to scan a tree,
detect ELF files, check their RPATH and fix it in a proper way.
Along to the fix-rpath script, it also adds a bunch of shell helper
functions grouped into modules. This will help writing scripts handling
RPATH and other things, while allowing to share and reuse these
functions between scripts.
These helpers are namespaced within the filename of the module in which
they are gathered.
This change adds 6 modules:
- source.sh: provides simple helper to easily source another module, taking
care of not sourcing again an already-sourced one;
- log.sh: provides logging helpers;
- utils.sh: provides simple functions to filter ELF files in a list;
- readelf.sh: provides functions retrieving ELF details from a file;
- patchelf.sh: provides function updating ELF files;
- sdk.sh: provides RPATH computation functions.
These 6 modules are used by the fix-rpath script.
Follow-up patches will make some scripts leveraging these modules.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- none
changes v7->v8:
- slightly rework log.error (return non-0 instead of exiting; exit
decision is deferred to the caller)
changes v6->v7 (includes changes started during BR dev. days):
- log module:
- debug env. var. renamed: DEBUG -> SHELL_DEBUG
- trace now prints the message level
- patchelf module:
- get rid of XRPATH/XORIGIN stuff (Arnout)
- set_rpath now handles ELF files with no .dynamic section
- cleanup and minor fix in sanitize_rpath
- readelf module:
- move {filter,is}_elf* functions from utils to readelf modules
- add list_sections and has_section functions (needed in
sanitize_rpath)
- sdk modules:
- get rid of XRPATH/XORIGIN stuff (Arnout)
- minor functions doc. fixes
- source module:
- move "Could not load module" error message in load_module (Arnout)
- utils module:
- move {filter,is}_elf* functions from utils to readelf modules
- add list_has function
- fix-rpath:
- runs with "set -e" (exit on error)
- improve help text
- slight refactoring because of modules changes
changes v5->v6:
- fully rewritten in shell
changes v4->v5:
- add verbose support
- rename shrink_rpath -> clear_rpath
- add sanitize_rpath function
changes v3->v4:
- fix typos and license (Baruch)
changes v2->v3:
- no change
---
support/scripts/fix-rpath | 116 +++++++++++++++++++++++++
support/scripts/shell/log.sh | 61 +++++++++++++
support/scripts/shell/patchelf.sh | 178 ++++++++++++++++++++++++++++++++++++++
support/scripts/shell/readelf.sh | 173 ++++++++++++++++++++++++++++++++++++
support/scripts/shell/sdk.sh | 68 +++++++++++++++
support/scripts/shell/source.sh | 77 +++++++++++++++++
support/scripts/shell/utils.sh | 60 +++++++++++++
7 files changed, 733 insertions(+)
create mode 100755 support/scripts/fix-rpath
create mode 100644 support/scripts/shell/log.sh
create mode 100644 support/scripts/shell/patchelf.sh
create mode 100644 support/scripts/shell/readelf.sh
create mode 100644 support/scripts/shell/sdk.sh
create mode 100644 support/scripts/shell/source.sh
create mode 100644 support/scripts/shell/utils.sh
diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
new file mode 100755
index 0000000..fb2bfeb
--- /dev/null
+++ b/support/scripts/fix-rpath
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+set -e
+
+usage() {
+ cat <<EOF >&2
+Usage: ${0} TREE_KIND TREE_ROOT
+
+Description:
+
+ This script scans a tree and sanitize ELF files' RPATH found in there.
+
+ Sanitization behaves the same whatever the kindd of the processed tree, but
+ the resulting RPATH differs.
+
+ Sanitization action:
+ - remove RPATH pointing outside of the tree
+ - for RPATH pointing in the tree:
+ - if they point to standard location (/lib, /usr/lib): remove them
+ - otherwise: make them relative using \$ORIGIN
+
+ For the target tree:
+ - scan the whole tree for sanitization
+
+ For the staging tree :
+ - scan the whole tree for sanitization
+
+ For the host tree:
+ - skip the staging tree for sanitization
+ - add \$HOST_DIR/{lib,usr/lib} to RPATH (as relative pathes)
+
+Arguments:
+
+ TREE_KIND Kind of tree to be processed.
+ Allowed values: host, target, staging
+
+ TREE_ROOT Path to the root of the tree to be scaned
+
+Environment:
+
+ HOST_READELF readelf program to use for host ELF files
+ (default: readelf)
+
+ HOST_READELF readelf program to use for host ELF files
+ (default: readelf)
+
+ PATCHELF patchelf program to use
+ (default: patchelf)
+EOF
+}
+
+source "${0%/*}/shell/source.sh"
+
+source.load_module readelf
+source.load_module patchelf
+
+: ${HOST_READELF:=readelf}
+: ${TARGET_READELF:=readelf}
+: ${PATCHELF:=patchelf}
+
+main() {
+ local tree="${1}"
+ local basedir="$(readlink -f "${2}")"
+
+ local find_args=( "${basedir}" )
+ local sanitize_extra_args=()
+ local readelf
+
+ case "${tree}" in
+ host)
+ # do not process the sysroot (only contains target binaries)
+ find_args+=( "-name" "sysroot" "-prune" "-o" )
+
+ # do not process the external toolchain installation directory to
+ # to avoid breaking it.
+ find_args+=( "-path" "*/opt/ext-toolchain" "-prune" "-o" )
+
+ # make sure RPATH will point to ${hostdir}/lib and ${hostdir}/usr/lib
+ sanitize_extra_args+=( "keep_lib_and_usr_lib" )
+
+ readelf="${HOST_READELF}"
+ ;;
+ staging|target)
+ readelf="${TARGET_READELF}"
+ ;;
+ *)
+ usage
+ exit 1
+ ;;
+ esac
+
+ find_args+=( "-type" "f" "-print" )
+
+ while read file ; do
+ READELF="${readelf}" PATCHELF="${PATCHELF}" \
+ patchelf.sanitize_rpath "${basedir}" "${file}" ${sanitize_extra_args[@]}
+ done < <(find ${find_args[@]} | readelf.filter_elf)
+}
+
+main ${@}
diff --git a/support/scripts/shell/log.sh b/support/scripts/shell/log.sh
new file mode 100644
index 0000000..efadb3f
--- /dev/null
+++ b/support/scripts/shell/log.sh
@@ -0,0 +1,61 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Logging helpers
+#
+# This module defines the following functions:
+# log.trace
+# log.debug
+# log.info
+# log.warn
+# log.errorN
+# log.error
+#
+# This module sets the following variables:
+# my_name
+#
+# This module is sensitive to the following environment variables:
+# DEBUG
+
+source.declare_module log
+
+# Debug level:
+# - 0 or empty: only show errors
+# - 1 : show errors and warnings
+# - 2 : show errors, warnings, and info
+# - 3 : show errors, warnings, info and debug
+: ${SHELL_DEBUG:=0}
+
+# Low level utility function
+log.trace() {
+ local level="${1}" msg="${2}"
+ shift 2
+ printf "[%-5s] %s: ${msg}" "${level:0:5}" "${my_name}" "${@}"
+}
+
+# Public logging functions
+log.debug() { :; }
+[ ${SHELL_DEBUG} -lt 3 ] || log.debug() { log.trace DEBUG "${@}" >&2; }
+log.info() { :; }
+[ ${SHELL_DEBUG} -lt 2 ] || log.info() { log.trace INFO "${@}" >&2; }
+log.warn() { :; }
+[ ${SHELL_DEBUG} -lt 1 ] || log.warn() { log.trace WARN "${@}" >&2; }
+log.errorN() { local ret="${1}" ; shift ; log.trace ERROR "${@}" ; return ${ret} ; }
+log.error() { log.errorN 1 "${@}"; }
+
+# Program name
+my_name="${0##*/}"
+
diff --git a/support/scripts/shell/patchelf.sh b/support/scripts/shell/patchelf.sh
new file mode 100644
index 0000000..a035305
--- /dev/null
+++ b/support/scripts/shell/patchelf.sh
@@ -0,0 +1,178 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Patchelf helpers
+#
+# This module defines the following functions:
+# patchelf.set_rpath
+# patchelf.update_rpath
+# patchelf.sanitize_rpath
+#
+# This module is sensitive to the following environment variables:
+# PATCHELF
+# READELF
+
+source.declare_module patchelf
+
+source.load_module log
+source.load_module sdk
+source.load_module utils
+source.load_module readelf
+
+: ${PATCHELF:=patchelf}
+
+# patchelf.set_xrpath file rpath...
+#
+# Set RPATH in $file.
+# Automatically join all RPATH with the correct separator.
+#
+# file : ELF file path
+# rpath : RPATH element
+#
+# environment:
+# PATCHELF: patchelf program path
+patchelf.set_rpath() {
+ local file="${1}"
+ shift
+ local rpath="$(sed -e 's/ +/:/g' <<<"${@}")"
+ # Sanity check: patchelf needs the ELF file to have a .dynamic section.
+ # So, check for it and behaves in a proper way:
+ # - returns immediatly if no .dynamic section, and RPATH is empty;
+ # - bail out if no .dynamic section and RPATH is not empty.
+ if ! readelf.has_section "${file}" '.dynamic' ; then
+ if test -z "${rpath}" ; then
+ return 0
+ else
+ local fmt="Trying to set a RPATH to a ELF file with no .dynamic section\n"
+ fmt="${fmt}\tfile : %s\n"
+ fmt="${fmt}\tRPATH: %s\n"
+ log.error "${fmt}" "${file}" "${rpath}" || return 1
+ fi
+ fi
+ "${PATCHELF}" --set-rpath "${rpath}" "${file}"
+}
+
+# patchelf.update_rpath basedir binary libdirs...
+#
+# Set RPATH in $binary computing them from the paths $libdirs (and $basedir).
+# Existing RPATH in $file will be overwritten if any.
+#
+# basedir : absolute path of the tree in which the $bindir and $libdirs must be
+# binary : ELF file absolute path
+# libdirs : list of library location (absolute paths)
+#
+# environment:
+# PATCHELF: patchelf program path
+patchelf.update_rpath() {
+ local basedir="${1}"
+ local binary="${2}"
+ shift 2
+ local libdirs=( ${@} )
+ log.debug " basedir: %s\n" "${basedir}"
+ log.debug " elf: %s\n" "${binary}"
+ log.debug " libdirs: %s\n" "${libdirs[*]}"
+ log.info " rpath: %s\n" \
+ "$(sdk.compute_rpath "${basedir}" "${binary%/*}" ${libdirs[@]})"
+ patchelf.set_rpath "${binary}" \
+ "$(sdk.compute_rpath "${basedir}" "${binary%/*}" ${libdirs[@]})"
+}
+
+# patchelf.sanitize_rpath basedir binary [keep_lib_usr_lib]
+#
+# Scan $binary's RPATH, remove any of them pointing outside of $basedir.
+# If $keep_lib_usr_lib in not empty, the library directories $basedir/lib and
+# $basedir/usr/lib will be added to the RPATH.
+#
+# Note:
+# Absolute paths is needed to correctly handle symlinks and or mount-bind in
+# the $basedir path.
+#
+# basedir : absolute path of the tree in which the $bindir and $libdirs
+# must be
+# binary : ELF file absolute path
+# keep_lib_usr_lib : add to RPATH $basedir/lib and $basedir/usr/lib
+#
+# environment:
+# PATCHELF: patchelf program path
+# READELF : readelf program path
+patchelf.sanitize_rpath() {
+ local basedir="$(readlink -f "${1}")"
+ local binary="${2}"
+ local keep_lib_usr_lib="${3}"
+
+ readelf.is_elf_shared_object "${binary}" ||
+ readelf.is_elf_executable "${binary}" ||
+ return 0
+
+ local path abspath rpath
+ local libdirs=()
+
+ if test -n "${keep_lib_usr_lib}" ; then
+ libdirs+=( "${basedir}/lib" "${basedir}/usr/lib" )
+ fi
+
+ log.info "ELF: %s\n" "${binary}"
+
+ local rpaths="$(readelf.get_rpath "${binary}")"
+
+ for rpath in ${rpaths//:/ } ; do
+ # figure out if we should keep or discard the path; there are several
+ # cases to handled:
+ # - $path starts with "$ORIGIN":
+ # The original build-system already took care of setting a relative
+ # RPATH, resolve it and test if it is worthwhile to keep it;
+ # - $basedir/$path exists:
+ # The original build-system already took care of setting an absolute
+ # RPATH (absolute in the final rootfs), resolve it and test if it is
+ # worthwhile to keep it;
+ # - $path start with $basedir:
+ # The original build-system added some absolute RPATH (absolute on
+ # the build machine). While this is wrong, it can still be fixed; so
+ # test if it is worthwhile to keep it;
+ # - $path points somewhere else:
+ # (can be anywhere: build trees, staging tree, host location,
+ # non-existing location, etc.)
+ # Just discard such a path.
+ if grep -q '^$ORIGIN/' <<<"${rpath}" ; then
+ path="${binary%/*}/${rpath#*ORIGIN/}"
+ elif test -e "${basedir}/${rpath}" ; then
+ path="${basedir}/${rpath}"
+ elif grep -q "^${basedir}/" <<<"$(readlink -f "${rpath}")" ; then
+ path="${rpath}"
+ else
+ log.debug "\tDROPPED [out-of-tree]: %s\n" "${rpath}"
+ continue
+ fi
+
+ abspath="$(readlink -f "${path}")"
+
+ # discard path pointing to default locations handled by ld-linux
+ if grep -qE "^${basedir}/(lib|usr/lib)$" <<<"${abspath}" ; then
+ log.debug \
+ "\tDROPPED [std libdirs]: %s (%s)\n" "${rpath}" "${abspath}"
+ continue
+ fi
+
+ log.debug "\tKEPT %s (%s)\n" "${rpath}" "${abspath}"
+
+ libdirs+=( "${abspath}" )
+
+ done
+
+ libdirs=( $(utils.list_reduce ${libdirs[@]}) )
+
+ patchelf.update_rpath "${basedir}" "${binary}" ${libdirs[@]}
+}
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
new file mode 100644
index 0000000..c8ad38b
--- /dev/null
+++ b/support/scripts/shell/readelf.sh
@@ -0,0 +1,173 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Readelf helpers
+#
+# This module defines the following functions:
+# readelf._filter_elf_regexp
+# readelf.filter_elf
+# readelf.filter_elf_executable
+# readelf.filter_elf_shared_object
+# readelf.is_elf_executable
+# readelf.is_elf_shared_object
+# readelf.get_rpath
+# readelf.list_sections
+# readelf.has_section
+#
+# This module is sensitive to the following environment variables:
+# READELF
+source.declare_module readelf
+
+# When calling readelf(1) program, the user's locale will be overriden with the
+# C locale, so we are sure we can reliably parse its output.
+: ${READELF:=readelf}
+
+# readelf._filter_elf_regexp filter_cmd file...
+#
+# Filters ELF files WRT the given regular extended expression.
+# This funtion can take one or several files, or read them from stdin.
+#
+# filter_cmd : filter command (usually based on grep)
+# file : list of files to be filtered
+#
+# environment:
+# READELF: readelf program path
+readelf._filter_elf_regexp() {
+ local regexp="${1}"
+ shift
+ local in file
+ test ${#} -gt 0 && in='printf "%s\n" "${@}"' || in='dd 2>/dev/null'
+ eval "${in}" |
+ while read file ; do
+ LC_ALL=C ${READELF} -h "${file}" 2>/dev/null |
+ grep -qE "${regexp}" ||
+ continue
+ printf "%s\n" "${file}"
+ done
+}
+
+# readelf.filter_elf file...
+#
+# Filters ELF files; if $file is an ELF file, $file is printed, else it is
+# discarded.
+# This funtion can take one or several arguments, or read them from stdin.
+#
+# file : path of file to be filtered
+#
+# environment:
+# READELF: readelf program path
+readelf.filter_elf() {
+ readelf._filter_elf_regexp "Class:\s+ELF" "${@}"
+}
+
+# readelf.filter_elf_shared_object file...
+#
+# Filters ELF files; if $file is an ELF file, $file is printed, else it is
+# discarded.
+# This funtion can take one or several arguments, or read them from stdin.
+#
+# file : path of file to be filtered
+#
+# environment:
+# READELF: readelf program path
+readelf.filter_elf_shared_object() {
+ readelf._filter_elf_regexp "Type:\s+DYN\s\(Shared\sobject\sfile\)" "${@}"
+}
+
+# readelf.filter_elf_executable file...
+#
+# Filters ELF files; if $file is an ELF file, $file is printed, else it is
+# discarded.
+# This funtion can take one or several arguments, or read them from stdin.
+#
+# file : path of file to be filtered
+#
+# environment:
+# READELF: readelf program path
+readelf.filter_elf_executable() {
+ readelf._filter_elf_regexp "Type:\s+EXEC\s\(Executable\sfile\)" "${@}"
+}
+
+# readelf.is_elf_shared_object file
+#
+# Returns 0 if $file is an ELF file, non-0 otherwise.
+#
+# file : path of file to be tested
+#
+# environment:
+# READELF: readelf program path
+readelf.is_elf_shared_object() {
+ test "$(readelf.filter_elf_shared_object "${1}")" != ""
+}
+
+# readelf.is_elf_executable file
+#
+# Returns 0 if $file is an ELF file, non-0 otherwise.
+#
+# file : path of file to be tested
+#
+# environment:
+# READELF: readelf program path
+readelf.is_elf_executable() {
+ test "$(readelf.filter_elf_executable "${1}")" != ""
+}
+
+# readelf.get_rpath file
+#
+# Return the unsplitted RPATH/RUNPATH of $file.
+#
+# To split the returned RPATH string and store them in an array, do:
+#
+# paths=( $(readelf.get_rpath "${file}" | sed -e 's/:/ /g') )
+#
+# file : ELF file path
+#
+# environment:
+# READELF: readelf program path
+readelf.get_rpath() {
+ local file="${1}"
+ LC_ALL=C "${READELF}" --dynamic "${file}" |
+ sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d ; s//\3/'
+}
+
+# readelf.list_sections file
+#
+# Returns the list of ELF sections in $file.
+#
+# file : ELF file path
+#
+# environment:
+# READELF: readelf program path
+readelf.list_sections() {
+ local file="${1}"
+ LC_ALL=C "${READELF}" --sections "${file}" |
+ sed -re '/^ \[ *[0-9]+\] (\S+).*/!d ; s//\1/' |
+ sort
+}
+
+# readelf.has_section file section
+#
+# Return 0 if $file has a section named $section
+#
+# file : ELF file path
+# section : ELF section name
+#
+# environment:
+# READELF: readelf program path
+readelf.has_section() {
+ local file="${1}" section_name="${2}"
+ readelf.list_sections "${file}" | grep -q "^${section_name}$"
+}
diff --git a/support/scripts/shell/sdk.sh b/support/scripts/shell/sdk.sh
new file mode 100644
index 0000000..b2f699c
--- /dev/null
+++ b/support/scripts/shell/sdk.sh
@@ -0,0 +1,68 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# SDK helpers
+#
+# This module defines the following functions:
+# sdk.compute_relative_path
+# sdk.compute_rpath
+
+source.declare_module sdk
+
+# sdk.compute_relative_path basedir path start
+#
+# Computes and prints the relative path between $start and $path within $basedir.
+#
+# basedir : absolute path of the tree in which the $path and $start must be
+# path : destination absolute path
+# start : origin absolute path
+sdk.compute_relative_path() {
+ local basedir="${1}"
+ local path="${2}"
+ local start="${3}"
+ # sanity checks: make sure $path and $start starts with $basedir
+ grep -q "^${basedir}" <<<"${path}" || return 1
+ grep -q "^${basedir}" <<<"${start}" || return 1
+ local i
+ local backward="${start#${basedir}}"
+ local relative=()
+ for i in ${backward//\// } ; do
+ # don't need to check for empty items they are already discarded
+ test "${i}" != '.' || continue
+ relative+=( ".." )
+ done
+ relative+=( ${path#${basedir}} )
+ sed -r -e 's:[ /]+:/:g' <<<"${relative[@]}"
+}
+
+# sdk.compute_rpath basedir bindir libdirs...
+#
+# Computes and prints the list of RPATH.
+#
+# basedir : absolute path of the tree in which the $bindir and $libdirs must be
+# bindir : binary directory absolute path
+# libdirs : list of library directories (absolute paths)
+sdk.compute_rpath() {
+ local basedir="${1}"
+ local bindir="${2}"
+ shift 2
+ local libdirs=( ${@} )
+ local rpath=()
+ for libdir in ${libdirs[@]} ; do
+ rpath+=( "\$ORIGIN/$(sdk.compute_relative_path "${basedir}" "${libdir}" "${bindir}")" )
+ done
+ sed -e 's/ /:/g' <<<"${rpath[@]}"
+}
diff --git a/support/scripts/shell/source.sh b/support/scripts/shell/source.sh
new file mode 100644
index 0000000..70b7fac
--- /dev/null
+++ b/support/scripts/shell/source.sh
@@ -0,0 +1,77 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Source helpers
+#
+# This module defines the following functions:
+# source.declare_module
+# source.load_module
+#
+# This module is sensitive to the following environment variables:
+# TOPDIR
+
+# Assuming the script sourcing this file is in support/scripts/
+: ${TOPDIR:=$(readlink -f "${0}" | sed -re 's:(/[^/]+){3}$::')}
+
+# source.declare_module module_name
+#
+# Declare a shell module.
+# Set the variable '_source_${module_name}'.
+# Should be called once per module, in the global scope.
+#
+# module_name : Module name (allowed char.: [_a-zA-Z0-9])
+source.declare_module() {
+ local module_name="${1}"
+ # use printf from bash to set the variable in the environment:
+ printf -v "_source_${module_name}" "%s" "${module_name}"
+}
+
+# source.load_module module_name
+#
+# Load the given shell module, making available all functions declared
+# in it, ensuring it is not reloaded if it already is.
+# Should be called in the global scope.
+# Need the TOPDIR environment variable.
+#
+# param module_name: Module name
+source.load_module() {
+ local module_name="${1}"
+ local loaded="loaded=\${_source_${module_name}}"
+ eval "${loaded}"
+ local module_file="${TOPDIR}/support/scripts/shell/${module_name}.sh"
+
+ if [ ! -f "${module_file}" ] ; then
+ cat <<EOF >&2
+error: Could load module '${module_name}',
+ ${module_file} does not exists.
+
+ Maybe TOPDIR does not point to Buildroot's '\$(TOPDIR)'.
+
+ Or this script '${0##*/}' is most not installed in Buildroot's
+ '\$(TOPDIR)/support/scripts' directory.
+
+ You can fix this by:
+ - either installing '${0##*/}' in the support/scripts/ directory;
+ - or setting the TOPDIR variable in the '${0##*/}' script, before
+ sourcing anything.
+EOF
+ exit 1
+ fi
+
+ test -n "${loaded}" || source "${module_file}"
+}
+
+source.declare_module source
diff --git a/support/scripts/shell/utils.sh b/support/scripts/shell/utils.sh
new file mode 100644
index 0000000..9e9aaab
--- /dev/null
+++ b/support/scripts/shell/utils.sh
@@ -0,0 +1,60 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Miscellaneous helpers
+#
+# This module defines the following functions:
+# utils.list_has
+# utils.list_reduce
+
+source.declare_module utils
+
+# utils.list_has value list_items...
+#
+# Returns 0 if $list_items contains $value, returns 1 otherwise.
+#
+# value : item to be checked if it is in the list
+# list_items : list of items
+utils.list_has() {
+ local key=$1
+ shift
+ for val in $@ ; do
+ if test "$val" = "$key" ; then
+ return 0
+ fi
+ done
+ return 1
+}
+
+# utils.list_reduce input_list
+#
+# Prints the $input_list list with duplicated items removed.
+# Order is preserved, WRT the first occurence of duplicated items.
+#
+# input_list : list of items
+utils.list_reduce() {
+ local -a lout # return list
+ local i
+
+ for i in ${@} ; do
+ if utils.list_has "${i}" ${lout[@]} ; then
+ continue
+ fi
+ lout+=( "${i}" )
+ done
+
+ echo ${lout[@]}
+}
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 04/11] core: sanitize HOST_DIR at the very end of the build
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
` (2 preceding siblings ...)
2016-04-22 20:50 ` [Buildroot] [PATCH v9 03/11] support/scripts: add fix-rpath script + a bunch of helpers Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 05/11] core: add {TARGET, STAGING}_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS Samuel Martin
` (7 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
This patch adds host-patchelf to the list of package to build, and use
it in the SANITIZE_RPATH_HOST helper.
This SANITIZE_RPATH_HOST helper is executed in the world target, which
ensure:
- to be built after the target-post-image target (because of the
dependency);
- to always be built since it is the only dependency of the all (and
default) target.
The SANITIZE_RPATH_HOST helper fixes the ELF files' RPATH from the
HOST_DIR location (excluding the sysroot).
After running this helper is run, the RPATH from any host ELF files is
relative to the binary location itself.
Notes:
- we avoid to fix RPATH in the sysroot.
- we do not try to fix RPATH in the external toolchain installation
location as they may have been built in a way, this is already
correct; furthermore, attempting to fix RPATH in those programs may
result in breaking them.
- the whole host directory is processed because a number of
host-package install programs that could be useful in places
different from $(HOST_DIR)/{bin,sbin,usr/bin,usr/sbin}.
- the shared libraries are also processed in case they have a 'main'
function.
As a step toward a fully relocatable SDK, this change allows to get the
toolchain relocatable, but not yet the whole SDK.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- remove irrelevant comment (ThomasP)
- move the helper call to the world target instead of as a
target-finalize hooks (ThomasP)
changes v7->v8:
- none
changes v6->v7:
- remove the DEBUG env. var. specific handling in Makefile, now just
place the variable in the Buildroot environment, to enable debug logs.
This is documented in a follow-up patch of this series.
changes v5->v6:
- update for the new script version
- add debug mode support
changes v4->v5:
- add verbose support
changes v3->v4:
- add host-patchelf to PACKAGES instead of target-finalize (Baruch)
- add comment
changes v2->v3:
- move hook in Makefile (Baruch)
---
Makefile | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/Makefile b/Makefile
index a05b9e1..0c9bb98 100644
--- a/Makefile
+++ b/Makefile
@@ -526,7 +526,20 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
prepare: $(BUILD_DIR)/buildroot-config/auto.conf
+# RPATH fixing
+# - The host hook sets RPATH in host ELF binaries, using relative paths to the
+# library locations.
+PACKAGES += host-patchelf
+
+define SANITIZE_RPATH_HOST
+ PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+ READELF=readelf \
+ $(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
+endef
+
world: target-post-image
+ @$(call MESSAGE,"Rendering the SDK relocatable")
+ $(SANITIZE_RPATH_HOST)
.PHONY: all world toolchain dirs clean distclean source outputmakefile \
legal-info legal-info-prepare legal-info-clean printvars help \
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 05/11] core: add {TARGET, STAGING}_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
` (3 preceding siblings ...)
2016-04-22 20:50 ` [Buildroot] [PATCH v9 04/11] core: sanitize HOST_DIR at the very end of the build Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 06/11] package/speex: remove no longer needed hook Samuel Martin
` (6 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
This patch introduces the TARGET_SANITIZE_RPATH_HOOK and
STAGING_SANITIZE_RPATH_HOOK hooks fixing the ELF files' RPATH of
binaries from, respectively, the TARGET_DIR and the STAGING_DIR
locations.
It is a fair assumption that all target package has been built before
reaching the target-finalize target; hence the execution of these hooks
as TARGET_FINALIZE_HOOKS.
After running these hooks, the RPATH from any target ELF files from both
the target and the staging locations won't contain any occurrence of the
sysroot or some build locations.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- none
changes v7->v8:
- none
changes v6->v7:
- remove the DEBUG env. var. specific handling in Makefile
changes v5->v6:
- update for the new script version
- add debug mode support
changes v4->v5:
- target hook now sanitizes the rpath (Baruch)
- add verbose support
- update comment
changes v3->v4:
- rebase
- add comment
changes v2->v3:
- move hook in Makefile (Baruch)
---
Makefile | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/Makefile b/Makefile
index 0c9bb98..e92826f 100644
--- a/Makefile
+++ b/Makefile
@@ -651,6 +651,24 @@ endef
TARGET_FINALIZE_HOOKS += PURGE_LOCALES
endif
+# Function sanitizing target/staging ELF files' RPATH.
+# i.e. it removes paths pointing to the staging or build location from the ELF
+# files' RPATH.
+define TARGET_SANITIZE_RPATH_HOOK
+ PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+ READELF=$(TARGET_READELF) \
+ $(TOPDIR)/support/scripts/fix-rpath target $(TARGET_DIR)
+endef
+
+define STAGING_SANITIZE_RPATH_HOOK
+ PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+ READELF=$(TARGET_READELF) \
+ $(TOPDIR)/support/scripts/fix-rpath staging $(STAGING_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += TARGET_SANITIZE_RPATH_HOOK \
+ STAGING_SANITIZE_RPATH_HOOK
+
$(TARGETS_ROOTFS): target-finalize
target-finalize: $(PACKAGES)
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 06/11] package/speex: remove no longer needed hook
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
` (4 preceding siblings ...)
2016-04-22 20:50 ` [Buildroot] [PATCH v9 05/11] core: add {TARGET, STAGING}_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 07/11] support/scripts: update check-host-rpath to use the shell helpers Samuel Martin
` (5 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
Remove the LIBTOOL_FIXUP hook since RPATH are now sanitized in
target-finalize hooks.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- remove hook addition (ThomasP)
changes v7->v8:
- none
changes v6->v7:
- none
changes v5->v6:
- none
changes v4->v5:
- new patch (suggested by Baruch)
---
package/speex/speex.mk | 7 -------
1 file changed, 7 deletions(-)
diff --git a/package/speex/speex.mk b/package/speex/speex.mk
index a34bfac..86c5cef 100644
--- a/package/speex/speex.mk
+++ b/package/speex/speex.mk
@@ -24,13 +24,6 @@ ifeq ($(BR2_PACKAGE_SPEEX_ARM5E),y)
SPEEX_CONF_OPTS += --enable-arm5e-asm
endif
-define SPEEX_LIBTOOL_FIXUP
- $(SED) 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' $(@D)/libtool
- $(SED) 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' $(@D)/libtool
-endef
-
-SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP
-
define SPEEX_BUILD_CMDS
$($(PKG)_MAKE_ENV) $(MAKE) $($(PKG)_MAKE_OPTS) -C $(@D)/$($(PKG)_SUBDIR)
endef
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 07/11] support/scripts: update check-host-rpath to use the shell helpers
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
` (5 preceding siblings ...)
2016-04-22 20:50 ` [Buildroot] [PATCH v9 06/11] package/speex: remove no longer needed hook Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 08/11] support/scripts: add check-host-leaks script + all needed helpers Samuel Martin
` (4 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
This change moves and refactors the helper functions from the
check-host-rpath into the readelf module.
The replacement of the elf_needs_rpath function: readelf.needs_rpath
does not fundamentally change the implementation, it only uses the APIs
offered by the readelf module.
The replacement of the check_elf_has_rpath function: readelf.has_rpath
does a bit more than the original implementation - adding the support
for relative RPATH (those strating with '$ORIGIN').
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- update commit log (ThomasP)
- minor fix
changes v7->v8:
- none
changes v6->v7:
- minor updates after shell modules shell changes
changes v5->v6:
- new patch
---
support/scripts/check-host-rpath | 73 +++++++++++++++-------------------------
support/scripts/shell/readelf.sh | 69 +++++++++++++++++++++++++++++++++++++
2 files changed, 97 insertions(+), 45 deletions(-)
diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath
index 6ce547c..e6e82d6 100755
--- a/support/scripts/check-host-rpath
+++ b/support/scripts/check-host-rpath
@@ -1,12 +1,29 @@
#!/usr/bin/env bash
+# Copyright (C) 2015 Yann E. MORIN <yann.morin.1998@free.fr>
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
# This script scans $(HOST_DIR)/{bin,sbin} for all ELF files, and checks
# they have an RPATH to $(HOST_DIR)/usr/lib if they need libraries from
# there.
-# Override the user's locale so we are sure we can parse the output of
-# readelf(1) and file(1)
-export LC_ALL=C
+source "${0%/*}/shell/source.sh"
+
+source.load_module readelf
main() {
local pkg="${1}"
@@ -14,58 +31,24 @@ main() {
local file ret
# Remove duplicate and trailing '/' for proper match
- hostdir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${hostdir}" )"
+ hostdir="$(sed -r -e 's:/+:/:g; s:/$::;' <<<"${hostdir}")"
ret=0
while read file; do
- elf_needs_rpath "${file}" "${hostdir}" || continue
- check_elf_has_rpath "${file}" "${hostdir}" && continue
+ READELF=readelf readelf.needs_rpath "${file}" "${hostdir}" || continue
+ READELF=readelf readelf.has_rpath "${file}" "${hostdir}" && continue
if [ ${ret} -eq 0 ]; then
ret=1
printf "***\n"
- printf "*** ERROR: package %s installs executables without proper RPATH:\n" "${pkg}"
+ printf \
+ "*** ERROR: package %s installs executables without proper RPATH:\n" \
+ "${pkg}"
fi
printf "*** %s\n" "${file}"
- done < <( find "${hostdir}"/{,usr/}{bin,sbin} -type f -exec file {} + 2>/dev/null \
- |sed -r -e '/^([^:]+):.*\<ELF\>.*\<executable\>.*/!d' \
- -e 's//\1/' \
- )
+ done < <(find "${hostdir}"/{,usr/}{bin,sbin} -type f -print 2>/dev/null |
+ readelf.filter_elf_executable)
return ${ret}
}
-elf_needs_rpath() {
- local file="${1}"
- local hostdir="${2}"
- local lib
-
- while read lib; do
- [ -e "${hostdir}/usr/lib/${lib}" ] && return 0
- done < <( readelf -d "${file}" \
- |sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d;' \
- -e 's//\1/;' \
- )
-
- return 1
-}
-
-check_elf_has_rpath() {
- local file="${1}"
- local hostdir="${2}"
- local rpath dir
-
- while read rpath; do
- for dir in ${rpath//:/ }; do
- # Remove duplicate and trailing '/' for proper match
- dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
- [ "${dir}" = "${hostdir}/usr/lib" ] && return 0
- done
- done < <( readelf -d "${file}" \
- |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \
- -e 's//\3/;' \
- )
-
- return 1
-}
-
main "${@}"
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
index c8ad38b..78a1a9f 100644
--- a/support/scripts/shell/readelf.sh
+++ b/support/scripts/shell/readelf.sh
@@ -24,11 +24,15 @@
# readelf.is_elf_executable
# readelf.is_elf_shared_object
# readelf.get_rpath
+# readelf.get_neededs
+# readelf.needs_rpath
+# readelf.has_rpath
# readelf.list_sections
# readelf.has_section
#
# This module is sensitive to the following environment variables:
# READELF
+
source.declare_module readelf
# When calling readelf(1) program, the user's locale will be overriden with the
@@ -143,6 +147,71 @@ readelf.get_rpath() {
sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d ; s//\3/'
}
+# readelf.get_neededs file
+#
+# Returns the list of the NEEDED libraries of $file.
+#
+# file : ELF file path
+#
+# environment:
+# READELF: readelf program path
+readelf.get_neededs() {
+ local file="${1}"
+ LC_ALL=C "${READELF}" --dynamic "${file}" |
+ sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d ; s//\1/'
+}
+
+# readelf.needs_rpath file basedir
+#
+# Returns 0 if $file needs to have RPATH set, 1 otherwise.
+#
+# file : path of file to be tested
+# basedir : path of the tree in which $basedir/lib and $basedir/usr/lib are
+# checked for belonging to RPATH
+#
+# environment:
+# READELF: readelf program path
+readelf.needs_rpath() {
+ local file="${1}"
+ local basedir="${2}"
+ local lib
+
+ while read lib; do
+ [ -e "${basedir}/lib/${lib}" ] && return 0
+ [ -e "${basedir}/usr/lib/${lib}" ] && return 0
+ done < <(readelf.get_neededs "${file}")
+ return 1
+}
+
+# readelf.has_rpath file basedir
+#
+# Returns 0 if $file has RPATH already set to $basedir/lib or $basedir/usr/lib,
+# or uses relative RPATH (starting with "$ORIGIN"); returns 1 otherwise.
+#
+# file : path of file to be tested
+# basedir : path of the tree in which $basedir/lib and $basedir/usr/lib are
+# checked for belonging to RPATH
+#
+# environment:
+# READELF: readelf program path
+readelf.has_rpath() {
+ local file="${1}"
+ local basedir="${2}"
+ local rpath dir
+
+ while read rpath; do
+ for dir in ${rpath//:/ }; do
+ # Remove duplicate and trailing '/' for proper match
+ dir="$(sed -r -e "s:/+:/:g; s:/$::" <<<"${dir}")"
+ [ "${dir}" = "${basedir}/lib" ] && return 0
+ [ "${dir}" = "${basedir}/usr/lib" ] && return 0
+ grep -q '^\$ORIGIN/' <<<"${dir}" && return 0
+ done
+ done < <(readelf.get_rpath "${file}")
+
+ return 1
+}
+
# readelf.list_sections file
#
# Returns the list of ELF sections in $file.
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 08/11] support/scripts: add check-host-leaks script + all needed helpers
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
` (6 preceding siblings ...)
2016-04-22 20:50 ` [Buildroot] [PATCH v9 07/11] support/scripts: update check-host-rpath to use the shell helpers Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 09/11] core: add check-leaks-in-{target, host, staging} targets Samuel Martin
` (3 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- none
changes v7->v8:
- move sdk.check_host_leaks in the check-host-leaks script itself
(Arnout)
- drop leak classification (Arnout)
- rework readelf.is_elf_{static_library,object} (Arnout)
- add utils.assert_absolute_canonical_path
- drop utils.guess_gnu_target_name
- rework/simplify the check-host-leaks script command line
changes v6->v7:
- {filer,is}_elf* functions moved from utils to readelf module
- update sdk.check_host_leaks
changes v5->v6:
- new patch
---
support/scripts/check-host-leaks | 126 +++++++++++++++++++++++++++++++++++++++
support/scripts/shell/readelf.sh | 77 ++++++++++++++++++++++--
support/scripts/shell/sdk.sh | 7 +++
support/scripts/shell/utils.sh | 14 +++++
4 files changed, 218 insertions(+), 6 deletions(-)
create mode 100755 support/scripts/check-host-leaks
diff --git a/support/scripts/check-host-leaks b/support/scripts/check-host-leaks
new file mode 100755
index 0000000..9f363af
--- /dev/null
+++ b/support/scripts/check-host-leaks
@@ -0,0 +1,126 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+usage() {
+ local ret="${1:-0}"
+ cat <<EOF
+
+Usage: ${0##*/} [options] ROOT_DIR LEAK_PATH...
+
+Description:
+
+ This script scans a tree for host paths leaked into it.
+ Prints out leaks alongside with the kind of leak.
+
+Arguments:
+
+ ROOT_DIR Path to the root of the tree to be scanned
+
+ LEAK_PATH Paths to check for occurence. Several paths can be
+ passed.
+ A LEAK_PATH must be an absolute canonical path.
+ In the Buildroot context, the LEAK_PATH are Buildroot's
+ TOPDIR, BASE_DIR, HOST_DIR and STAGING_DIR.
+
+Options:
+
+ --exclude=PATH
+ Excludes files from ROOT_DIR starting with PATH. Obviously, the
+ excluded paths should point to some sub-location of ROOT_DIR.
+ Can be set more than once.
+ An excluded path must be an absolute canonical path.
+
+
+EOF
+ return ${ret}
+}
+
+source "${0%/*}/shell/source.sh"
+
+source.load_module log
+source.load_module utils
+source.load_module sdk
+
+main() {
+ local root_dir
+ local class_leaks
+ local excluded=()
+ local leak_paths=()
+ while test ${#} -gt 0 ; do
+ case "${1}" in
+ --exclude=*) excluded+=( "${1#*=}" )
+ ;;
+ --exclude) shift ; excluded+=( "${1}" )
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *) if test -z "${root_dir}" ; then root_dir="${1}"
+ else leak_paths+=( "${1}" )
+ fi
+ ;;
+ esac
+ shift
+ done
+
+ excluded=( $(utils.list_reduce ${excluded[@]}) )
+ leak_paths=( $(utils.list_reduce ${leak_paths[@]}) )
+
+ # sanity checks
+ if test -z "${root_dir}" ; then
+ log.error "Not enough argument: no ROOT_DIR given.\n" ||
+ usage ${?} || exit ${?}
+ fi
+ if test ${#leak_paths[@]} -lt 1 ; then
+ log.error "Not enough argument. No LEAK_PATH given.\n" ||
+ usage ${?} || exit ${?}
+ fi
+
+ local path has_non_abspath
+ for path in ${leak_paths[@]} ; do
+ utils.assert_absolute_canonical_path "${path}" || has_non_abspath=y
+ done
+ if test -n "${has_non_abspath}" ; then
+ log.error "Some LEAK_PATH are not absolute canonical paths.\n" ||
+ usage ${?} || exit ${?}
+ fi
+ has_non_abspath=
+ for path in ${excluded[@]} ; do
+ utils.assert_absolute_canonical_path "${path}" || has_non_abspath=y
+ done
+ if test -n "${has_non_abspath}" ; then
+ log.error "Some excluded path are not absolute canonical paths.\n" ||
+ usage ${?} || exit ${?}
+ fi
+
+ local re_leaks="$(sed -re 's/^/(/ ; s/$/)/ ; s/ +/|/g' <<<"${leak_paths[*]}")"
+ local re_excl="$(sed -re 's/ +/|/g' <<<"${excluded[*]}")"
+ if test -n "${re_excl}" ; then
+ re_excl="\\:^(${re_excl}):d"
+ fi
+ pushd "${root_dir}" >/dev/null
+ local f
+ grep -raEl "${re_leaks}" . |
+ sed -re "${re_excl} ; s:^\.:${root_dir}:" |
+ while read f ; do
+ printf "%s\n" "${f}"
+ done | sort
+ popd >/dev/null
+}
+main "${@}"
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
index 78a1a9f..b23666e 100644
--- a/support/scripts/shell/readelf.sh
+++ b/support/scripts/shell/readelf.sh
@@ -17,18 +17,22 @@
# Readelf helpers
#
# This module defines the following functions:
+# readelf._match_elf_regexp
# readelf._filter_elf_regexp
# readelf.filter_elf
# readelf.filter_elf_executable
# readelf.filter_elf_shared_object
# readelf.is_elf_executable
# readelf.is_elf_shared_object
+# readelf.is_elf_static_library
+# readelf.is_elf_object
# readelf.get_rpath
# readelf.get_neededs
# readelf.needs_rpath
# readelf.has_rpath
# readelf.list_sections
# readelf.has_section
+# readelf.string_section
#
# This module is sensitive to the following environment variables:
# READELF
@@ -39,12 +43,28 @@ source.declare_module readelf
# C locale, so we are sure we can reliably parse its output.
: ${READELF:=readelf}
-# readelf._filter_elf_regexp filter_cmd file...
+# readelf._match_elf_regexp regexp file
#
-# Filters ELF files WRT the given regular extended expression.
+# Returns 0 if the ELF file matches the ELF type given in extended regular
+# expression, non-0 otherwise.
+#
+# regexp : extended regular expression
+# file : list of files to be filtered
+#
+# environment:
+# READELF: readelf program path
+readelf._match_elf_regexp() {
+ log._trace_func
+ local regexp="${1}" file="${2}"
+ LC_ALL=C ${READELF} -h "${file}" 2>/dev/null | grep -qE "${regexp}"
+}
+
+# readelf._filter_elf_regexp regexp file...
+#
+# Filters ELF files WRT the given extended regular expression.
# This funtion can take one or several files, or read them from stdin.
#
-# filter_cmd : filter command (usually based on grep)
+# regexp : extended regular expression
# file : list of files to be filtered
#
# environment:
@@ -56,9 +76,7 @@ readelf._filter_elf_regexp() {
test ${#} -gt 0 && in='printf "%s\n" "${@}"' || in='dd 2>/dev/null'
eval "${in}" |
while read file ; do
- LC_ALL=C ${READELF} -h "${file}" 2>/dev/null |
- grep -qE "${regexp}" ||
- continue
+ readelf._match_elf_regexp "${regexp}" "${file}" || continue
printf "%s\n" "${file}"
done
}
@@ -129,6 +147,39 @@ readelf.is_elf_executable() {
test "$(readelf.filter_elf_executable "${1}")" != ""
}
+# readelf.is_elf file
+#
+# Returns 0 if $file is an ELF file, non-0 otherwise.
+#
+# file : path of file to be tested
+#
+# environment:
+# READELF: readelf program path
+readelf.is_elf() {
+ test "$(readelf.filter_elf "${1}")" != ""
+}
+
+# readelf.is_elf_static_library file
+#
+# Return 0 if $file is a Linux static libraries, i.e. an ar-archive
+# containing *.o files.
+#
+# file : path of file to be tested
+readelf.is_elf_static_library() {
+ readelf._match_elf_regexp "Type:\s+REL\s\(Relocatable\sfile\)" "${@}" &&
+ readelf._match_elf_regexp "^File:\s+\S+\)$" "${@}"
+}
+
+# readelf.is_elf_object file
+#
+# Return 0 if $file is an ELF object file, i.e. a *.o (or *.ko) file.
+#
+# file : path of file to be tested
+readelf.is_elf_object() {
+ readelf._match_elf_regexp "Type:\s+REL\s\(Relocatable\sfile\)" "${@}" &&
+ ! readelf._match_elf_regexp "^File:\s+\S+\)$" "${@}"
+}
+
# readelf.get_rpath file
#
# Return the unsplitted RPATH/RUNPATH of $file.
@@ -240,3 +291,17 @@ readelf.has_section() {
local file="${1}" section_name="${2}"
readelf.list_sections "${file}" | grep -q "^${section_name}$"
}
+
+# readelf.string_section file section
+#
+# Return the given $section of $file.
+#
+# file : ELF file path
+# section : ELF section name
+#
+# environment:
+# READELF: readelf program path
+readelf.string_section() {
+ local file="${1}" section="${2}"
+ LC_ALL=C "${READELF}" --string-dump "${section}" "${file}" 2>/dev/null
+}
diff --git a/support/scripts/shell/sdk.sh b/support/scripts/shell/sdk.sh
index b2f699c..6cb3bbe 100644
--- a/support/scripts/shell/sdk.sh
+++ b/support/scripts/shell/sdk.sh
@@ -19,9 +19,16 @@
# This module defines the following functions:
# sdk.compute_relative_path
# sdk.compute_rpath
+# sdk.check_host_leaks
+#
+# This module is sensitive to the following environment variables:
+# READELF
source.declare_module sdk
+source.load_module utils
+source.load_module readelf
+
# sdk.compute_relative_path basedir path start
#
# Computes and prints the relative path between $start and $path within $basedir.
diff --git a/support/scripts/shell/utils.sh b/support/scripts/shell/utils.sh
index 9e9aaab..eac3158 100644
--- a/support/scripts/shell/utils.sh
+++ b/support/scripts/shell/utils.sh
@@ -19,6 +19,7 @@
# This module defines the following functions:
# utils.list_has
# utils.list_reduce
+# utils.assert_absolute_canonical_path
source.declare_module utils
@@ -58,3 +59,16 @@ utils.list_reduce() {
echo ${lout[@]}
}
+
+# utils.assert_absolute_canonical_path path
+#
+# Returns 0 if 'path' is the absolute canonical path, returns non-0
+# otherwise.
+#
+# If the test failed, an error message will be issued to stderr.
+#
+# path : path to be tested
+utils.assert_absolute_canonical_path() {
+ test "$(readlink -f "${1}")" = "${1}" ||
+ log.error "%s is not the absolute canonical path.\n" "${1}" >&2
+}
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 09/11] core: add check-leaks-in-{target, host, staging} targets
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
` (7 preceding siblings ...)
2016-04-22 20:50 ` [Buildroot] [PATCH v9 08/11] support/scripts: add check-host-leaks script + all needed helpers Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 10/11] support/scripts/check-host-leaks: add option to classify leaks Samuel Martin
` (2 subsequent siblings)
11 siblings, 0 replies; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
This new targets allow identifying what need to be fixed to fulfill the
ultimate goal of removing any trace of the host in the target filesystem
or the SDK.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- none
changes v7->v8:
- update the check-host-leak script call
changes v6->v7:
- none
changes v5->v6:
- new patch
---
Makefile | 15 +++++++++++++++
package/pkg-utils.mk | 5 +++++
2 files changed, 20 insertions(+)
diff --git a/Makefile b/Makefile
index e92826f..a986c49 100644
--- a/Makefile
+++ b/Makefile
@@ -651,6 +651,21 @@ endef
TARGET_FINALIZE_HOOKS += PURGE_LOCALES
endif
+# Checks for build machine leaks into target filesystem, SDK
+check-leaks-in-target:
+ @$(call MESSAGE,Checking leaks in the target)
+ $(Q)$(call check-for-build-machine-leaks-in,$(TARGET_DIR))
+
+check-leaks-in-host:
+ @$(call MESSAGE,Checking leaks in the host)
+ $(Q)$(call check-for-build-machine-leaks-in,$(HOST_DIR),--exclude=$(STAGING_DIR))
+
+check-leaks-in-staging:
+ @$(call MESSAGE,Checking leaks in the staging)
+ $(Q)$(call check-for-build-machine-leaks-in,$(STAGING_DIR))
+
+check-leaks: check-leaks-in-target check-leaks-in-host check-leaks-in-staging
+
# Function sanitizing target/staging ELF files' RPATH.
# i.e. it removes paths pointing to the staging or build location from the ELF
# files' RPATH.
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index f88313a..8b80028 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -150,3 +150,8 @@ define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET}
mkdir -p $(LICENSE_FILES_DIR_$(4))/$(1)/$(dir $(2)) && \
cp $(3) $(LICENSE_FILES_DIR_$(4))/$(1)/$(2)
endef
+
+define check-for-build-machine-leaks-in
+ $(TOPDIR)/support/scripts/check-host-leaks $(1) \
+ $(TOPDIR) $(BASE_DIR) $(HOST_DIR) $(STAGING_DIR) $(2)
+endef
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 10/11] support/scripts/check-host-leaks: add option to classify leaks
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
` (8 preceding siblings ...)
2016-04-22 20:50 ` [Buildroot] [PATCH v9 09/11] core: add check-leaks-in-{target, host, staging} targets Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-04-22 21:31 ` Yegor Yefremov
2016-04-22 20:50 ` [Buildroot] [PATCH v9 11/11] docs/manual: document how to debug shell script Samuel Martin
2016-06-01 5:26 ` [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
11 siblings, 1 reply; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
This option can be useful to figure out patterns in the leaks.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- update commit log
changes v7->v8:
- new patch. Move this feature in its own change-set as it seems a bit
controversial.
- make leaks classification optional
---
support/scripts/check-host-leaks | 59 ++++++++++++++++++++++++++++++++++++++--
support/scripts/shell/readelf.sh | 1 -
2 files changed, 57 insertions(+), 3 deletions(-)
diff --git a/support/scripts/check-host-leaks b/support/scripts/check-host-leaks
index 9f363af..bd92493 100755
--- a/support/scripts/check-host-leaks
+++ b/support/scripts/check-host-leaks
@@ -45,6 +45,12 @@ Options:
Can be set more than once.
An excluded path must be an absolute canonical path.
+ --classify-leaks
+ Enables leak classification. Categories are defined according to
+ the matched files' type or ELF kinds.
+ For ELF files, the sections containing the leaks will displayed.
+ Enabling this option substantially increases the execution time
+ of the script.
EOF
return ${ret}
@@ -56,9 +62,51 @@ source.load_module log
source.load_module utils
source.load_module sdk
+classify_leak() {
+ local f="${1}" regexp="${2}"
+ local leak
+ if test -h "${f}" ; then leak="symlink"
+ elif readelf.is_elf "${f}" ; then
+ if readelf.is_elf_executable "${f}" ; then leak="ELF/exe"
+ elif readelf.is_elf_shared_object "${f}" ; then leak="ELF/*.so"
+ elif readelf.is_elf_static_library "${f}" ; then leak="ELF/*.a"
+ elif readelf.is_elf_object "${f}" ; then
+ case "${f}" in
+ *.ko) leak="ELF/*.ko" ;;
+ *) leak="ELF/*.o" ;;
+ esac
+ else leak="ELF/?"
+ fi
+ local section
+ local sections=()
+ for section in $(readelf.list_sections "${f}") ; do
+ if readelf.string_section "${f}" "${section}" |
+ grep -qaE "${regexp}" ; then
+ if ! utils.list_has ${section} ${sections[@]} ; then
+ sections+=( "${section}" )
+ fi
+ fi
+ done
+ leak="${leak} [${sections[*]}]"
+ else
+ case "${f}" in
+ *-config) leak="*-config script" ;;
+ *.la) leak="*.la" ;;
+ *.pc) leak="*.pc" ;;
+ *.py) leak="*.py" ;;
+ *.pyc|*.pyo) leak="*.py[co]" ;;
+ esac
+ fi
+ if test -z "${leak}" ; then
+ leak="? [$(file -z "${f}" | sed -e 's/.*: //')]"
+ fi
+ printf "${leak}"
+}
+
+
main() {
local root_dir
- local class_leaks
+ local classify_leaks
local excluded=()
local leak_paths=()
while test ${#} -gt 0 ; do
@@ -67,6 +115,8 @@ main() {
;;
--exclude) shift ; excluded+=( "${1}" )
;;
+ --classify-leaks) classify_leaks=y
+ ;;
-h|--help)
usage
exit 0
@@ -119,7 +169,12 @@ main() {
grep -raEl "${re_leaks}" . |
sed -re "${re_excl} ; s:^\.:${root_dir}:" |
while read f ; do
- printf "%s\n" "${f}"
+ if test -n "${classify_leaks}" ; then
+ local leak="$(classify_leak "${f}" "${regexp}")"
+ printf "%-70s : %-120s\n" "${leak}" "${f}"
+ else
+ printf "%s\n" "${f}"
+ fi
done | sort
popd >/dev/null
}
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
index b23666e..a7d790e 100644
--- a/support/scripts/shell/readelf.sh
+++ b/support/scripts/shell/readelf.sh
@@ -54,7 +54,6 @@ source.declare_module readelf
# environment:
# READELF: readelf program path
readelf._match_elf_regexp() {
- log._trace_func
local regexp="${1}" file="${2}"
LC_ALL=C ${READELF} -h "${file}" 2>/dev/null | grep -qE "${regexp}"
}
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 11/11] docs/manual: document how to debug shell script
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
` (9 preceding siblings ...)
2016-04-22 20:50 ` [Buildroot] [PATCH v9 10/11] support/scripts/check-host-leaks: add option to classify leaks Samuel Martin
@ 2016-04-22 20:50 ` Samuel Martin
2016-04-22 21:25 ` Yegor Yefremov
2016-06-01 5:26 ` [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
11 siblings, 1 reply; 22+ messages in thread
From: Samuel Martin @ 2016-04-22 20:50 UTC (permalink / raw)
To: buildroot
The method describes here is only true for scripts using the shell
modules from support/scripts/shell/*.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---
changes v8->v9:
- none
changes v7->v8:
- put instrumentation in a subsection (Arnout)
- minor rewording
changes v6->v7:
- new patch
---
docs/manual/debugging-buildroot.txt | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/docs/manual/debugging-buildroot.txt b/docs/manual/debugging-buildroot.txt
index f575fc4..b809c2d 100644
--- a/docs/manual/debugging-buildroot.txt
+++ b/docs/manual/debugging-buildroot.txt
@@ -5,6 +5,8 @@
== Debugging Buildroot
+=== Running scripts before and after each build step
+
It is possible to instrument the steps +Buildroot+ does when building
packages. Define the variable +BR2_INSTRUMENTATION_SCRIPTS+ to contain
the path of one or more scripts (or other executables), in a
@@ -41,3 +43,18 @@ The script has access to the following variables:
- +BINARIES_DIR+: the place where all binary files (aka images) are
stored
- +BASE_DIR+: the base output directory
+
+=== Debugging shell scripts in Buildroot
+
+For shell scripts using the _shell modules_ (available in
++support/scripts/shell/+), an environment variable is available to trace
+the scripts:
+
+ - +SHELL_DEBUG+ adjustes the debug level, from 0 to 3 (only errors to
+ all debug messages)
+
+To enable these debug outputs:
+
+----
+SHELL_DEBUG=3 make
+----
--
2.8.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 11/11] docs/manual: document how to debug shell script
2016-04-22 20:50 ` [Buildroot] [PATCH v9 11/11] docs/manual: document how to debug shell script Samuel Martin
@ 2016-04-22 21:25 ` Yegor Yefremov
0 siblings, 0 replies; 22+ messages in thread
From: Yegor Yefremov @ 2016-04-22 21:25 UTC (permalink / raw)
To: buildroot
On Fri, Apr 22, 2016 at 10:50 PM, Samuel Martin <s.martin49@gmail.com> wrote:
> The method describes here is only true for scripts using the shell
s/describes/described ?
> modules from support/scripts/shell/*.
>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
>
> ---
> changes v8->v9:
> - none
>
> changes v7->v8:
> - put instrumentation in a subsection (Arnout)
> - minor rewording
>
> changes v6->v7:
> - new patch
> ---
> docs/manual/debugging-buildroot.txt | 17 +++++++++++++++++
> 1 file changed, 17 insertions(+)
>
> diff --git a/docs/manual/debugging-buildroot.txt b/docs/manual/debugging-buildroot.txt
> index f575fc4..b809c2d 100644
> --- a/docs/manual/debugging-buildroot.txt
> +++ b/docs/manual/debugging-buildroot.txt
> @@ -5,6 +5,8 @@
>
> == Debugging Buildroot
>
> +=== Running scripts before and after each build step
> +
> It is possible to instrument the steps +Buildroot+ does when building
> packages. Define the variable +BR2_INSTRUMENTATION_SCRIPTS+ to contain
> the path of one or more scripts (or other executables), in a
> @@ -41,3 +43,18 @@ The script has access to the following variables:
> - +BINARIES_DIR+: the place where all binary files (aka images) are
> stored
> - +BASE_DIR+: the base output directory
> +
> +=== Debugging shell scripts in Buildroot
> +
> +For shell scripts using the _shell modules_ (available in
> ++support/scripts/shell/+), an environment variable is available to trace
> +the scripts:
> +
> + - +SHELL_DEBUG+ adjustes the debug level, from 0 to 3 (only errors to
> + all debug messages)
> +
> +To enable these debug outputs:
> +
> +----
> +SHELL_DEBUG=3 make
> +----
> --
> 2.8.0
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 10/11] support/scripts/check-host-leaks: add option to classify leaks
2016-04-22 20:50 ` [Buildroot] [PATCH v9 10/11] support/scripts/check-host-leaks: add option to classify leaks Samuel Martin
@ 2016-04-22 21:31 ` Yegor Yefremov
0 siblings, 0 replies; 22+ messages in thread
From: Yegor Yefremov @ 2016-04-22 21:31 UTC (permalink / raw)
To: buildroot
On Fri, Apr 22, 2016 at 10:50 PM, Samuel Martin <s.martin49@gmail.com> wrote:
> This option can be useful to figure out patterns in the leaks.
>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
>
> ---
> changes v8->v9:
> - update commit log
>
> changes v7->v8:
> - new patch. Move this feature in its own change-set as it seems a bit
> controversial.
> - make leaks classification optional
> ---
> support/scripts/check-host-leaks | 59 ++++++++++++++++++++++++++++++++++++++--
> support/scripts/shell/readelf.sh | 1 -
> 2 files changed, 57 insertions(+), 3 deletions(-)
>
> diff --git a/support/scripts/check-host-leaks b/support/scripts/check-host-leaks
> index 9f363af..bd92493 100755
> --- a/support/scripts/check-host-leaks
> +++ b/support/scripts/check-host-leaks
> @@ -45,6 +45,12 @@ Options:
> Can be set more than once.
> An excluded path must be an absolute canonical path.
>
> + --classify-leaks
> + Enables leak classification. Categories are defined according to
> + the matched files' type or ELF kinds.
> + For ELF files, the sections containing the leaks will displayed.
s/will displayed/will be displayed
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
> + Enabling this option substantially increases the execution time
> + of the script.
>
> EOF
> return ${ret}
> @@ -56,9 +62,51 @@ source.load_module log
> source.load_module utils
> source.load_module sdk
>
> +classify_leak() {
> + local f="${1}" regexp="${2}"
> + local leak
> + if test -h "${f}" ; then leak="symlink"
> + elif readelf.is_elf "${f}" ; then
> + if readelf.is_elf_executable "${f}" ; then leak="ELF/exe"
> + elif readelf.is_elf_shared_object "${f}" ; then leak="ELF/*.so"
> + elif readelf.is_elf_static_library "${f}" ; then leak="ELF/*.a"
> + elif readelf.is_elf_object "${f}" ; then
> + case "${f}" in
> + *.ko) leak="ELF/*.ko" ;;
> + *) leak="ELF/*.o" ;;
> + esac
> + else leak="ELF/?"
> + fi
> + local section
> + local sections=()
> + for section in $(readelf.list_sections "${f}") ; do
> + if readelf.string_section "${f}" "${section}" |
> + grep -qaE "${regexp}" ; then
> + if ! utils.list_has ${section} ${sections[@]} ; then
> + sections+=( "${section}" )
> + fi
> + fi
> + done
> + leak="${leak} [${sections[*]}]"
> + else
> + case "${f}" in
> + *-config) leak="*-config script" ;;
> + *.la) leak="*.la" ;;
> + *.pc) leak="*.pc" ;;
> + *.py) leak="*.py" ;;
> + *.pyc|*.pyo) leak="*.py[co]" ;;
> + esac
> + fi
> + if test -z "${leak}" ; then
> + leak="? [$(file -z "${f}" | sed -e 's/.*: //')]"
> + fi
> + printf "${leak}"
> +}
> +
> +
> main() {
> local root_dir
> - local class_leaks
> + local classify_leaks
> local excluded=()
> local leak_paths=()
> while test ${#} -gt 0 ; do
> @@ -67,6 +115,8 @@ main() {
> ;;
> --exclude) shift ; excluded+=( "${1}" )
> ;;
> + --classify-leaks) classify_leaks=y
> + ;;
> -h|--help)
> usage
> exit 0
> @@ -119,7 +169,12 @@ main() {
> grep -raEl "${re_leaks}" . |
> sed -re "${re_excl} ; s:^\.:${root_dir}:" |
> while read f ; do
> - printf "%s\n" "${f}"
> + if test -n "${classify_leaks}" ; then
> + local leak="$(classify_leak "${f}" "${regexp}")"
> + printf "%-70s : %-120s\n" "${leak}" "${f}"
> + else
> + printf "%s\n" "${f}"
> + fi
> done | sort
> popd >/dev/null
> }
> diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
> index b23666e..a7d790e 100644
> --- a/support/scripts/shell/readelf.sh
> +++ b/support/scripts/shell/readelf.sh
> @@ -54,7 +54,6 @@ source.declare_module readelf
> # environment:
> # READELF: readelf program path
> readelf._match_elf_regexp() {
> - log._trace_func
> local regexp="${1}" file="${2}"
> LC_ALL=C ${READELF} -h "${file}" 2>/dev/null | grep -qE "${regexp}"
> }
> --
> 2.8.0
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
` (10 preceding siblings ...)
2016-04-22 20:50 ` [Buildroot] [PATCH v9 11/11] docs/manual: document how to debug shell script Samuel Martin
@ 2016-06-01 5:26 ` Samuel Martin
2016-06-08 21:12 ` Samuel Martin
11 siblings, 1 reply; 22+ messages in thread
From: Samuel Martin @ 2016-06-01 5:26 UTC (permalink / raw)
To: buildroot
Ping?
On Apr 22, 2016 10:50 PM, "Samuel Martin" <s.martin49@gmail.com> wrote:
>
> Hi all,
>
>
> Here is yet another round of the series aiming improving the
relocatability
> of the SDK built by Buildroot.
>
> Still slowly reducing the number of patch in the series, as they got
merged...
> For this round, just fixes from comments, nothing really noticeable,
except
> maybe the move of the HOST_DIR RPATH sanitation.
>
> For further details, check the per-commit changelog in each commit log.
>
>
> Patches 1 and 2:
> Fix the O variable definition.
>
> Patches 3 to 5:
> Add the fix-rpath script, then run it on the Buildroot host, target and
> staging trees.
>
> Patch 6:
> Clean-up speex package WRT its RPATH hook.
>
> Patch 7:
> Update and fix the check-host-rpath to re-use the shell helpers.
>
> Patches 8 to 10:
> Add means to track build machine leaks into the host, target and staging
> trees. These patches will allow to identify what remains to be fixed WRT
> build machine leaks.
>
> Patches 11:
> Update documentation about how to enable/adjust log level on script
using
> the shell modules introduced in patch 3/11.
>
>
> Regards,
> Samuel
>
>
> Samuel Martin (11):
> core: split variables definition related to in/out-of-tree build from
> O itself
> core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
> path
> support/scripts: add fix-rpath script + a bunch of helpers
> core: sanitize HOST_DIR at the very end of the build
> core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
> TARGET_FINALIZE_HOOKS
> package/speex: remove no longer needed hook
> support/scripts: update check-host-rpath to use the shell helpers
> support/scripts: add check-host-leaks script + all needed helpers
> core: add check-leaks-in-{target,host,staging} targets
> support/scripts/check-host-leaks: add option to classify leaks
> docs/manual: document how to debug shell script
>
> Makefile | 126 ++++++++++++---
> docs/manual/debugging-buildroot.txt | 17 ++
> package/pkg-utils.mk | 5 +
> package/speex/speex.mk | 7 -
> support/scripts/check-host-leaks | 181 +++++++++++++++++++++
> support/scripts/check-host-rpath | 73 ++++-----
> support/scripts/fix-rpath | 116 ++++++++++++++
> support/scripts/shell/log.sh | 61 +++++++
> support/scripts/shell/patchelf.sh | 178 +++++++++++++++++++++
> support/scripts/shell/readelf.sh | 306
++++++++++++++++++++++++++++++++++++
> support/scripts/shell/sdk.sh | 75 +++++++++
> support/scripts/shell/source.sh | 77 +++++++++
> support/scripts/shell/utils.sh | 74 +++++++++
> 13 files changed, 1223 insertions(+), 73 deletions(-)
> create mode 100755 support/scripts/check-host-leaks
> create mode 100755 support/scripts/fix-rpath
> create mode 100644 support/scripts/shell/log.sh
> create mode 100644 support/scripts/shell/patchelf.sh
> create mode 100644 support/scripts/shell/readelf.sh
> create mode 100644 support/scripts/shell/sdk.sh
> create mode 100644 support/scripts/shell/source.sh
> create mode 100644 support/scripts/shell/utils.sh
>
> --
> 2.8.0
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160601/c47c2a90/attachment.html>
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing
2016-06-01 5:26 ` [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
@ 2016-06-08 21:12 ` Samuel Martin
2016-06-16 4:33 ` Samuel Martin
0 siblings, 1 reply; 22+ messages in thread
From: Samuel Martin @ 2016-06-08 21:12 UTC (permalink / raw)
To: buildroot
up!
and the shortcuts:
http://patchwork.ozlabs.org/patch/613853/
http://patchwork.ozlabs.org/patch/613854/
http://patchwork.ozlabs.org/patch/613855/
http://patchwork.ozlabs.org/patch/613856/
http://patchwork.ozlabs.org/patch/613857/
http://patchwork.ozlabs.org/patch/613858/
http://patchwork.ozlabs.org/patch/613860/
http://patchwork.ozlabs.org/patch/613859/
http://patchwork.ozlabs.org/patch/613862/
http://patchwork.ozlabs.org/patch/613861/
http://patchwork.ozlabs.org/patch/613863/
Happy review! ;-)
On Wed, Jun 1, 2016 at 7:26 AM, Samuel Martin <s.martin49@gmail.com> wrote:
> Ping?
>
> On Apr 22, 2016 10:50 PM, "Samuel Martin" <s.martin49@gmail.com> wrote:
>
>>
>> Hi all,
>>
>>
>> Here is yet another round of the series aiming improving the
>> relocatability
>> of the SDK built by Buildroot.
>>
>> Still slowly reducing the number of patch in the series, as they got
>> merged...
>> For this round, just fixes from comments, nothing really noticeable,
>> except
>> maybe the move of the HOST_DIR RPATH sanitation.
>>
>> For further details, check the per-commit changelog in each commit log.
>>
>>
>> Patches 1 and 2:
>> Fix the O variable definition.
>>
>> Patches 3 to 5:
>> Add the fix-rpath script, then run it on the Buildroot host, target and
>> staging trees.
>>
>> Patch 6:
>> Clean-up speex package WRT its RPATH hook.
>>
>> Patch 7:
>> Update and fix the check-host-rpath to re-use the shell helpers.
>>
>> Patches 8 to 10:
>> Add means to track build machine leaks into the host, target and staging
>> trees. These patches will allow to identify what remains to be fixed WRT
>> build machine leaks.
>>
>> Patches 11:
>> Update documentation about how to enable/adjust log level on script
>> using
>> the shell modules introduced in patch 3/11.
>>
>>
>> Regards,
>> Samuel
>>
>>
>> Samuel Martin (11):
>> core: split variables definition related to in/out-of-tree build from
>> O itself
>> core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
>> path
>> support/scripts: add fix-rpath script + a bunch of helpers
>> core: sanitize HOST_DIR at the very end of the build
>> core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
>> TARGET_FINALIZE_HOOKS
>> package/speex: remove no longer needed hook
>> support/scripts: update check-host-rpath to use the shell helpers
>> support/scripts: add check-host-leaks script + all needed helpers
>> core: add check-leaks-in-{target,host,staging} targets
>> support/scripts/check-host-leaks: add option to classify leaks
>> docs/manual: document how to debug shell script
>>
>> Makefile | 126 ++++++++++++---
>> docs/manual/debugging-buildroot.txt | 17 ++
>> package/pkg-utils.mk | 5 +
>> package/speex/speex.mk | 7 -
>> support/scripts/check-host-leaks | 181 +++++++++++++++++++++
>> support/scripts/check-host-rpath | 73 ++++-----
>> support/scripts/fix-rpath | 116 ++++++++++++++
>> support/scripts/shell/log.sh | 61 +++++++
>> support/scripts/shell/patchelf.sh | 178 +++++++++++++++++++++
>> support/scripts/shell/readelf.sh | 306
>> ++++++++++++++++++++++++++++++++++++
>> support/scripts/shell/sdk.sh | 75 +++++++++
>> support/scripts/shell/source.sh | 77 +++++++++
>> support/scripts/shell/utils.sh | 74 +++++++++
>> 13 files changed, 1223 insertions(+), 73 deletions(-)
>> create mode 100755 support/scripts/check-host-leaks
>> create mode 100755 support/scripts/fix-rpath
>> create mode 100644 support/scripts/shell/log.sh
>> create mode 100644 support/scripts/shell/patchelf.sh
>> create mode 100644 support/scripts/shell/readelf.sh
>> create mode 100644 support/scripts/shell/sdk.sh
>> create mode 100644 support/scripts/shell/source.sh
>> create mode 100644 support/scripts/shell/utils.sh
>>
>> --
>> 2.8.0
>>
--
Samuel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing
2016-06-08 21:12 ` Samuel Martin
@ 2016-06-16 4:33 ` Samuel Martin
0 siblings, 0 replies; 22+ messages in thread
From: Samuel Martin @ 2016-06-16 4:33 UTC (permalink / raw)
To: buildroot
up!
Any feedback (review/comment/tests/etc) is more than welcome! ;-)
On Wed, Jun 8, 2016 at 11:12 PM, Samuel Martin <s.martin49@gmail.com> wrote:
> up!
>
> and the shortcuts:
> http://patchwork.ozlabs.org/patch/613853/
> http://patchwork.ozlabs.org/patch/613854/
> http://patchwork.ozlabs.org/patch/613855/
> http://patchwork.ozlabs.org/patch/613856/
> http://patchwork.ozlabs.org/patch/613857/
> http://patchwork.ozlabs.org/patch/613858/
> http://patchwork.ozlabs.org/patch/613860/
> http://patchwork.ozlabs.org/patch/613859/
> http://patchwork.ozlabs.org/patch/613862/
> http://patchwork.ozlabs.org/patch/613861/
> http://patchwork.ozlabs.org/patch/613863/
>
> Happy review! ;-)
>
> On Wed, Jun 1, 2016 at 7:26 AM, Samuel Martin <s.martin49@gmail.com> wrote:
>> Ping?
>>
>> On Apr 22, 2016 10:50 PM, "Samuel Martin" <s.martin49@gmail.com> wrote:
>>
>>>
>>> Hi all,
>>>
>>>
>>> Here is yet another round of the series aiming improving the
>>> relocatability
>>> of the SDK built by Buildroot.
>>>
>>> Still slowly reducing the number of patch in the series, as they got
>>> merged...
>>> For this round, just fixes from comments, nothing really noticeable,
>>> except
>>> maybe the move of the HOST_DIR RPATH sanitation.
>>>
>>> For further details, check the per-commit changelog in each commit log.
>>>
>>>
>>> Patches 1 and 2:
>>> Fix the O variable definition.
>>>
>>> Patches 3 to 5:
>>> Add the fix-rpath script, then run it on the Buildroot host, target and
>>> staging trees.
>>>
>>> Patch 6:
>>> Clean-up speex package WRT its RPATH hook.
>>>
>>> Patch 7:
>>> Update and fix the check-host-rpath to re-use the shell helpers.
>>>
>>> Patches 8 to 10:
>>> Add means to track build machine leaks into the host, target and staging
>>> trees. These patches will allow to identify what remains to be fixed WRT
>>> build machine leaks.
>>>
>>> Patches 11:
>>> Update documentation about how to enable/adjust log level on script
>>> using
>>> the shell modules introduced in patch 3/11.
>>>
>>>
>>> Regards,
>>> Samuel
>>>
>>>
>>> Samuel Martin (11):
>>> core: split variables definition related to in/out-of-tree build from
>>> O itself
>>> core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
>>> path
>>> support/scripts: add fix-rpath script + a bunch of helpers
>>> core: sanitize HOST_DIR at the very end of the build
>>> core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
>>> TARGET_FINALIZE_HOOKS
>>> package/speex: remove no longer needed hook
>>> support/scripts: update check-host-rpath to use the shell helpers
>>> support/scripts: add check-host-leaks script + all needed helpers
>>> core: add check-leaks-in-{target,host,staging} targets
>>> support/scripts/check-host-leaks: add option to classify leaks
>>> docs/manual: document how to debug shell script
>>>
>>> Makefile | 126 ++++++++++++---
>>> docs/manual/debugging-buildroot.txt | 17 ++
>>> package/pkg-utils.mk | 5 +
>>> package/speex/speex.mk | 7 -
>>> support/scripts/check-host-leaks | 181 +++++++++++++++++++++
>>> support/scripts/check-host-rpath | 73 ++++-----
>>> support/scripts/fix-rpath | 116 ++++++++++++++
>>> support/scripts/shell/log.sh | 61 +++++++
>>> support/scripts/shell/patchelf.sh | 178 +++++++++++++++++++++
>>> support/scripts/shell/readelf.sh | 306
>>> ++++++++++++++++++++++++++++++++++++
>>> support/scripts/shell/sdk.sh | 75 +++++++++
>>> support/scripts/shell/source.sh | 77 +++++++++
>>> support/scripts/shell/utils.sh | 74 +++++++++
>>> 13 files changed, 1223 insertions(+), 73 deletions(-)
>>> create mode 100755 support/scripts/check-host-leaks
>>> create mode 100755 support/scripts/fix-rpath
>>> create mode 100644 support/scripts/shell/log.sh
>>> create mode 100644 support/scripts/shell/patchelf.sh
>>> create mode 100644 support/scripts/shell/readelf.sh
>>> create mode 100644 support/scripts/shell/sdk.sh
>>> create mode 100644 support/scripts/shell/source.sh
>>> create mode 100644 support/scripts/shell/utils.sh
>>>
>>> --
>>> 2.8.0
>>>
>
>
>
> --
> Samuel
--
Samuel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself
2016-04-22 20:50 ` [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself Samuel Martin
@ 2016-06-26 22:27 ` Arnout Vandecappelle
2016-06-27 4:33 ` Samuel Martin
2016-07-05 17:02 ` Yann E. MORIN
1 sibling, 1 reply; 22+ messages in thread
From: Arnout Vandecappelle @ 2016-06-26 22:27 UTC (permalink / raw)
To: buildroot
On 22-04-16 22:50, Samuel Martin wrote:
> This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from
> the presence of the O variable in the command line.
>
> Now, the condition used to set these variables is the value of O itself.
>
> This change is a preparatory work since the O definition will need to
> be moved around when we will make Buildroot run with absolute canonical
> paths for both its root directory and the output location.
> This will be addressed in a follow-up patch.
>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Tested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(finally :-) but don't count on me being able to finish the series tonight)
Regards,
Arnout
>
> ---
> changes v8->v9:
> - none
>
> changes v7->v8:
> - new patch
> ---
> Makefile | 14 +++++++++++---
> 1 file changed, 11 insertions(+), 3 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 0e4beb2..3d86c9b 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -110,10 +110,11 @@ comma := ,
> empty :=
> space := $(empty) $(empty)
>
> +# Set O variable if not already done on the command line;
> +# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
> +# build by preventing it from being forwarded to sub-make calls.
> ifneq ("$(origin O)", "command line")
> O := output
> -CONFIG_DIR := $(TOPDIR)
> -NEED_WRAPPER =
> else
> # other packages might also support Linux-style out of tree builds
> # with the O=<dir> syntax (E.G. BusyBox does). As make automatically
> @@ -126,9 +127,16 @@ MAKEOVERRIDES =
> # Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
> # To really make O go away, we have to override it.
> override O := $(O)
> -CONFIG_DIR := $(O)
> # we need to pass O= everywhere we call back into the toplevel makefile
> EXTRAMAKEARGS = O=$(O)
> +endif
> +
> +# Set variables related to in-tree or out-of-tree build.
> +ifeq ($(O),output)
> +CONFIG_DIR := $(TOPDIR)
> +NEED_WRAPPER =
> +else
> +CONFIG_DIR := $(O)
> NEED_WRAPPER = y
> endif
>
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 02/11] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path
2016-04-22 20:50 ` [Buildroot] [PATCH v9 02/11] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path Samuel Martin
@ 2016-06-26 23:08 ` Arnout Vandecappelle
0 siblings, 0 replies; 22+ messages in thread
From: Arnout Vandecappelle @ 2016-06-26 23:08 UTC (permalink / raw)
To: buildroot
On 22-04-16 22:50, Samuel Martin wrote:
> When $(CURDIR) or $(O) contain symlinks in their path, they can be
> resolved differently, depending on each package build-system (whether it
> uses the given paths or get the absolute canonical ones).
>
> This will make easier tracking down host machine paths leaking into the
^^^^To
> host, target or staging trees, the CURDIR and O variables are set to
> their absolute canonical paths.
>
> In order to recall the toplevel makefile with absolute canonical paths
> for $(CURDIR) and $(O), we need to:
> 1- Move the O variable definition out of any if-block; so they are
> always available.
> 2- Compute the absolute canonical paths for $(CURDIR) and $(O) that will
> be passed to the sub-make. This is achieved using the 'realpath' make
> primitive. However, some care must be taken when manipulating O:
> - the out-of-tree makefile wrapper happens a trailing "/.", we need
^^^^^^^appends
> to strip this part away to not break the comparison driving the
> sub-make call;
This is just to avoid another level of recursion, right? It doesn't seem to
work, see below.
> - according to [1,2], realpath returns an empty string in case of
> non-existing entry. So, to avoid passing an empty O= variable to
> sub-make, it is necessary to define the output directory and create
> it prior to call realpath on it (because on the first invocation,
> $(O) usually does not yet exists), hence the trick doing the mkdir
> right before calling realpath.
> 3- Update EXTRAMAKEARGS with the absobulte canonical $(O) and use it
^^^^^^^^^absolute
> when call recalling the toplevel makefile with umask and paths
> correctly set.
> 4- Lastly, update the condition for setting the CONFIG_DIR and
> NEED_WRAPPER variables.
>
> Notes:
> * This change takes care of the makefile wrapper installed in $(O) to
> avoid unneeded make recursion.
> * Now, only $(O) is strip away from MAKEOVERRIDES whatever the build is
^^^^^stripped ^^^^^^^^whereever
> done in- or out-of-tree (i.o.w. without or with O set in the command
> line); wheares is the previous implementation, all variables set on
> the command line were stripped away only in the case of out-of-tree
> build.
I don't see how this change is related to the rest, actually.
Maybe this patch is doing a little too much in one go...
There is a lot of stuff being moved around by this patch, and very little
actual changes. It would help to separate that to start with - now the diff is a
little large. You can make such a patch that has no use at all except for making
the next patch simpler.
>
> [1] https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html
> [2] http://man7.org/linux/man-pages/man3/realpath.3.html
>
> Reported-by: Matthew Weber <matt@thewebers.ws>
> Cc: Matthew Weber <matt@thewebers.ws>
> Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
>
> ---
[snip]
> diff --git a/Makefile b/Makefile
> index 3d86c9b..a05b9e1 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -24,18 +24,68 @@
> # You shouldn't need to mess with anything beyond this point...
> #--------------------------------------------------------------
>
> -# Trick for always running with a fixed umask
> +# Set O variable if not already done on the command line;
> +# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
> +# build by preventing it from being forwarded to sub-make calls.
> +ifneq ("$(origin O)", "command line")
> +O := $(CURDIR)/output
> +else
> +# Other packages might also support Linux-style out of tree builds
> +# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
> +# forwards command line variable definitions those packages get very
> +# confused. Fix this by telling make to not do so, only for O=..., but
> +# keep all others (such as BR2_EXTERNAL, BR2_DL_DIR, etc).
> +MAKEOVERRIDES := $(filter-out O=%,$(MAKEOVERRIDES))
> +# Strangely enough O is still passed to submakes with MAKEOVERRIDES
> +# (with make 3.81 atleast), the only thing that changes is the output
> +# of the origin function (command line -> environment).
> +# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
> +# To really make O go away, we have to override it.
> +override O := $(O)
> +endif
> +
> +# Check if the current Buildroot execution meets all the pre-requisites.
> +# If they are not met, Buildroot will actually do its job in a sub-make meeting
> +# its pre-requisites, which are:
> +# 1- Permissive enough umask:
> +# Wrong or too restrictive umask will prevent Buildroot and packages from
> +# creating files and directories.
> +# 2- Absolute canonical CWD (i.e. $(CURDIR)):
> +# Otherwise, some packages will use CWD as-is, others will compute its
> +# absolute canonical path. This makes harder tracking and fixing host
> +# machine path leaks.
> +# 3- Absolute canonical output location (i.e. $(O)):
> +# For the same reason as the one for CWD.
> +
> +# Current state:
> +CUR_UMASK := $(shell umask)
We normally use = rather than := unless strictly necessary. Here it is not
strictly necessary, becauce CUR_UMASK is used only once anyway.
> +# Remove the trailing '/.' from $(O) as it can be added by the makefile wrapper
> +# installed in the $(O) directory.
> +O := $(patsubst %/.,%,$(O))
This actually doesn't work, because of the override above you need to use
override here as well. Well, things will still _work_, you just have a redundant
recursion level.
> +
> +# Buildroot requirements:
This comment (and the 'Current state' above) isn't very well placed here,
because there is a bit too much other stuff (comments etc.) intermingled. I
think it would be better to keep the corresponding options together, so put
REAL_O right below the last O replacement above, then REAL_CURDIR, and finally
both UMASKs. I'm just not sure where to put the EXTRAMAKEARGS.
> UMASK = 0022
> -ifneq ($(shell umask),$(UMASK))
> +REAL_CURDIR := $(realpath $(CURDIR))
Here also no real need to use := even though it's used twice, realpath is not
an expensive function.
> +# realpath needs the entry to exists, otherwise an empty string is returned.
> +REAL_O := $(shell mkdir -p $(O) >/dev/null 2>&1)$(realpath $(O))
Here the := is needed because of the mkdir.
I don't think redirecting the stderr is a good idea, because the error you get
from it is actually interesting. Instead, I think it's best to use the same
command as for BASE_DIR and just slap a $(realpath ...) around it. Also, we need
that check like for BASE_DIR to error out immediately if the mkdir failed. And
finally, the equivalent for BASE_DIR can be removed. Actually, the whole of
BASE_DIR can be removed - O is now exactly the same as BASE_DIR.
I propose to split off a separate patch that moves the mkdir from BASE_DIR up
here to REAL_O, and just sets BASE_DIR = $(REAL_O). Then a second patch can add
the recursion, and set BASE_DIR = $(O). And finally a third patch can remove
BASE_DIR completely. It's possible that that third patch gets rejected in the
end BTW.
Regards,
Arnout
> +
> +# Make sure O= is passed (with its absolute canonical path) everywhere the
> +# toplevel makefile is called back.
> +EXTRAMAKEARGS := O=$(REAL_O)
> +
> +# Check Buildroot execution pre-requisites here.
> +ifneq ($(CUR_UMASK):$(CURDIR):$(O),$(UMASK):$(REAL_CURDIR):$(REAL_O))
> .PHONY: _all $(MAKECMDGOALS)
>
> $(MAKECMDGOALS): _all
> @:
>
> _all:
> - @umask $(UMASK) && $(MAKE) --no-print-directory $(MAKECMDGOALS)
> + @umask $(UMASK) && \
> + $(MAKE) -C $(REAL_CURDIR) --no-print-directory \
> + $(MAKECMDGOALS) $(EXTRAMAKEARGS)
>
> -else # umask
> +else # umask / $(CURDIR) / $(O)
>
> # This is our default rule, so must come first
> all:
> @@ -110,30 +160,10 @@ comma := ,
> empty :=
> space := $(empty) $(empty)
>
> -# Set O variable if not already done on the command line;
> -# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
> -# build by preventing it from being forwarded to sub-make calls.
> -ifneq ("$(origin O)", "command line")
> -O := output
> -else
> -# other packages might also support Linux-style out of tree builds
> -# with the O=<dir> syntax (E.G. BusyBox does). As make automatically
> -# forwards command line variable definitions those packages get very
> -# confused. Fix this by telling make to not do so
> -MAKEOVERRIDES =
> -# strangely enough O is still passed to submakes with MAKEOVERRIDES
> -# (with make 3.81 atleast), the only thing that changes is the output
> -# of the origin function (command line -> environment).
> -# Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
> -# To really make O go away, we have to override it.
> -override O := $(O)
> -# we need to pass O= everywhere we call back into the toplevel makefile
> -EXTRAMAKEARGS = O=$(O)
> -endif
> -
> # Set variables related to in-tree or out-of-tree build.
> -ifeq ($(O),output)
> -CONFIG_DIR := $(TOPDIR)
> +# Here, both $(O) and $(CURDIR) are absolute canonical paths.
> +ifeq ($(O),$(CURDIR)/output)
> +CONFIG_DIR := $(CURDIR)
> NEED_WRAPPER =
> else
> CONFIG_DIR := $(O)
> @@ -1017,4 +1047,4 @@ include docs/manual/manual.mk
>
> .PHONY: $(noconfig_targets)
>
> -endif #umask
> +endif #umask / $(CURDIR) / $(O)
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself
2016-06-26 22:27 ` Arnout Vandecappelle
@ 2016-06-27 4:33 ` Samuel Martin
0 siblings, 0 replies; 22+ messages in thread
From: Samuel Martin @ 2016-06-27 4:33 UTC (permalink / raw)
To: buildroot
On Mon, Jun 27, 2016 at 12:27 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 22-04-16 22:50, Samuel Martin wrote:
>> This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from
>> the presence of the O variable in the command line.
>>
>> Now, the condition used to set these variables is the value of O itself.
>>
>> This change is a preparatory work since the O definition will need to
>> be moved around when we will make Buildroot run with absolute canonical
>> paths for both its root directory and the output location.
>> This will be addressed in a follow-up patch.
>>
>> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
>
> Tested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>
> (finally :-) but don't count on me being able to finish the series tonight)
Thanks, it's ok :)
Anyway, I think I'll have to resend this series after rebasing on top
of master (potential conflicts on Makefile due to some recent
patches).
Regards,
--
Samuel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself
2016-04-22 20:50 ` [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself Samuel Martin
2016-06-26 22:27 ` Arnout Vandecappelle
@ 2016-07-05 17:02 ` Yann E. MORIN
1 sibling, 0 replies; 22+ messages in thread
From: Yann E. MORIN @ 2016-07-05 17:02 UTC (permalink / raw)
To: buildroot
Samuel, All,
On 2016-04-22 22:50 +0200, Samuel Martin spake thusly:
> This change uncorrolates the CONFIG_DIR and NEED_WRAPPER definition from
> the presence of the O variable in the command line.
>
> Now, the condition used to set these variables is the value of O itself.
>
> This change is a preparatory work since the O definition will need to
> be moved around when we will make Buildroot run with absolute canonical
> paths for both its root directory and the output location.
> This will be addressed in a follow-up patch.
>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
This can probably go in now, without waiting for the rest of the series.
Regards,
Yann E. MORIN.
> ---
> changes v8->v9:
> - none
>
> changes v7->v8:
> - new patch
> ---
> Makefile | 14 +++++++++++---
> 1 file changed, 11 insertions(+), 3 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 0e4beb2..3d86c9b 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -110,10 +110,11 @@ comma := ,
> empty :=
> space := $(empty) $(empty)
>
> +# Set O variable if not already done on the command line;
> +# or avoid confusing packages that can use the O=<dir> syntax for out-of-tree
> +# build by preventing it from being forwarded to sub-make calls.
> ifneq ("$(origin O)", "command line")
> O := output
> -CONFIG_DIR := $(TOPDIR)
> -NEED_WRAPPER =
> else
> # other packages might also support Linux-style out of tree builds
> # with the O=<dir> syntax (E.G. BusyBox does). As make automatically
> @@ -126,9 +127,16 @@ MAKEOVERRIDES =
> # Unfortunately some packages don't look at origin (E.G. uClibc 0.9.31+)
> # To really make O go away, we have to override it.
> override O := $(O)
> -CONFIG_DIR := $(O)
> # we need to pass O= everywhere we call back into the toplevel makefile
> EXTRAMAKEARGS = O=$(O)
> +endif
> +
> +# Set variables related to in-tree or out-of-tree build.
> +ifeq ($(O),output)
> +CONFIG_DIR := $(TOPDIR)
> +NEED_WRAPPER =
> +else
> +CONFIG_DIR := $(O)
> NEED_WRAPPER = y
> endif
>
> --
> 2.8.0
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Buildroot] [PATCH v9 03/11] support/scripts: add fix-rpath script + a bunch of helpers
2016-04-22 20:50 ` [Buildroot] [PATCH v9 03/11] support/scripts: add fix-rpath script + a bunch of helpers Samuel Martin
@ 2016-07-07 5:23 ` Benjamin Kamath
0 siblings, 0 replies; 22+ messages in thread
From: Benjamin Kamath @ 2016-07-07 5:23 UTC (permalink / raw)
To: buildroot
Samuel Martin <s.martin49@...> writes:
>
> This commit introduces a fix-rpath shell-script able to scan a tree,
> detect ELF files, check their RPATH and fix it in a proper way.
>
> Along to the fix-rpath script, it also adds a bunch of shell helper
> functions grouped into modules. This will help writing scripts
handling
> RPATH and other things, while allowing to share and reuse these
> functions between scripts.
>
> These helpers are namespaced within the filename of the module in
which
> they are gathered.
>
> This change adds 6 modules:
> - source.sh: provides simple helper to easily source another module,
taking
> care of not sourcing again an already-sourced one;
> - log.sh: provides logging helpers;
> - utils.sh: provides simple functions to filter ELF files in a list;
> - readelf.sh: provides functions retrieving ELF details from a file;
> - patchelf.sh: provides function updating ELF files;
> - sdk.sh: provides RPATH computation functions.
>
> These 6 modules are used by the fix-rpath script.
> Follow-up patches will make some scripts leveraging these modules.
>
> Signed-off-by: Samuel Martin <s.martin49@...>
Tested-by: Benjamin Kamath <kamath.ben@gmail.com>
>
> ---
> changes v8->v9:
> - none
>
> changes v7->v8:
> - slightly rework log.error (return non-0 instead of exiting; exit
> decision is deferred to the caller)
>
> changes v6->v7 (includes changes started during BR dev. days):
> - log module:
> - debug env. var. renamed: DEBUG -> SHELL_DEBUG
> - trace now prints the message level
> - patchelf module:
> - get rid of XRPATH/XORIGIN stuff (Arnout)
> - set_rpath now handles ELF files with no .dynamic section
> - cleanup and minor fix in sanitize_rpath
> - readelf module:
> - move {filter,is}_elf* functions from utils to readelf modules
> - add list_sections and has_section functions (needed in
> sanitize_rpath)
> - sdk modules:
> - get rid of XRPATH/XORIGIN stuff (Arnout)
> - minor functions doc. fixes
> - source module:
> - move "Could not load module" error message in load_module (Arnout)
> - utils module:
> - move {filter,is}_elf* functions from utils to readelf modules
> - add list_has function
> - fix-rpath:
> - runs with "set -e" (exit on error)
> - improve help text
> - slight refactoring because of modules changes
>
> changes v5->v6:
> - fully rewritten in shell
>
> changes v4->v5:
> - add verbose support
> - rename shrink_rpath -> clear_rpath
> - add sanitize_rpath function
>
> changes v3->v4:
> - fix typos and license (Baruch)
>
> changes v2->v3:
> - no change
> ---
> support/scripts/fix-rpath | 116 +++++++++++++++++++++++++
> support/scripts/shell/log.sh | 61 +++++++++++++
> support/scripts/shell/patchelf.sh | 178
++++++++++++++++++++++++++++++++++++++
> support/scripts/shell/readelf.sh | 173
++++++++++++++++++++++++++++++++++++
> support/scripts/shell/sdk.sh | 68 +++++++++++++++
> support/scripts/shell/source.sh | 77 +++++++++++++++++
> support/scripts/shell/utils.sh | 60 +++++++++++++
> 7 files changed, 733 insertions(+)
> create mode 100755 support/scripts/fix-rpath
> create mode 100644 support/scripts/shell/log.sh
> create mode 100644 support/scripts/shell/patchelf.sh
> create mode 100644 support/scripts/shell/readelf.sh
> create mode 100644 support/scripts/shell/sdk.sh
> create mode 100644 support/scripts/shell/source.sh
> create mode 100644 support/scripts/shell/utils.sh
>
I tested the fix-rpath script on multiple toolchains, and it
successfully made them relocatable. One thing I did notice was that once
the TREE_ROOT was set and called with the script once, I could no longer
call the script to set a more specific TREE_ROOT, i.e. moving ${O} down
a folder. This isn't a huge deal, but did mean I needed to rebuild the
toolchain before packaging it the way I wanted to. Another option would
be trim automatically, so if the user selected a TREE_ROOT as
$(some_path)/foo, but there weren't any elfs until a depth of
$(some_path)/foo/bar/cow, automatically readjust the TREE_ROOT to be the
more specific path.
This is a minor issue though, and it was pretty handy to be able to make
relocatable host tools so I think functionally it is still very useful
as is.
Also there were quite a few spelling typos in the scripts themselves /
commit messages, so I'd run it through a quick spellcheck as well.
Cheers,
Ben
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2016-07-07 5:23 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-04-22 20:50 [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 01/11] core: split variables definition related to in/out-of-tree build from O itself Samuel Martin
2016-06-26 22:27 ` Arnout Vandecappelle
2016-06-27 4:33 ` Samuel Martin
2016-07-05 17:02 ` Yann E. MORIN
2016-04-22 20:50 ` [Buildroot] [PATCH v9 02/11] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path Samuel Martin
2016-06-26 23:08 ` Arnout Vandecappelle
2016-04-22 20:50 ` [Buildroot] [PATCH v9 03/11] support/scripts: add fix-rpath script + a bunch of helpers Samuel Martin
2016-07-07 5:23 ` Benjamin Kamath
2016-04-22 20:50 ` [Buildroot] [PATCH v9 04/11] core: sanitize HOST_DIR at the very end of the build Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 05/11] core: add {TARGET, STAGING}_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 06/11] package/speex: remove no longer needed hook Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 07/11] support/scripts: update check-host-rpath to use the shell helpers Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 08/11] support/scripts: add check-host-leaks script + all needed helpers Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 09/11] core: add check-leaks-in-{target, host, staging} targets Samuel Martin
2016-04-22 20:50 ` [Buildroot] [PATCH v9 10/11] support/scripts/check-host-leaks: add option to classify leaks Samuel Martin
2016-04-22 21:31 ` Yegor Yefremov
2016-04-22 20:50 ` [Buildroot] [PATCH v9 11/11] docs/manual: document how to debug shell script Samuel Martin
2016-04-22 21:25 ` Yegor Yefremov
2016-06-01 5:26 ` [Buildroot] [PATCH v9 00/11] Relocatable SDK/Build machine leaks: RPATH fixing Samuel Martin
2016-06-08 21:12 ` Samuel Martin
2016-06-16 4:33 ` Samuel Martin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.