[Buildroot] [PATCH v4 00/11] Make the SDK relocatable

[Buildroot] [PATCH v4 00/11] Make the SDK relocatable

[Wolfgang Grandegger]

Hello,

this is v4 of my patch series to make the buildroot SDK (HOST_DIR)
relocatable. It sanitizes the RPATH of all ELF files in the "target",
"staging" and "host" tree using "patchelf --make-rpath-relative". I
have started the mainlining process to implement "--make-rpath-relative"
using GitHub pull request [1]... till now, no answer!

Furthermore this patch creates the script "relocate-sdk.sh" in the top
directory of the "host" tree allowing to relocate the SDK after it has
been moved to a new location. It replaces the old path with the new
one in all text files identified by "file --mime-type". The location
is stored in "usr/share/buildroot/sdk-location".

Unfortunately, "qmake" uses hard-coded pathes compiled into the QT5
libraries. To overcome this problem, "qt5pase" now creates "qt.conf".

I also checked if the rpath sanitation could be done per package
after package installation into the host, staging or target tree
using GLOBAL_INSTRUMENTATION_HOOKS, similar to "check-bin-arch"
or "check-host-rpath". Unfortunately, this approach does required
bigger changes to the infrastructure, mainly to maintain a list of
ELF files installed per package into the various trees. This needs
more thought/discussion...

Other Questions:

Things not yet addressed:

- "make toolchain" creates a toolchain tree which still has references
  to the build system (in ELF and text files).

Changes since v3:

- The patchelf patch implementing " --make-rpath-relative" now supports
  the option "--relative-to-file" instructing to use "$ORIGIN" in
  RPATHs. Otherwise an absolute path relative to the root directory will
  be used.
- The staging tree is now sanitized as well using the options
  "--relative-to-file" and "--no-standard-libs".
- For the "target" tree, relative RPATHs do not use "$ORIGIN" any
  longer. An absolute path relative to the root directory is used
  instead.

Changes since v2:

- provide "qt.conf" to make "qmake" relocatable
- sed now uses the separator "\" to substitute the directory path.
  It's one of the few characters not allowed in file names. To
  avoid interpreting it as escape character, the "read -r" is used.
- The paranoia substituion check is done before doing the real
  substituion.

Changes since v1:

- The name SDK has been chosen for the relocatabed "HOST_DIR" (instead
  of toolchanin).
- The patchelf version bump and patching are now done by 2 patches
- No more helper functions are used in the Makefile to call "fix-rpath"
  but added directly.
- The staging tree is not touched any more... until we have a good
  reason to do so. 
- The sanitation is now performed by an optimized "fix-rpath" script.
- The relocate-sdk script is now copied for support/misc to the
  top directory of the host tree.

[1] https://github.com/NixOS/patchelf/pull/118

Samuel Martin (1):
  support/scripts: add fix-rpath script to sanitize the rpath

Wolfgang Grandegger (10):
  package/patchelf: use most recent version as a base for rpath
    sanitation
  package/patchelf: add patch for rpath sanitation under a root
    directory
  core: sanitize RPATH in staging tree at the end of the target
    finalization
  core: sanitize RPATH in target tree at the end of the target
    finalization
  core: sanitize RPATH in host tree at the very end of the build
  support/scripts: add reloacte-sdk.sh script for SDK relocation
  core: install relocation script and location at the end of the build
  external-toolchain: check if a buildroot SDK has already been
    relocated
  support/scripts: check-host-rpath now handles $ORIGIN as well
  package/qt5base: provide "qt.conf" to make "qmake" relocatable

 Makefile                                           |  12 +
 ...to-make-the-rpath-relative-under-a-specif.patch | 326 +++++++++++++++++++++
 package/patchelf/patchelf.hash                     |   2 +-
 package/patchelf/patchelf.mk                       |   8 +-
 package/qt5/qt5base/qt.conf.in                     |   6 +
 package/qt5/qt5base/qt5base.mk                     |   8 +
 support/misc/relocate-sdk.sh                       |  47 +++
 support/scripts/check-host-rpath                   |   2 +-
 support/scripts/fix-rpath                          | 112 +++++++
 toolchain/helpers.mk                               |  15 +
 .../toolchain-external/pkg-toolchain-external.mk   |   1 +
 11 files changed, 533 insertions(+), 6 deletions(-)
 create mode 100644 package/patchelf/0001-Add-option-to-make-the-rpath-relative-under-a-specif.patch
 create mode 100644 package/qt5/qt5base/qt.conf.in
 create mode 100755 support/misc/relocate-sdk.sh
 create mode 100755 support/scripts/fix-rpath

-- 
2.7.4

[Buildroot] [PATCH v4 01/11] package/patchelf: use most recent version as a base for rpath sanitation

[Wolfgang Grandegger]

We would like to use "patchelf" to do rpath sanitation of all ELF files
in the "host", "staging" and "target" directory, mainly because a script
based solutions is too complex and slow.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 package/patchelf/patchelf.hash | 2 +-
 package/patchelf/patchelf.mk   | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/package/patchelf/patchelf.hash b/package/patchelf/patchelf.hash
index 653eb46..95b067b 100644
--- a/package/patchelf/patchelf.hash
+++ b/package/patchelf/patchelf.hash
@@ -1,2 +1,2 @@
 # Locally calculated
-sha256	a0f65c1ba148890e9f2f7823f4bedf7ecad5417772f64f994004f59a39014f83	patchelf-0.9.tar.bz2
+sha256  c8f1e4d2d41d5b390931e9876ccab39050182dc0003e865d3edd06684dbf9d8d  patchelf-29c085fd9d3fc972f75b3961905d6b4ecce7eb2b.tar.gz
diff --git a/package/patchelf/patchelf.mk b/package/patchelf/patchelf.mk
index 74e6ccc..9750351 100644
--- a/package/patchelf/patchelf.mk
+++ b/package/patchelf/patchelf.mk
@@ -4,10 +4,10 @@
 #
 ################################################################################
 
-PATCHELF_VERSION = 0.9
-PATCHELF_SITE = http://releases.nixos.org/patchelf/patchelf-$(PATCHELF_VERSION)
-PATCHELF_SOURCE = patchelf-$(PATCHELF_VERSION).tar.bz2
-PATCHELF_LICENSE = GPL-3.0+
+PATCHELF_VERSION = 29c085fd9d3fc972f75b3961905d6b4ecce7eb2b
+PATCHELF_SITE = $(call github,NixOS,patchelf,$(PATCHELF_VERSION))
+PATCHELF_AUTORECONF = YES
+PATCHELF_LICENSE = GPLv3+
 PATCHELF_LICENSE_FILES = COPYING
 
 $(eval $(host-autotools-package))
-- 
2.7.4

[Buildroot] [PATCH v4 02/11] package/patchelf: add patch for rpath sanitation under a root directory

[Wolfgang Grandegger]

The patch allows to use patchelf to sanitize the rpath of the buildroot
libraries and binaries using the option "--make-rpath-relative <rootdir>".

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 ...to-make-the-rpath-relative-under-a-specif.patch | 326 +++++++++++++++++++++
 1 file changed, 326 insertions(+)
 create mode 100644 package/patchelf/0001-Add-option-to-make-the-rpath-relative-under-a-specif.patch

diff --git a/package/patchelf/0001-Add-option-to-make-the-rpath-relative-under-a-specif.patch b/package/patchelf/0001-Add-option-to-make-the-rpath-relative-under-a-specif.patch
new file mode 100644
index 0000000..f9e5699
--- /dev/null
+++ b/package/patchelf/0001-Add-option-to-make-the-rpath-relative-under-a-specif.patch
@@ -0,0 +1,326 @@
+From 6e28f6bc688b369e5e201058c0ac77f5bfeb822e Mon Sep 17 00:00:00 2001
+From: Wolfgang Grandegger <wg@grandegger.com>
+Date: Mon, 20 Feb 2017 16:29:24 +0100
+Subject: [PATCH] Add option to make the rpath relative under a specified root
+ directory
+
+Running "patchelf" with the option "--make-rpath-relative ROOTDIR" will
+modify or delete the RPATHDIRs according the following rules
+similar to Martin's patches [1] making the Buildroot toolchaing/SDK
+relocatable.
+
+RPATHDIR starts with "$ORIGIN":
+    The original build-system already took care of setting a relative
+    RPATH, resolve it and test if it's valid (does exist)
+
+RPATHDIR starts with ROOTDIR:
+    The original build-system added some absolute RPATH (absolute on
+    the build machine). Test if it's valid (does exist).
+
+ROOTDIR/RPATHDIR exists:
+    The original build-system already took care of setting an absolute
+    RPATH (absolute in the final rootfs), resolve it and test if it's
+    valid (does exist).
+
+RPATHDIR points somewhere else:
+    (can be anywhere: build trees, staging tree, host location,
+    non-existing location, etc.). Just discard such a path.
+
+The option "--no-standard-libs" will discard RPATHDIRs ROOTDIR/lib and
+ROOTDIR/usr/lib. Like "--shrink-rpath", RPATHDIRs are also discarded
+if the directories do not contain a library referenced by the
+DT_NEEDED fields.
+If the option "--relative-to-file" is given, the rpath will start
+with "$ORIGIN" making it relative to the ELF file, otherwise an
+absolute path relative to ROOTDIR will be used.
+
+[1] http://lists.busybox.net/pipermail/buildroot/2016-April/159422.html
+
+Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
+---
+ src/patchelf.cc | 187 ++++++++++++++++++++++++++++++++++++++++++++++++--------
+ 1 file changed, 161 insertions(+), 26 deletions(-)
+
+diff --git a/src/patchelf.cc b/src/patchelf.cc
+index 55b38e3..7dd1af2 100644
+--- a/src/patchelf.cc
++++ b/src/patchelf.cc
+@@ -50,6 +50,9 @@ static int pageSize = PAGESIZE;
+ 
+ typedef std::shared_ptr<std::vector<unsigned char>> FileContents;
+ 
++#define MODIFY_FLAG_NO_STD_LIB_DIRS 0x1
++#define MODIFY_FLAG_RELATIVE_TO_FILE 0x2
++static int modifyFlags;
+ 
+ #define ElfFileParams class Elf_Ehdr, class Elf_Phdr, class Elf_Shdr, class Elf_Addr, class Elf_Off, class Elf_Dyn, class Elf_Sym, class Elf_Verneed
+ #define ElfFileParamNames Elf_Ehdr, Elf_Phdr, Elf_Shdr, Elf_Addr, Elf_Off, Elf_Dyn, Elf_Sym, Elf_Verneed
+@@ -84,6 +87,36 @@ static unsigned int getPageSize()
+     return pageSize;
+ }
+ 
++static bool absolutePathExists(const std::string & path, std::string & canonicalPath)
++{
++    char *cpath = realpath(path.c_str(), NULL);
++    if (cpath) {
++        canonicalPath = cpath;
++        free(cpath);
++        return true;
++    } else {
++        return false;
++    }
++}
++
++static std::string makePathRelative(const std::string & path,
++    const std::string & refPath, const std::string & rootDir)
++{
++    std::string relPath = "$ORIGIN";
++
++    /* Strip root path first */
++    std::string p = path.substr(rootDir.length());
++    std::string refP = refPath.substr(rootDir.length());
++
++    std::size_t pos = refP.find_first_of('/');
++    while (pos != std::string::npos) {
++        pos =refP.find_first_of('/', pos + 1);
++        relPath.append("/..");
++    }
++    relPath.append(p);
++
++    return relPath;
++}
+ 
+ template<ElfFileParams>
+ class ElfFile
+@@ -194,9 +227,14 @@ public:
+ 
+     void setInterpreter(const std::string & newInterpreter);
+ 
+-    typedef enum { rpPrint, rpShrink, rpSet, rpRemove } RPathOp;
++    typedef enum { rpPrint, rpShrink, rpMakeRelative, rpSet, rpRemove} RPathOp;
+ 
+-    void modifyRPath(RPathOp op, const std::vector<std::string> & allowedRpathPrefixes, std::string newRPath);
++    bool libFoundInRPath(const std::string & dirName,
++                         const std::vector<std::string> neededLibs);
++
++    void modifyRPath(RPathOp op,
++                     const std::vector<std::string> & allowedRpathPrefixes,
++                     std::string rootDir, int flags, std::string newRPath);
+ 
+     void addNeeded(const std::set<std::string> & libs);
+ 
+@@ -1108,10 +1146,35 @@ static void concatToRPath(std::string & rpath, const std::string & path)
+     rpath += path;
+ }
+ 
++template<ElfFileParams>
++bool ElfFile<ElfFileParamNames>::libFoundInRPath(const std::string & dirName,
++    const std::vector<std::string> neededLibs)
++{
++    std::vector<bool> neededLibFound(neededLibs.size(), false);
++
++    /* For each library that we haven't found yet, see if it
++       exists in this directory. */
++    bool libFound = false;
++    for (unsigned int j = 0; j < neededLibs.size(); ++j)
++        if (!neededLibFound[j]) {
++            std::string libName = dirName + "/" + neededLibs[j];
++            try {
++                if (getElfType(readFile(libName, sizeof(Elf32_Ehdr))).machine == rdi(hdr->e_machine)) {
++                    neededLibFound[j] = true;
++                    libFound = true;
++                } else
++                    debug("ignoring library '%s' because its machine type differs\n", libName.c_str());
++            } catch (SysError & e) {
++                if (e.errNo != ENOENT) throw;
++            }
++        }
++    return libFound;
++}
+ 
+ template<ElfFileParams>
+ void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op,
+-    const std::vector<std::string> & allowedRpathPrefixes, std::string newRPath)
++    const std::vector<std::string> & allowedRpathPrefixes,
++    std::string rootDir, int flags, std::string newRPath)
+ {
+     Elf_Shdr & shdrDynamic = findSection(".dynamic");
+ 
+@@ -1162,11 +1225,14 @@ void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op,
+         return;
+     }
+ 
++    if (op == rpMakeRelative && !rpath) {
++        debug("no RPATH to make relative\n");
++        return;
++    }
+ 
+     /* For each directory in the RPATH, check if it contains any
+        needed library. */
+     if (op == rpShrink) {
+-        std::vector<bool> neededLibFound(neededLibs.size(), false);
+ 
+         newRPath = "";
+ 
+@@ -1186,30 +1252,81 @@ void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op,
+                 continue;
+             }
+ 
+-            /* For each library that we haven't found yet, see if it
+-               exists in this directory. */
+-            bool libFound = false;
+-            for (unsigned int j = 0; j < neededLibs.size(); ++j)
+-                if (!neededLibFound[j]) {
+-                    std::string libName = dirName + "/" + neededLibs[j];
+-                    try {
+-                        if (getElfType(readFile(libName, sizeof(Elf32_Ehdr))).machine == rdi(hdr->e_machine)) {
+-                            neededLibFound[j] = true;
+-                            libFound = true;
+-                        } else
+-                            debug("ignoring library '%s' because its machine type differs\n", libName.c_str());
+-                    } catch (SysError & e) {
+-                        if (e.errNo != ENOENT) throw;
+-                    }
+-                }
+-
+-            if (!libFound)
++            if (!libFoundInRPath(dirName, neededLibs))
+                 debug("removing directory '%s' from RPATH\n", dirName.c_str());
+             else
+                 concatToRPath(newRPath, dirName);
+         }
+     }
+ 
++    /* Make the the RPATH relative to the specified path */
++    if (op == rpMakeRelative) {
++        std::string fileDir = fileName.substr(0, fileName.find_last_of("/"));
++        newRPath = "";
++
++        for (auto & dirName : splitColonDelimitedString(rpath)) {
++            std::string canonicalPath;
++
++            /* Figure out if we should keep or discard the path. There are several
++               cases to be handled:
++               "dirName" starts with "$ORIGIN":
++                   The original build-system already took care of setting a relative
++                   RPATH. Resolve it and test if it's valid (does exist).
++               "dirName" start with "rootDir":
++                   The original build-system added some absolute RPATH (absolute on
++                   the build machine). Test if it's valid (does exist).
++               "rootDir"/"dirName" exists:
++                    The original build-system already took care of setting an absolute
++                    RPATH (absolute in the final rootfs). Resolve it and test if it's
++                    valid (does exist).
++               "dirName" points somewhere else:
++                    (can be anywhere: build trees, staging tree, host location,
++                    non-existing location, etc.). Just discard such a path. */
++            if (!dirName.compare(0, 7, "$ORIGIN")) {
++                std::string path = fileDir + dirName.substr(7);
++                if (!absolutePathExists(path, canonicalPath)) {
++                    debug("removing directory '%s' from RPATH because '%s' doesn't exist\n",
++			  dirName.c_str(), path.c_str());
++                    continue;
++                }
++            } else if (!dirName.compare(0, rootDir.length(), rootDir)) {
++                if (!absolutePathExists(dirName, canonicalPath)) {
++                    debug("removing directory '%s' from RPATH because it doesn't exist\n", dirName.c_str());
++                    continue;
++                }
++            } else {
++                std::string path = rootDir + dirName;
++                if (!absolutePathExists(path, canonicalPath)) {
++                    debug("removing directory '%s' from RPATH because '%s' doesn't exist\n",
++			  dirName.c_str(), path.c_str());
++                    continue;
++                }
++            }
++
++            if (flags & MODIFY_FLAG_NO_STD_LIB_DIRS) {
++                if (!canonicalPath.compare(rootDir + "/lib") ||
++                    !canonicalPath.compare(rootDir + "/usr/lib")) {
++                    debug("removing directory '%s' from RPATH because it's a standard library directory\n",
++                         dirName.c_str());
++                    continue;
++                }
++            }
++
++            if (!libFoundInRPath(canonicalPath, neededLibs)) {
++                debug("removing directory '%s' from RPATH because it does not contain needed libs\n",
++		      dirName.c_str());
++                continue;
++            }
++
++            /* Finally make "canonicalPath" relative to "filedir" in "rootDir" */
++	    if (flags & MODIFY_FLAG_RELATIVE_TO_FILE)
++		concatToRPath(newRPath, makePathRelative(canonicalPath, fileDir, rootDir));
++	    else
++		concatToRPath(newRPath, canonicalPath.substr(rootDir.length()));
++	    debug("keeping relative path of %s\n", canonicalPath.c_str());
++	}
++    }
++
+     if (op == rpRemove) {
+         if (!rpath) {
+             debug("no RPATH to delete\n");
+@@ -1538,7 +1655,9 @@ static std::vector<std::string> allowedRpathPrefixes;
+ static bool removeRPath = false;
+ static bool setRPath = false;
+ static bool printRPath = false;
++static bool makeRPathRelative = false;
+ static std::string newRPath;
++static std::string rootDir;
+ static std::set<std::string> neededLibsToRemove;
+ static std::map<std::string, std::string> neededLibsToReplace;
+ static std::set<std::string> neededLibsToAdd;
+@@ -1561,14 +1680,16 @@ static void patchElf2(ElfFile && elfFile)
+         elfFile.setInterpreter(newInterpreter);
+ 
+     if (printRPath)
+-        elfFile.modifyRPath(elfFile.rpPrint, {}, "");
++        elfFile.modifyRPath(elfFile.rpPrint, {}, {}, modifyFlags, "");
+ 
+     if (shrinkRPath)
+-        elfFile.modifyRPath(elfFile.rpShrink, allowedRpathPrefixes, "");
++        elfFile.modifyRPath(elfFile.rpShrink, allowedRpathPrefixes, "", modifyFlags, "");
+     else if (removeRPath)
+-        elfFile.modifyRPath(elfFile.rpRemove, {}, "");
++        elfFile.modifyRPath(elfFile.rpRemove, {}, "", modifyFlags, "");
+     else if (setRPath)
+-        elfFile.modifyRPath(elfFile.rpSet, {}, newRPath);
++        elfFile.modifyRPath(elfFile.rpSet, {}, "", modifyFlags, newRPath);
++    else if (makeRPathRelative)
++        elfFile.modifyRPath(elfFile.rpMakeRelative, {}, rootDir, modifyFlags, "");
+ 
+     if (printNeeded) elfFile.printNeededLibs();
+ 
+@@ -1614,6 +1735,9 @@ void showHelp(const std::string & progName)
+   [--remove-rpath]\n\
+   [--shrink-rpath]\n\
+   [--allowed-rpath-prefixes PREFIXES]\t\tWith '--shrink-rpath', reject rpath entries not starting with the allowed prefix\n\
++  [--make-rpath-relative ROOTDIR]\n\
++  [--no-standard-lib-dirs]\n\
++  [--relative-to-file]\n\
+   [--print-rpath]\n\
+   [--force-rpath]\n\
+   [--add-needed LIBRARY]\n\
+@@ -1674,6 +1798,17 @@ int mainWrapped(int argc, char * * argv)
+             setRPath = true;
+             newRPath = argv[i];
+         }
++        else if (arg == "--make-rpath-relative") {
++            if (++i == argc) error("missing argument to --make-rpath-relative");
++            makeRPathRelative = true;
++            rootDir = argv[i];
++        }
++        else if (arg == "--no-standard-lib-dirs") {
++            modifyFlags |= MODIFY_FLAG_NO_STD_LIB_DIRS;
++        }
++        else if (arg == "--relative-to-file") {
++            modifyFlags |= MODIFY_FLAG_RELATIVE_TO_FILE;
++        }
+         else if (arg == "--print-rpath") {
+             printRPath = true;
+         }
+-- 
+1.9.1
+
-- 
2.7.4

[Buildroot] [PATCH v4 03/11] support/scripts: add fix-rpath script to sanitize the rpath

[Wolfgang Grandegger]

From: Samuel Martin <s.martin49@gmail.com>

This commit introduces the script "fix-rpath" able to scan a tree,
detect ELF files, check their RPATH and fix it in a proper way.
The RPATH fixup is done by the patchelf utility using the option
"--make-rpath-relative <root-directory>".

Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 support/scripts/fix-rpath | 112 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 112 insertions(+)
 create mode 100755 support/scripts/fix-rpath

diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
new file mode 100755
index 0000000..5d40657
--- /dev/null
+++ b/support/scripts/fix-rpath
@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+usage() {
+  cat <<EOF >&2
+Usage:	${0} TREE_KIND
+
+Description:
+
+    This script scans a tree and sanitize ELF files' RPATH found in there.
+
+    Sanitization behaves the same whatever the kind of the processed tree,
+    but the resulting RPATH differs. The rpath sanitization is done using
+    "patchelf --make-rpath-relazive".
+
+Arguments:
+
+    TREE_KIND	Kind of tree to be processed.
+		Allowed values: host, target, staging
+
+Environment:
+
+    PATCHELF	patchelf program to use
+		(default: HOST_DIR/usr/bin/patchelf)
+EOF
+}
+
+: ${PATCHELF:=${HOST_DIR}/usr/bin/patchelf}
+
+main() {
+    local rootdir
+    local tree="${1}"
+    local find_args=( )
+    local sanitize_extra_args=( )
+
+    case "${tree}" in
+	host)
+	    rootdir="${HOST_DIR}"
+
+	    # do not process the sysroot (only contains target binaries)
+	    find_args+=( "-path" "${STAGING_DIR}" "-prune" "-o" )
+
+	    # do not process the external toolchain installation directory to
+	    # avoid breaking it.
+	    test "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" != "" && \
+		find_args+=( "-path" "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" "-prune" "-o" )
+
+	    # ELF files should not be in these sub-directories
+	    find_args+=( "-path" "${STAGING_DIR}/usr/share/terminfo" "-prune" "-o" )
+
+	    # do not process the patchelf binary but a copy to work-around "file in use"
+	    find_args+=( "-path" "${PATCHELF}" "-prune" "-o" )
+	    cp "${PATCHELF}" "${PATCHELF}.__to_be_patched"
+
+	    sanitize_extra_args+=( "--relative-to-file" )
+	    ;;
+
+	staging)
+	    rootdir="${STAGING_DIR}"
+	    # supress include files
+	    find_args+=( "-path" "${STAGING_DIR}/usr/include" "-prune" "-o" )
+	    sanitize_extra_args+=( "--no-standard-lib-dirs" "--relative-to-file" )
+	    ;;
+
+	target)
+	    rootdir="${TARGET_DIR}"
+	    sanitize_extra_args+=( "--no-standard-lib-dirs" )
+	    ;;
+
+	*)
+	    usage
+	    exit 1
+	    ;;
+    esac
+
+    find_args+=( "-type" "f" "-print" )
+
+    while read file ; do
+	# check if it's an ELF file
+	if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then
+	    # make files writable if necessary
+	    changed=$(chmod -c u+w "${file}")
+	    # call patchelf to sanitize the rpath
+	    ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}"
+	    # restore the original permission
+	    test "${changed}" != "" && chmod u-w "${file}"
+	fi
+    done < <(find "${rootdir}" ${find_args[@]})
+
+    # Restore patched patchelf utility
+    test "${tree}" = "host" && mv "${PATCHELF}.__to_be_patched" "${PATCHELF}"
+
+    # ignore errors
+    return 0
+}
+
+main ${@}
-- 
2.7.4

[Buildroot] [PATCH v4 04/11] core: sanitize RPATH in staging tree at the end of the target finalization

[Wolfgang Grandegger]

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 Makefile | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Makefile b/Makefile
index 88d98e0..3b0e336 100644
--- a/Makefile
+++ b/Makefile
@@ -715,6 +715,9 @@ endif
 		$(call MESSAGE,"Executing post-build script $(s)"); \
 		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
+	@$(call MESSAGE,"Sanitizing RPATH in staging tree")
+	$(TOPDIR)/support/scripts/fix-rpath staging
+
 .PHONY: target-post-image
 target-post-image: $(TARGETS_ROOTFS) target-finalize
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
-- 
2.7.4

[Buildroot] [PATCH v4 05/11] core: sanitize RPATH in target tree at the end of the target finalization

[Wolfgang Grandegger]

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Makefile b/Makefile
index 3b0e336..bda4a05 100644
--- a/Makefile
+++ b/Makefile
@@ -717,6 +717,8 @@ endif
 
 	@$(call MESSAGE,"Sanitizing RPATH in staging tree")
 	$(TOPDIR)/support/scripts/fix-rpath staging
+	@$(call MESSAGE,"Sanitizing RPATH in target tree")
+	$(TOPDIR)/support/scripts/fix-rpath target
 
 .PHONY: target-post-image
 target-post-image: $(TARGETS_ROOTFS) target-finalize
-- 
2.7.4

[Buildroot] [PATCH v4 06/11] core: sanitize RPATH in host tree at the very end of the build

[Wolfgang Grandegger]

We need the patchelf host utility to check and fix the rpath.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Makefile b/Makefile
index bda4a05..e492342 100644
--- a/Makefile
+++ b/Makefile
@@ -551,6 +551,11 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
 .PHONY: world
 world: target-post-image
+	@$(call MESSAGE,"Rendering the SDK relocatable")
+	$(TOPDIR)/support/scripts/fix-rpath host
+
+# We need patchelf for RPATH sanitation
+PACKAGES += host-patchelf
 
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):
-- 
2.7.4

[Buildroot] [PATCH v4 07/11] support/scripts: add reloacte-sdk.sh script for SDK relocation

[Wolfgang Grandegger]

It will install the script "relocate-sdk.sh" in the HOST_DIR
allowing to adjust the path to the SDK directory in all text
files after it has been moved to a new location.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 support/misc/relocate-sdk.sh | 47 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 support/misc/relocate-sdk.sh

diff --git a/support/misc/relocate-sdk.sh b/support/misc/relocate-sdk.sh
new file mode 100755
index 0000000..1b0be33
--- /dev/null
+++ b/support/misc/relocate-sdk.sh
@@ -0,0 +1,47 @@
+#!/bin/bash
+#
+if [ "$#" -ne 0 ]; then
+    echo "Run this script to relocate the buildroot SDK at that location"
+    exit 1
+fi
+
+LOCFILE="./usr/share/buildroot/sdk-location"
+FILEPATH="$(readlink -f "$0")"
+NEWPATH="$(dirname "${FILEPATH}")"
+
+cd "${NEWPATH}"
+if [ ! -r "${LOCFILE}" ]; then
+    echo "Previous location of the buildroot SDK not found!"
+    exit 1
+fi
+OLDPATH="$(cat "${LOCFILE}")"
+
+if [ "${NEWPATH}" = "${OLDPATH}" ]; then
+    echo "This buildroot SDK has already been relocated!"
+    exit 0
+fi
+
+# Check if the path substitution does work properly, e.g.
+# a tree "/a/b/c" copied into "/a/b/c/" would not be allowed.
+newpath="$(sed -e "s\\${OLDPATH}\\${NEWPATH}\\g" "${LOCFILE}")"
+if [ "${NEWPATH}" != "${newpath}" ]; then
+    echo "Something went wrong with substituting the path!"
+    echo "Please choose another location for your SDK!"
+    exit 1
+fi
+
+echo "Relocating the buildroot SDK from ${OLDPATH} to ${NEWPATH} ..."
+
+# Make sure file uses the right language
+export LC_ALL=C
+# Replace the old path with the new one in all text files
+while read -r FILE ; do
+    if file -b --mime-type "${FILE}" | grep -q '^text/' && [ "${FILE}" != "${LOCFILE}" ]
+    then
+	sed -i "s\\${OLDPATH}\\${NEWPATH}\\g" "${FILE}"
+    fi;
+done < <(grep -lr "${OLDPATH}" .)
+
+# At the very end, we update the location file to not break the
+# SDK if this script gets interruted.
+sed -i "s\\${OLDPATH}\\${NEWPATH}\\g" ${LOCFILE}
-- 
2.7.4

[Buildroot] [PATCH v4 08/11] core: install relocation script and location at the end of the build

[Wolfgang Grandegger]

The script "relocate-sdk.sh" is installed into the top directory of
the SDK (HOST_DIR) and the SDK location path is stored in the file
"HOST_DIR/usr/share/buildroot/sdk-location"-

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Makefile b/Makefile
index e492342..6181d9f 100644
--- a/Makefile
+++ b/Makefile
@@ -553,6 +553,8 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 world: target-post-image
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
+	install $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR) && \
+	echo $(HOST_DIR) > $(HOST_DIR)/usr/share/buildroot/sdk-location
 
 # We need patchelf for RPATH sanitation
 PACKAGES += host-patchelf
-- 
2.7.4

[Buildroot] [PATCH v4 09/11] external-toolchain: check if a buildroot SDK has already been relocated

[Wolfgang Grandegger]

The location of the buildroot SDK is stored in the file "sdk-location"
in "usr/share/buildroot". If it's content does not match the current
SDK location, ask the user to run the script "relocate-sdk.sh" in the
top directory once.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 toolchain/helpers.mk                                   | 15 +++++++++++++++
 toolchain/toolchain-external/pkg-toolchain-external.mk |  1 +
 2 files changed, 16 insertions(+)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 90834f4..43e89fc 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -479,3 +479,18 @@ define simplify_symlink
 	ln -sf "$${DOTS}$${REL_DEST}" "$${FULL_SRC}" ; \
 )
 endef
+
+#
+# Check if it's a buildroot toolchain and if it's already relocatable by
+# reading and testing the toolchain location file
+#
+# $1: toolchain installation directory
+#
+check_buildroot_sdk_relocated = \
+	if [ -r $(1)/share/buildroot/sdk-location ]; then \
+		sdkroot=`dirname "$(1)"`; \
+		if [ "`cat $(1)/share/buildroot/sdk-location`" != "$${sdkroot}" ]; then \
+			echo "Please relocate the buildroot SDK by executing \"$${sdkroot}/relocate-sdk.sh\" once!" ; \
+			exit 1 ; \
+		fi \
+	fi
diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
index 8269345..761221b 100644
--- a/toolchain/toolchain-external/pkg-toolchain-external.mk
+++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
@@ -545,6 +545,7 @@ endif
 # matches the configuration provided in Buildroot: ABI, C++ support,
 # kernel headers version, type of C library and all C library features.
 define $(2)_CONFIGURE_CMDS
+	$$(Q)$$(call check_buildroot_sdk_relocated,$$(TOOLCHAIN_EXTERNAL_INSTALL_DIR))
 	$$(Q)$$(call check_cross_compiler_exists,$$(TOOLCHAIN_EXTERNAL_CC))
 	$$(Q)$$(call check_unusable_toolchain,$$(TOOLCHAIN_EXTERNAL_CC))
 	$$(Q)SYSROOT_DIR="$$(call toolchain_find_sysroot,$$(TOOLCHAIN_EXTERNAL_CC))" ; \
-- 
2.7.4

[Buildroot] [PATCH v4 10/11] support/scripts: check-host-rpath now handles $ORIGIN as well

[Wolfgang Grandegger]

"$ORIGIN/../../usr/lib" is also a valid RPATH for binaries in
"$hostdir/usr/bin". After RPATH sanitation, all RPATH
directories start with "$ORIGIN".

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 support/scripts/check-host-rpath | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath
index 6ce547c..020c123 100755
--- a/support/scripts/check-host-rpath
+++ b/support/scripts/check-host-rpath
@@ -58,7 +58,7 @@ check_elf_has_rpath() {
         for dir in ${rpath//:/ }; do
             # Remove duplicate and trailing '/' for proper match
             dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
-            [ "${dir}" = "${hostdir}/usr/lib" ] && return 0
+            [ "${dir}" = "${hostdir}/usr/lib" -o "${dir}" = "\$ORIGIN/../../usr/lib" ] && return 0
         done
     done < <( readelf -d "${file}"                                              \
               |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \
-- 
2.7.4

[Buildroot] [PATCH v4 11/11] package/qt5base: provide "qt.conf" to make "qmake" relocatable

[Wolfgang Grandegger]

The file "qt.conf" can be used to override the hard-coded paths that are
compiled into the Qt library. We need it to make "qmake" relocatable.

CC: Julien Corjon <corjon.j@ecagroup.com>
CC: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 package/qt5/qt5base/qt.conf.in | 6 ++++++
 package/qt5/qt5base/qt5base.mk | 8 ++++++++
 2 files changed, 14 insertions(+)
 create mode 100644 package/qt5/qt5base/qt.conf.in

diff --git a/package/qt5/qt5base/qt.conf.in b/package/qt5/qt5base/qt.conf.in
new file mode 100644
index 0000000..48e4b92
--- /dev/null
+++ b/package/qt5/qt5base/qt.conf.in
@@ -0,0 +1,6 @@
+[Paths]
+Prefix=@@HOST_DIR@@/usr
+Sysroot=@@STAGING_DIR@@
+Headers=/usr/include/qt5
+Plugins=/usr/lib/qt/plugins
+Examples=/usr/lib/qt/examples
diff --git a/package/qt5/qt5base/qt5base.mk b/package/qt5/qt5base/qt5base.mk
index 5fe8bb8..71dbd5d 100644
--- a/package/qt5/qt5base/qt5base.mk
+++ b/package/qt5/qt5base/qt5base.mk
@@ -258,9 +258,17 @@ define QT5BASE_BUILD_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
+# The file "qt.conf" can be used to override the hard-coded paths that are
+# compiled into the Qt library. We need it to make "qmake" relocatable.
+define QT5BASE_INSTALL_QT_CONF
+	sed -e "s\\@@HOST_DIR@@\\$(HOST_DIR)\\" -e "s\\@@STAGING_DIR@@\\$(STAGING_DIR)\\" \
+	$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/usr/bin/qt.conf
+endef
+
 define QT5BASE_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install
 	$(QT5_LA_PRL_FILES_FIXUP)
+	$(QT5BASE_INSTALL_QT_CONF)
 endef
 
 define QT5BASE_INSTALL_TARGET_LIBS
-- 
2.7.4

[Buildroot] [PATCH v4 03/11] support/scripts: add fix-rpath script to sanitize the rpath

[Arnout Vandecappelle]

 Since you're going to revert to this v4, I'm going to add a few comments here.

On 27-06-17 12:26, Wolfgang Grandegger wrote:
> From: Samuel Martin <s.martin49@gmail.com>
> 
> This commit introduces the script "fix-rpath" able to scan a tree,
> detect ELF files, check their RPATH and fix it in a proper way.
> The RPATH fixup is done by the patchelf utility using the option
> "--make-rpath-relative <root-directory>".
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
> ---
>  support/scripts/fix-rpath | 112 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 112 insertions(+)
>  create mode 100755 support/scripts/fix-rpath
> 
> diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
> new file mode 100755
> index 0000000..5d40657
> --- /dev/null
> +++ b/support/scripts/fix-rpath
> @@ -0,0 +1,112 @@
> +#!/usr/bin/env bash
> +
> +# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +
> +usage() {
> +  cat <<EOF >&2
> +Usage:	${0} TREE_KIND
> +
> +Description:
> +
> +    This script scans a tree and sanitize ELF files' RPATH found in there.
> +
> +    Sanitization behaves the same whatever the kind of the processed tree,
> +    but the resulting RPATH differs. The rpath sanitization is done using
> +    "patchelf --make-rpath-relazive".
                                  ^t
> +
> +Arguments:
> +
> +    TREE_KIND	Kind of tree to be processed.
> +		Allowed values: host, target, staging
> +
> +Environment:
> +
> +    PATCHELF	patchelf program to use
> +		(default: HOST_DIR/usr/bin/patchelf)
> +EOF
> +}
> +
> +: ${PATCHELF:=${HOST_DIR}/usr/bin/patchelf}
> +
> +main() {
> +    local rootdir
> +    local tree="${1}"
> +    local find_args=( )
> +    local sanitize_extra_args=( )
> +
> +    case "${tree}" in
> +	host)

 Since now you need all this find stuff, my earlier suggestion to not pass the
tree as an argument is no longer valid.

> +	    rootdir="${HOST_DIR}"
> +
> +	    # do not process the sysroot (only contains target binaries)
> +	    find_args+=( "-path" "${STAGING_DIR}" "-prune" "-o" )
> +
> +	    # do not process the external toolchain installation directory to
> +	    # avoid breaking it.
> +	    test "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" != "" && \
> +		find_args+=( "-path" "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" "-prune" "-o" )
> +
> +	    # ELF files should not be in these sub-directories
> +	    find_args+=( "-path" "${STAGING_DIR}/usr/share/terminfo" "-prune" "-o" )

 I don't understand how this can be needed, since STAGING_DIR is already pruned
above.

> +
> +	    # do not process the patchelf binary but a copy to work-around "file in use"
> +	    find_args+=( "-path" "${PATCHELF}" "-prune" "-o" )
> +	    cp "${PATCHELF}" "${PATCHELF}.__to_be_patched"
> +
> +	    sanitize_extra_args+=( "--relative-to-file" )
> +	    ;;
> +
> +	staging)
> +	    rootdir="${STAGING_DIR}"
> +	    # supress include files
> +	    find_args+=( "-path" "${STAGING_DIR}/usr/include" "-prune" "-o" )

 Perhaps it's better to define a variable with all the paths we want to exclude
- we may want to add more. OTOH, we can do that whenever we indeed do add more.

> +	    sanitize_extra_args+=( "--no-standard-lib-dirs" "--relative-to-file" )
> +	    ;;
> +
> +	target)
> +	    rootdir="${TARGET_DIR}"
> +	    sanitize_extra_args+=( "--no-standard-lib-dirs" )
> +	    ;;
> +
> +	*)
> +	    usage
> +	    exit 1
> +	    ;;
> +    esac
> +
> +    find_args+=( "-type" "f" "-print" )
> +
> +    while read file ; do
> +	# check if it's an ELF file
> +	if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then

 Wasn't "readelf -d ${file} | grep DT_RUNPATH" faster? You did some experiments
a while ago but I forgot what the conclusions were.


 Regards,
 Arnout


> +	    # make files writable if necessary
> +	    changed=$(chmod -c u+w "${file}")
> +	    # call patchelf to sanitize the rpath
> +	    ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}"
> +	    # restore the original permission
> +	    test "${changed}" != "" && chmod u-w "${file}"
> +	fi
> +    done < <(find "${rootdir}" ${find_args[@]})
> +
> +    # Restore patched patchelf utility
> +    test "${tree}" = "host" && mv "${PATCHELF}.__to_be_patched" "${PATCHELF}"
> +
> +    # ignore errors
> +    return 0
> +}
> +
> +main ${@}
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v4 03/11] support/scripts: add fix-rpath script to sanitize the rpath

[Wolfgang Grandegger]

Hello,

Am 04.07.2017 um 14:15 schrieb Arnout Vandecappelle:
>   Since you're going to revert to this v4, I'm going to add a few comments here.

Yes, that's what doing now...

> On 27-06-17 12:26, Wolfgang Grandegger wrote:
>> From: Samuel Martin <s.martin49@gmail.com>
>>
>> This commit introduces the script "fix-rpath" able to scan a tree,
>> detect ELF files, check their RPATH and fix it in a proper way.
>> The RPATH fixup is done by the patchelf utility using the option
>> "--make-rpath-relative <root-directory>".
>>
>> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
>> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
>> ---
>>   support/scripts/fix-rpath | 112 ++++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 112 insertions(+)
>>   create mode 100755 support/scripts/fix-rpath
>>
>> diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
>> new file mode 100755
>> index 0000000..5d40657
>> --- /dev/null
>> +++ b/support/scripts/fix-rpath
>> @@ -0,0 +1,112 @@
>> +#!/usr/bin/env bash
>> +
>> +# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
>> +#
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 2 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
>> +# General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write to the Free Software
>> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
>> +
>> +usage() {
>> +  cat <<EOF >&2
>> +Usage:	${0} TREE_KIND
>> +
>> +Description:
>> +
>> +    This script scans a tree and sanitize ELF files' RPATH found in there.
>> +
>> +    Sanitization behaves the same whatever the kind of the processed tree,
>> +    but the resulting RPATH differs. The rpath sanitization is done using
>> +    "patchelf --make-rpath-relazive".
>                                    ^t
>> +
>> +Arguments:
>> +
>> +    TREE_KIND	Kind of tree to be processed.
>> +		Allowed values: host, target, staging
>> +
>> +Environment:
>> +
>> +    PATCHELF	patchelf program to use
>> +		(default: HOST_DIR/usr/bin/patchelf)
>> +EOF
>> +}
>> +
>> +: ${PATCHELF:=${HOST_DIR}/usr/bin/patchelf}
>> +
>> +main() {
>> +    local rootdir
>> +    local tree="${1}"
>> +    local find_args=( )
>> +    local sanitize_extra_args=( )
>> +
>> +    case "${tree}" in
>> +	host)
> 
>   Since now you need all this find stuff, my earlier suggestion to not pass the
> tree as an argument is no longer valid.

I didn't change it so far.

> 
>> +	    rootdir="${HOST_DIR}"
>> +
>> +	    # do not process the sysroot (only contains target binaries)
>> +	    find_args+=( "-path" "${STAGING_DIR}" "-prune" "-o" )
>> +
>> +	    # do not process the external toolchain installation directory to
>> +	    # avoid breaking it.
>> +	    test "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" != "" && \
>> +		find_args+=( "-path" "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" "-prune" "-o" )
>> +
>> +	    # ELF files should not be in these sub-directories
>> +	    find_args+=( "-path" "${STAGING_DIR}/usr/share/terminfo" "-prune" "-o" )
> 
>   I don't understand how this can be needed, since STAGING_DIR is already pruned
> above.

That's wrong here. But there are many files in both, the host and 
staging tree:

$ find host/usr/share/terminfo/ -type f | wc -l
2616
$ find  host/usr/x86_64-buildroot-linux-gnu/sysroot/usr/share/terminfo/ 
share/terminfo/ -type f | wc -l
2616

> 
>> +
>> +	    # do not process the patchelf binary but a copy to work-around "file in use"
>> +	    find_args+=( "-path" "${PATCHELF}" "-prune" "-o" )
>> +	    cp "${PATCHELF}" "${PATCHELF}.__to_be_patched"
>> +
>> +	    sanitize_extra_args+=( "--relative-to-file" )
>> +	    ;;
>> +
>> +	staging)
>> +	    rootdir="${STAGING_DIR}"
>> +	    # supress include files
>> +	    find_args+=( "-path" "${STAGING_DIR}/usr/include" "-prune" "-o" )
> 
>   Perhaps it's better to define a variable with all the paths we want to exclude
> - we may want to add more. OTOH, we can do that whenever we indeed do add more.

But it's not just a variable, I need the leading -path and the trailing 
-prune for each dir exclude.

>> +	    sanitize_extra_args+=( "--no-standard-lib-dirs" "--relative-to-file" )
>> +	    ;;
>> +
>> +	target)
>> +	    rootdir="${TARGET_DIR}"
>> +	    sanitize_extra_args+=( "--no-standard-lib-dirs" )
>> +	    ;;
>> +
>> +	*)
>> +	    usage
>> +	    exit 1
>> +	    ;;
>> +    esac
>> +
>> +    find_args+=( "-type" "f" "-print" )
>> +
>> +    while read file ; do
>> +	# check if it's an ELF file
>> +	if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then
> 
>   Wasn't "readelf -d ${file} | grep DT_RUNPATH" faster? You did some experiments
> a while ago but I forgot what the conclusions were.

There could also be DT_RPATH? But a grep for RPATH should fix that. I 
didn't test that, because I thought that pining is slower anyway. 
Without "grep DT_RUNPATH" it's faster but it will succeed then on 
non-dynamic ELF files as well. We could also tune patchelf to return 
faster if it's not an ELF file. I will remeasure with the command above.

Wolfgang.

[Buildroot] [PATCH v4 03/11] support/scripts: add fix-rpath script to sanitize the rpath

[Arnout Vandecappelle]

On 04-07-17 14:36, Wolfgang Grandegger wrote:
> Hello,
> 
> Am 04.07.2017 um 14:15 schrieb Arnout Vandecappelle:
>>   Since you're going to revert to this v4, I'm going to add a few comments here.
> 
> Yes, that's what doing now...
> 
>> On 27-06-17 12:26, Wolfgang Grandegger wrote:

[snip]
>>> +    staging)
>>> +        rootdir="${STAGING_DIR}"
>>> +        # supress include files
>>> +        find_args+=( "-path" "${STAGING_DIR}/usr/include" "-prune" "-o" )
>>
>>   Perhaps it's better to define a variable with all the paths we want to exclude
>> - we may want to add more. OTOH, we can do that whenever we indeed do add more.
> 
> But it's not just a variable, I need the leading -path and the trailing -prune
> for each dir exclude.

 Yeah, but you could do
    for excludepath in $EXCLUDEPATHS; do
        find_args += ( "-path" "${STAGING_DIR}""${excludepath}" "-prune" "-o" )

> 
>>> +        sanitize_extra_args+=( "--no-standard-lib-dirs" "--relative-to-file" )
>>> +        ;;
>>> +
>>> +    target)
>>> +        rootdir="${TARGET_DIR}"
>>> +        sanitize_extra_args+=( "--no-standard-lib-dirs" )
>>> +        ;;
>>> +
>>> +    *)
>>> +        usage
>>> +        exit 1
>>> +        ;;
>>> +    esac
>>> +
>>> +    find_args+=( "-type" "f" "-print" )
>>> +
>>> +    while read file ; do
>>> +    # check if it's an ELF file
>>> +    if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then
>>
>>   Wasn't "readelf -d ${file} | grep DT_RUNPATH" faster? You did some experiments
>> a while ago but I forgot what the conclusions were.
> 
> There could also be DT_RPATH? 

 Er, right, indeed.

> But a grep for RPATH should fix that. I didn't
> test that, because I thought that pining is slower anyway. Without "grep
> DT_RUNPATH" it's faster but it will succeed then on non-dynamic ELF files as
> well. We could also tune patchelf to return faster if it's not an ELF file. I
> will remeasure with the command above.

 Since this series is already in v6, I propose that you do the simple thing that
you have now first, and it can be optimised later.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v4 03/11] support/scripts: add fix-rpath script to sanitize the rpath

[Wolfgang Grandegger]

Am 04.07.2017 um 15:23 schrieb Arnout Vandecappelle:
> 
> 
> On 04-07-17 14:36, Wolfgang Grandegger wrote:
>> Hello,
>>
>> Am 04.07.2017 um 14:15 schrieb Arnout Vandecappelle:
>>>    Since you're going to revert to this v4, I'm going to add a few comments here.
>>
>> Yes, that's what doing now...
>>
>>> On 27-06-17 12:26, Wolfgang Grandegger wrote:
> 
> [snip]
>>>> +    staging)
>>>> +        rootdir="${STAGING_DIR}"
>>>> +        # supress include files
>>>> +        find_args+=( "-path" "${STAGING_DIR}/usr/include" "-prune" "-o" )
>>>
>>>    Perhaps it's better to define a variable with all the paths we want to exclude
>>> - we may want to add more. OTOH, we can do that whenever we indeed do add more.
>>
>> But it's not just a variable, I need the leading -path and the trailing -prune
>> for each dir exclude.
> 
>   Yeah, but you could do
>      for excludepath in $EXCLUDEPATHS; do
>          find_args += ( "-path" "${STAGING_DIR}""${excludepath}" "-prune" "-o" )

OK, and there will be exclude patches for each tree!

> 
>>
>>>> +        sanitize_extra_args+=( "--no-standard-lib-dirs" "--relative-to-file" )
>>>> +        ;;
>>>> +
>>>> +    target)
>>>> +        rootdir="${TARGET_DIR}"
>>>> +        sanitize_extra_args+=( "--no-standard-lib-dirs" )
>>>> +        ;;
>>>> +
>>>> +    *)
>>>> +        usage
>>>> +        exit 1
>>>> +        ;;
>>>> +    esac
>>>> +
>>>> +    find_args+=( "-type" "f" "-print" )
>>>> +
>>>> +    while read file ; do
>>>> +    # check if it's an ELF file
>>>> +    if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then
>>>
>>>    Wasn't "readelf -d ${file} | grep DT_RUNPATH" faster? You did some experiments
>>> a while ago but I forgot what the conclusions were.
>>
>> There could also be DT_RPATH?
> 
>   Er, right, indeed.
> 
>> But a grep for RPATH should fix that. I didn't
>> test that, because I thought that pining is slower anyway. Without "grep
>> DT_RUNPATH" it's faster but it will succeed then on non-dynamic ELF files as
>> well. We could also tune patchelf to return faster if it's not an ELF file. I
>> will remeasure with the command above.
> 
>   Since this series is already in v6, I propose that you do the simple thing that
> you have now first, and it can be optimised later.

Just playing with it. Here are some quick results:

   if patchelf --print-rpath "${file}" > /dev/null 2>&1; then

   35.5 sec

   if readelf -d "${file}" 2>/dev/null | grep RPATH > /dev/null; then

   30.4 sec

   if readelf -d "${file}" > /dev/null 2>&1 -a \
      patchelf --print-rpath "${file}" > /dev/null 2>&1; then

   27.7 sec

There is some saving but maybe not worth the effort for the moment. 
Concentrating on the next release now.

Wolfgang.

[Buildroot] [PATCH v4 03/11] support/scripts: add fix-rpath script to sanitize the rpath

[Samuel Martin]

Wolfgang

On Tue, Jul 4, 2017 at 4:52 PM, Wolfgang Grandegger <wg@grandegger.com> wrote:
[...]

> Just playing with it. Here are some quick results:
>
>   if patchelf --print-rpath "${file}" > /dev/null 2>&1; then
>
>   35.5 sec
>
>   if readelf -d "${file}" 2>/dev/null | grep RPATH > /dev/null; then
>
>   30.4 sec
>
>   if readelf -d "${file}" > /dev/null 2>&1 -a \
>      patchelf --print-rpath "${file}" > /dev/null 2>&1; then

hmm... I think you should do: s/-a/&&/
AFAIK, `-a' is only valid with `test' or '`[ some-expression -a
some-other-expression ]', `&&' should be used to combine the result of
several commands.

Regards,

>
>   27.7 sec
>
> There is some saving but maybe not worth the effort for the moment.
> Concentrating on the next release now.
>
> Wolfgang.
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot



-- 
Samuel