[Buildroot] [PATCHv2 00/10] toolchain: improvements to copy_toolchain_sysroot and copy_toolchain_lib_root

[Buildroot] [PATCHv2 00/10] toolchain: improvements to copy_toolchain_sysroot and copy_toolchain_lib_root

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

Hi,

This toolchain patch series started out to fix the handling of the Cavium Octeon
2 toolchain (see patch "cover multilib toolchains with lib/<variant>/layout")
but along the way I fixed/improved a few other things.
Details are found in the commit messages of these patches.

I tested a number of external toolchains and arches available in Buildroot, as
well as the Octeon SDK toolchain.  I think I understand the situation
sufficiently well to get some confidence, but this is the type of
change that is difficult to get 100% certainty about.

Best regards,
Thomas

v2:
- fix broken symlinks in patch "cover multilib toolchains ..."
- simplification of fixed broken symlinks
- improvements in copy_toolchain_lib_root

Thomas De Schampheleire (10):
  toolchain-external: clarify rsync excludes in copy_toolchain_sysroot
  toolchain-external: handle ld.so fixups centrally
  toolchain helpers: introduce function relpath_prefix
  toolchain-external: cover multilib toolchains with lib/<variant>
    layout
  toolchain helpers: introduce simplify_symlink
  toolchain-external: simplify previously-broken symbolic links
  toolchain: copy_toolchain_lib_root: remove unused variable LIBDIR
  toolchain: copy_toolchain_lib_root: clarify logic
  toolchain: copy_toolchain_lib_root: clarify input parameter
  toolchain: copy_toolchain_lib_root: copy symlinks instead of
    recreating them

 package/glibc/glibc.mk                             |   4 +-
 toolchain/helpers.mk                               | 126 +++++++++++++++++----
 .../toolchain-external/pkg-toolchain-external.mk   |  14 ++-
 .../toolchain-external-codesourcery-aarch64.mk     |  10 --
 4 files changed, 114 insertions(+), 40 deletions(-)

-- 
2.10.2

[Buildroot] [PATCHv2 01/10] toolchain-external: clarify rsync excludes in copy_toolchain_sysroot

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

The copy_toolchain_sysroot helper features a complex rsync loop that copies
various directories from the extracted toolchain to the staging directory.
The complexity mainly stems from the fact that we support multilib toolchain
tarballs but only copy one of the multilib variants into staging.

Increase understandability of this logic by explicitly restricting the
rsync excludes to the iteration of the for loop they are relevant for.
Additionally, update the function comment.

Note: all attempts to reduce duplication between both rsync while keeping
things nice and readable failed. One has to be extremely careful regarding
line continuation, indentation, and single vs double quoting. In the end, a
split up rsync seemed most clean.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: no changes

 toolchain/helpers.mk | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 72e7292..06dc197 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -39,11 +39,16 @@ copy_toolchain_lib_root = \
 # dir. The operation of this function is rendered a little bit
 # complicated by the support for multilib toolchains.
 #
-# We start by copying etc, lib, sbin and usr from the sysroot of the
-# selected architecture variant (as pointed by ARCH_SYSROOT_DIR). This
-# allows to import into the staging directory the C library and
-# companion libraries for the correct architecture variant. We
-# explictly only copy etc, lib, sbin and usr since other directories
+# We start by copying etc, 'lib', sbin, usr and usr/'lib' from the
+# sysroot of the selected architecture variant (as pointed to by
+# ARCH_SYSROOT_DIR). This allows to import into the staging directory
+# the C library and companion libraries for the correct architecture
+# variant. 'lib' may not be literally 'lib' but could be something else,
+# e.g. lib32-fp (as determined by ARCH_LIB_DIR) and we only want to copy
+# that lib directory and no other. When copying usr, we therefore need
+# to be extra careful not to include usr/lib* but we _do_ want to
+# include usr/libexec.
+# We are selective in the directories we copy since other directories
 # might exist for other architecture variants (on Codesourcery
 # toolchain, the sysroot for the default architecture variant contains
 # the armv4t and thumb2 subdirectories, which are the sysroot for the
@@ -90,9 +95,14 @@ copy_toolchain_sysroot = \
 	SUPPORT_LIB_DIR="$(strip $5)" ; \
 	for i in etc $${ARCH_LIB_DIR} sbin usr usr/$${ARCH_LIB_DIR}; do \
 		if [ -d $${ARCH_SYSROOT_DIR}/$$i ] ; then \
-			rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale' \
-				--include '/libexec*/' --exclude '/lib*/' \
-				$${ARCH_SYSROOT_DIR}/$$i/ $(STAGING_DIR)/$$i/ ; \
+			if [ "$$i" = "usr" ]; then \
+				rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale'\
+					--include '/libexec*/' --exclude '/lib*/' \
+					$${ARCH_SYSROOT_DIR}/$$i/ $(STAGING_DIR)/$$i/ ; \
+			else \
+				rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale'\
+					$${ARCH_SYSROOT_DIR}/$$i/ $(STAGING_DIR)/$$i/ ; \
+			fi ; \
 		fi ; \
 	done ; \
 	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
-- 
2.10.2

[Buildroot] [PATCHv2 02/10] toolchain-external: handle ld.so fixups centrally

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

Normally, the Buildroot toolchain logic copies all required libraries from
the external toolchain to the staging directory, including the dynamic
loader ld-*.so.
There are cases, however, where the dynamic loader is _not_ automatically
copied to staging. This happens when the dynamic loader is not inside
ARCH_LIB_DIR itself (e.g. lib64), but instead resides in 'lib' (assume, of
course, that ARCH_LIB_DIR != 'lib').

Currently, this is fixed in a toolchain-specific fixup, e.g. by recreating a
missing symlink or copying over a missing file. Such toolchain specific
fixups are not very nice.

Moreover, in a subsequent patch, the value of ARCH_LIB_DIR changes for some
toolchains, causing them to have the same problem of a missing dynamic
loader. This used to be the case for older Linaro toolchains with libraries
in 'lib/<tuple>': Buildroot used to set ARCH_LIB_DIR=lib but the mentioned
patch changes it to 'lib/<tuple>' instead. As a result, the files directly
under 'lib/' will no longer be copied. There should be none, but the dynamic
loader is a notable exception.
[Note: support for these older Linaro toolchain has been removed in 2016.11]

Instead, copy over the ld.so file(s)/link(s) from the extracted toolchain
into staging, in the central copy_toolchain_sysroot function. The existing
toolchain logic will then handle the copy of these files from staging to
target.
This means the toolchain-specific fixups can be removed.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: no changes

 toolchain/helpers.mk                                          | 11 +++++++++++
 .../toolchain-external-codesourcery-aarch64.mk                | 10 ----------
 2 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 06dc197..6720629 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -55,6 +55,11 @@ copy_toolchain_lib_root = \
 # corresponding architecture variants), and we don't want to import
 # them.
 #
+# It is possible that ARCH_LIB_DIR does not contain the dynamic loader
+# (ld*.so or similar) because it (or the main symlink to it) normally
+# resides in /lib while ARCH_LIB_DIR may be something else (e.g. lib64).
+# Therefore, copy the dynamic loader separately.
+#
 # Then, if the selected architecture variant is not the default one
 # (i.e, if SYSROOT_DIR != ARCH_SYSROOT_DIR), then we :
 #
@@ -105,6 +110,12 @@ copy_toolchain_sysroot = \
 			fi ; \
 		fi ; \
 	done ; \
+	if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so ]; then \
+		cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so $(STAGING_DIR)/lib/ ; \
+	fi ; \
+	if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so.* ]; then \
+		cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so.* $(STAGING_DIR)/lib/ ; \
+	fi ; \
 	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
 		if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
 			cp -a $${SYSROOT_DIR}/usr/include $(STAGING_DIR)/usr ; \
diff --git a/toolchain/toolchain-external/toolchain-external-codesourcery-aarch64/toolchain-external-codesourcery-aarch64.mk b/toolchain/toolchain-external/toolchain-external-codesourcery-aarch64/toolchain-external-codesourcery-aarch64.mk
index bc58c44..5f4453a 100644
--- a/toolchain/toolchain-external/toolchain-external-codesourcery-aarch64/toolchain-external-codesourcery-aarch64.mk
+++ b/toolchain/toolchain-external/toolchain-external-codesourcery-aarch64/toolchain-external-codesourcery-aarch64.mk
@@ -9,14 +9,4 @@ TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_VERSION = 2014.11-95
 TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_SOURCE = aarch64-amd-$(TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_VERSION)-$(TOOLCHAIN_EXTERNAL_PREFIX)-i686-pc-linux-gnu.tar.bz2
 TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_ACTUAL_SOURCE_TARBALL = aarch64-amd-$(TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_VERSION)-$(TOOLCHAIN_EXTERNAL_PREFIX).src.tar.bz2
 
-define TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_STAGING_FIXUP
-	ln -sf ld-2.20.so $(STAGING_DIR)/lib/ld-linux-aarch64.so.1
-endef
-TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_POST_INSTALL_STAGING_HOOKS += TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_STAGING_FIXUP
-
-define TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_TARGET_FIXUP
-	ln -sf ld-2.20.so $(TARGET_DIR)/lib/ld-linux-aarch64.so.1
-endef
-TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_POST_INSTALL_TARGET_HOOKS += TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_TARGET_FIXUP
-
 $(eval $(toolchain-external-package))
-- 
2.10.2

[Buildroot] [PATCHv2 03/10] toolchain helpers: introduce function relpath_prefix

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

The helper function copy_toolchain_sysroot has some logic to transform a
path into a number of '../' components based on the depth of that path.

As this same logic will be needed in another place in a subsequent patch,
extract it into a separate helper relpath_prefix.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: no changes

 toolchain/helpers.mk | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 6720629..ba14b38 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -121,11 +121,7 @@ copy_toolchain_sysroot = \
 			cp -a $${SYSROOT_DIR}/usr/include $(STAGING_DIR)/usr ; \
 		fi ; \
 		mkdir -p `dirname $(STAGING_DIR)/$${ARCH_SUBDIR}` ; \
-		relpath="./" ; \
-		nbslashs=`printf $${ARCH_SUBDIR} | sed 's%[^/]%%g' | wc -c` ; \
-		for slash in `seq 1 $${nbslashs}` ; do \
-			relpath=$${relpath}"../" ; \
-		done ; \
+		relpath="$(call relpath_prefix,$${ARCH_SUBDIR})./" ; \
 		ln -s $${relpath} $(STAGING_DIR)/$${ARCH_SUBDIR} ; \
 		echo "Symlinking $(STAGING_DIR)/$${ARCH_SUBDIR} -> $${relpath}" ; \
 	fi ; \
@@ -424,3 +420,22 @@ check_toolchain_ssp = \
 gen_gdbinit_file = \
 	mkdir -p $(STAGING_DIR)/usr/share/buildroot/ ; \
 	echo "set sysroot $(STAGING_DIR)" > $(STAGING_DIR)/usr/share/buildroot/gdbinit
+
+# Given a path, determine the relative prefix (../) needed to return to the
+# root level. Note that the last component is treated as a file component; use a
+# trailing slash to force treating it as a directory. Examples:
+#     relpath_prefix(lib32) = ""
+#     relpath_prefix(lib32/octeon2) = "../"
+#     relpath_prefix(lib32/octeon2/) = "../../"
+#
+# $1: input path
+define relpath_prefix
+$$( \
+	prefix="" ; \
+	nbslashs=`printf $1 | sed 's%[^/]%%g' | wc -c` ; \
+	for slash in `seq 1 $${nbslashs}` ; do \
+		prefix=$${prefix}"../" ; \
+	done ; \
+	printf "$$prefix" ;\
+)
+endef
-- 
2.10.2

[Buildroot] [PATCHv2 04/10] toolchain-external: cover multilib toolchains with lib/<variant> layout

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

The toolchain from the Cavium Octeon SDK has a sysroot layout as follows:

./lib32
./lib32/octeon2
./lib32-fp
./lib64
./lib64/octeon2
./lib64-fp
./usr
./usr/lib
./usr/lib32
./usr/lib32/octeon2
./usr/lib32-fp
./usr/lib64
./usr/lib64/octeon2
./usr/lib64-fp
./usr/bin
./usr/bin32
./usr/bin32-fp
./usr/bin64-fp
./usr/libexec
./usr/libexec32
./usr/libexec32-fp
./usr/libexec64-fp
./usr/sbin
./usr/sbin32
./usr/sbin32-fp
./usr/sbin64-fp
./usr/include
./usr/share
./sbin
./sbin32
./sbin32-fp
./sbin64-fp
./etc
./var

with the following selections:
- lib64          : default
- lib64/octeon2  : -march=octeon2
- lib64-fp       : -march=octeon3
- lib32          : -mabi=n32
- lib32/octeon2  : -mabi=n32 -march=octeon2
- lib32-fp       : -mabi=n32 -march=octeon3

In case of '-mabi=n32 -march=octeon2' (but same is true for n64+octeon2)the
original Buildroot toolchain logic would copy both the libraries in
lib32 as the subdirectory lib32/octeon2, which means that every library is
installed twice (but only one of each is really needed).

While ARCH_LIB_DIR is determined by the location of libc.a, which in this
case is effectively:
    <sysroot>/usr/lib32/octeon2/libc.a
the variable only retains 'lib32' and not 'lib32/octeon2' as expected.

To make Buildroot cope with this style of toolchain layout, we need to adapt
the calculation of ARCH_LIB_DIR to also include the second part.
This, in turn, means that ARCH_LIB_DIR is no longer guaranteed to be a
singular path component, resulting in some additional changes.

Certain older Linaro toolchains actually had the same layout. Libraries were
located in lib/<tuple> rather than lib directly. Previously, this was
handled by adding a toolchain-specific fixup that creates a symlink
lib/<tuple> -> lib, but with this patch this would no longer be needed.
Note that one difference with the Octeon case is that these Linaro
toolchains are not actually multilib, i.e. there is just one location with
the libraries and thus there is no problem with duplicated libraries.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: fix broken symbolic links in usr/lib/octeon2/

 toolchain/helpers.mk                               | 22 ++++++++++++++++++++--
 .../toolchain-external/pkg-toolchain-external.mk   | 10 +++++++---
 2 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index ba14b38..1e775ea 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -55,10 +55,19 @@ copy_toolchain_lib_root = \
 # corresponding architecture variants), and we don't want to import
 # them.
 #
+# If ARCH_LIB_DIR is not a singular directory component, e.g.
+# 'lib32/octeon2', then symbolic links in ARCH_LIB_DIR and
+# usr/ARCH_LIB_DIR may be broken because Buildroot will flatten the
+# directory structure (e.g. lib32/octeon2/foo is actually stored in
+# lib/foo). This is only relevant for links that contain one or more ../
+# components, as links to the current directory are always fine.
+# We need to fix the broken links by removing the right amount of ../
+# dots from the link destination.
+#
 # It is possible that ARCH_LIB_DIR does not contain the dynamic loader
 # (ld*.so or similar) because it (or the main symlink to it) normally
-# resides in /lib while ARCH_LIB_DIR may be something else (e.g. lib64).
-# Therefore, copy the dynamic loader separately.
+# resides in /lib while ARCH_LIB_DIR may be something else (e.g. lib64,
+# lib/<tuple>, ...). Therefore, copy the dynamic loader separately.
 #
 # Then, if the selected architecture variant is not the default one
 # (i.e, if SYSROOT_DIR != ARCH_SYSROOT_DIR), then we :
@@ -103,6 +112,7 @@ copy_toolchain_sysroot = \
 			if [ "$$i" = "usr" ]; then \
 				rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale'\
 					--include '/libexec*/' --exclude '/lib*/' \
+					--exclude '/$${ARCH_LIB_DIR}' \
 					$${ARCH_SYSROOT_DIR}/$$i/ $(STAGING_DIR)/$$i/ ; \
 			else \
 				rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale'\
@@ -110,6 +120,14 @@ copy_toolchain_sysroot = \
 			fi ; \
 		fi ; \
 	done ; \
+	relpath="$(call relpath_prefix,$${ARCH_LIB_DIR})" ; \
+	if [ "$${relpath}" != "" ]; then \
+		for i in $$(find -H $(STAGING_DIR)/$${ARCH_LIB_DIR} $(STAGING_DIR)/usr/$${ARCH_LIB_DIR} -type l -xtype l); do \
+			LINKTARGET=`readlink $$i` ; \
+			NEWLINKTARGET=$${LINKTARGET\#$$relpath} ; \
+			ln -sf $${NEWLINKTARGET} $$i ; \
+		done ; \
+	fi ; \
 	if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so ]; then \
 		cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so $(STAGING_DIR)/lib/ ; \
 	fi ; \
diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
index 6658875..ada767e 100644
--- a/toolchain/toolchain-external/pkg-toolchain-external.mk
+++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
@@ -332,7 +332,7 @@ endef
 # Returns the lib subdirectory for the given compiler + flags (i.e
 # typically lib32 or lib64 for some toolchains)
 define toolchain_find_libdir
-$$(printf $(call toolchain_find_libc_a,$(1)) | sed -r -e 's:.*/(usr/)?(lib(32|64)?([^/]*)?)/([^/]*/)?libc.a:\2:')
+$$(printf $(call toolchain_find_libc_a,$(1)) | sed -r -e 's:.*/(usr/)?(lib(32|64)?([^/]*)?(/[^/]*)?)/libc.a:\2:')
 endef
 
 # Returns the location of the libc.a file for the given compiler + flags
@@ -442,14 +442,18 @@ endef
 # Create a symlink from (usr/)$(ARCH_LIB_DIR) to lib.
 # Note: the skeleton package additionally creates lib32->lib or lib64->lib
 # (as appropriate)
+# The relpath handling is needed in case ARCH_LIB_DIR is not a singular path
+# like 'lib32-foo' but a composite one, like 'lib/foo'. In either case, we need
+# to link to the top-level 'lib'.
 #
 # $1: destination directory (TARGET_DIR / STAGING_DIR)
 create_lib_symlinks = \
        $(Q)DESTDIR="$(strip $1)" ; \
        ARCH_LIB_DIR="$(call toolchain_find_libdir,$(TOOLCHAIN_EXTERNAL_CC) $(TOOLCHAIN_EXTERNAL_CFLAGS))" ; \
        if [ ! -e "$${DESTDIR}/$${ARCH_LIB_DIR}" -a ! -e "$${DESTDIR}/usr/$${ARCH_LIB_DIR}" ]; then \
-               ln -snf lib "$${DESTDIR}/$${ARCH_LIB_DIR}" ; \
-               ln -snf lib "$${DESTDIR}/usr/$${ARCH_LIB_DIR}" ; \
+               relpath="$(call relpath_prefix,$${ARCH_LIB_DIR})" ; \
+               ln -snf $${relpath}lib "$${DESTDIR}/$${ARCH_LIB_DIR}" ; \
+               ln -snf $${relpath}lib "$${DESTDIR}/usr/$${ARCH_LIB_DIR}" ; \
        fi
 
 define TOOLCHAIN_EXTERNAL_CREATE_STAGING_LIB_SYMLINK
-- 
2.10.2

[Buildroot] [PATCHv2 05/10] toolchain helpers: introduce simplify_symlink

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

The external toolchain logic flattens the directory layout in the staging
directory.  Regardless of the number of levels present in the extracted
toolchain, libraries are always copied to lib/ and usr/lib/, and directory
symbolic links are provided to make the original paths available as well.

Due to this, the same library may be reachable through a number of paths:
one path without any symbolic link, and one or more paths using directory
symlinks.

Using a direct path in a symlink destination is generally preferred because
it is clearer, but it is also more robust against accidental removal of an
intermediate directory symlink.

Introduce a helper function to simplify a symlink's destination to such a
direct path. This function will be used in a subsequent patch.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: new patch

 toolchain/helpers.mk | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 1e775ea..0acbb52 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -457,3 +457,30 @@ $$( \
 	printf "$$prefix" ;\
 )
 endef
+
+# Replace the destination of a symbolic link with a simpler version
+# For example,
+#     usr/lib/libfoo.so -> ../../lib32/libfoo.so.1
+# becomes
+#     usr/lib/libfoo.so -> ../../lib/libfoo.so.1
+# since 'lib32' is a symlink to 'lib'.
+#
+# Likewise,
+#     usr/lib/octeon2/libbar.so -> ../../../lib32/octeon2/libbar.so.1
+# becomes
+#     usr/lib/octeon2/libbar.so -> ../../lib/libbar.so.1
+# assuming lib32->lib and lib/octeon2->lib.
+#
+# $1: symlink
+# $2: base path
+define simplify_symlink
+( \
+	FULL_SRC="$$(readlink -f $$(dirname $1))/$$(basename $1)" ; \
+	FULL_DEST="$$(readlink -f $1)" ; \
+	FULL_BASE="$$(readlink -f $2)" ; \
+	REL_SRC="$${FULL_SRC#$${FULL_BASE}/}" ; \
+	REL_DEST="$${FULL_DEST#$${FULL_BASE}/}" ; \
+	DOTS="$(call relpath_prefix,$${REL_SRC})" ; \
+	ln -sf "$${DOTS}$${REL_DEST}" "$${FULL_SRC}" ; \
+)
+endef
-- 
2.10.2

[Buildroot] [PATCHv2 06/10] toolchain-external: simplify previously-broken symbolic links

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

A previous commit rewrote broken symbolic links in staging, caused by a
non-singular ARCH_LIB_DIR. In this case, the symbolic links are typically
using one or more intermediate directory symlinks, which can be simplified
using the newly introduced simplify_symlink helper.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: new patch

 toolchain/helpers.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 0acbb52..611aaab 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -63,6 +63,8 @@ copy_toolchain_lib_root = \
 # components, as links to the current directory are always fine.
 # We need to fix the broken links by removing the right amount of ../
 # dots from the link destination.
+# Once the link destination is valid again, it can be simplified to
+# remove the dependency on intermediate directory symlinks.
 #
 # It is possible that ARCH_LIB_DIR does not contain the dynamic loader
 # (ld*.so or similar) because it (or the main symlink to it) normally
@@ -126,6 +128,7 @@ copy_toolchain_sysroot = \
 			LINKTARGET=`readlink $$i` ; \
 			NEWLINKTARGET=$${LINKTARGET\#$$relpath} ; \
 			ln -sf $${NEWLINKTARGET} $$i ; \
+			$(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
 		done ; \
 	fi ; \
 	if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so ]; then \
-- 
2.10.2

[Buildroot] [PATCHv2 07/10] toolchain: copy_toolchain_lib_root: remove unused variable LIBDIR

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: new patch

 toolchain/helpers.mk | 1 -
 1 file changed, 1 deletion(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 611aaab..dab9cd3 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -17,7 +17,6 @@ copy_toolchain_lib_root = \
 		mkdir -p $(TARGET_DIR)/$${DESTDIR}; \
 		while true ; do \
 			LIBNAME=`basename $${LIBPATH}`; \
-			LIBDIR=`dirname $${LIBPATH}` ; \
 			LINKTARGET=`readlink $${LIBPATH}` ; \
 			rm -fr $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
 			if test -h $${LIBPATH} ; then \
-- 
2.10.2

[Buildroot] [PATCHv2 08/10] toolchain: copy_toolchain_lib_root: clarify logic

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

copy_toolchain_lib_root has slightly different logic depending on the type
of library object: file or link. All actions related to links are not
relevant in case you are working with a file. Hence, try to increase clarity
by not executing unnecessary lines in the 'file' case.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: new patch

 toolchain/helpers.mk | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index dab9cd3..3e4c581 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -17,19 +17,17 @@ copy_toolchain_lib_root = \
 		mkdir -p $(TARGET_DIR)/$${DESTDIR}; \
 		while true ; do \
 			LIBNAME=`basename $${LIBPATH}`; \
-			LINKTARGET=`readlink $${LIBPATH}` ; \
 			rm -fr $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
 			if test -h $${LIBPATH} ; then \
+				LINKTARGET=`readlink $${LIBPATH}` ; \
 				ln -sf `basename $${LINKTARGET}` $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME} ; \
+				LIBPATH="`readlink -f $${LIBPATH}`"; \
 			elif test -f $${LIBPATH}; then \
 				$(INSTALL) -D -m0755 $${LIBPATH} $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
+				break ; \
 			else \
 				exit -1; \
 			fi; \
-			if test -z "$${LINKTARGET}" ; then \
-				break ; \
-			fi ; \
-			LIBPATH="`readlink -f $${LIBPATH}`"; \
 		done; \
 	done
 
-- 
2.10.2

[Buildroot] [PATCHv2 09/10] toolchain: copy_toolchain_lib_root: clarify input parameter

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

The input to copy_toolchain_lib_root is not one library, not a list of
libraries, but a library name pattern with glob wildcards.
This pattern is then passed to 'find' to get the actual list of libraries
matching the pattern. Reflect this using an appropriate variable name.

Note: if the root of the buildroot tree contains a file matching one of
these library patterns, the copying of libraries from staging to target will
not be correct. It is not impossible to fix that, e.g. using 'set -f', but
maybe it's not worth it.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: new patch

 package/glibc/glibc.mk                                 | 4 ++--
 toolchain/helpers.mk                                   | 6 +++---
 toolchain/toolchain-external/pkg-toolchain-external.mk | 4 ++--
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
index c11f7cd..2da9606 100644
--- a/package/glibc/glibc.mk
+++ b/package/glibc/glibc.mk
@@ -110,8 +110,8 @@ GLIBC_LIBS_LIB += libthread_db.so.*
 endif
 
 define GLIBC_INSTALL_TARGET_CMDS
-	for libs in $(GLIBC_LIBS_LIB); do \
-		$(call copy_toolchain_lib_root,$$libs) ; \
+	for libpattern in $(GLIBC_LIBS_LIB); do \
+		$(call copy_toolchain_lib_root,$$libpattern) ; \
 	done
 endef
 
diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 3e4c581..eb89e02 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -6,12 +6,12 @@
 # toolchain logic, and the glibc package, so care must be taken when
 # changing this function.
 #
-# $1: library name
+# $1: library name pattern (can include glob wildcards)
 #
 copy_toolchain_lib_root = \
-	LIB="$(strip $1)"; \
+	LIBPATTERN="$(strip $1)"; \
 \
-	LIBPATHS=`find $(STAGING_DIR)/ -name "$${LIB}" 2>/dev/null` ; \
+	LIBPATHS=`find $(STAGING_DIR)/ -name "$${LIBPATTERN}" 2>/dev/null` ; \
 	for LIBPATH in $${LIBPATHS} ; do \
 		DESTDIR=`echo $${LIBPATH} | sed "s,^$(STAGING_DIR)/,," | xargs dirname` ; \
 		mkdir -p $(TARGET_DIR)/$${DESTDIR}; \
diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
index ada767e..d62e2cf 100644
--- a/toolchain/toolchain-external/pkg-toolchain-external.mk
+++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
@@ -387,8 +387,8 @@ endef
 ifeq ($(BR2_STATIC_LIBS),)
 define TOOLCHAIN_EXTERNAL_INSTALL_TARGET_LIBS
 	$(Q)$(call MESSAGE,"Copying external toolchain libraries to target...")
-	$(Q)for libs in $(TOOLCHAIN_EXTERNAL_LIBS); do \
-		$(call copy_toolchain_lib_root,$$libs); \
+	$(Q)for libpattern in $(TOOLCHAIN_EXTERNAL_LIBS); do \
+		$(call copy_toolchain_lib_root,$$libpattern); \
 	done
 endef
 endif
-- 
2.10.2

[Buildroot] [PATCHv2 10/10] toolchain: copy_toolchain_lib_root: copy symlinks instead of recreating them

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>

copy_toolchain_lib_root handles symlinks by recreating them, disregarding
the original destination and assuming the destination is in the same
directory as the link itself.
When a library link points to the real library file in another directory,
for example:
    usr/lib/octeon2/libcrypt.so -> ../../../lib32/octeon2/libcrypt.so.1
then the link created by copy_toolchain_lib_root is broken.

It is more robust to copy the symlink to keep the destination intact. The
destination path should be present, possibly through other symbolic links.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
---
v2: new patch

 toolchain/helpers.mk | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index eb89e02..7f64680 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -19,8 +19,7 @@ copy_toolchain_lib_root = \
 			LIBNAME=`basename $${LIBPATH}`; \
 			rm -fr $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
 			if test -h $${LIBPATH} ; then \
-				LINKTARGET=`readlink $${LIBPATH}` ; \
-				ln -sf `basename $${LINKTARGET}` $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME} ; \
+				cp -d $${LIBPATH} $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
 				LIBPATH="`readlink -f $${LIBPATH}`"; \
 			elif test -f $${LIBPATH}; then \
 				$(INSTALL) -D -m0755 $${LIBPATH} $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
-- 
2.10.2

[Buildroot] [PATCHv2 01/10] toolchain-external: clarify rsync excludes in copy_toolchain_sysroot

[Romain Naour]

Hi Thomas,

Le 04/02/2017 ? 14:42, Thomas De Schampheleire a ?crit :
> From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
> 
> The copy_toolchain_sysroot helper features a complex rsync loop that copies
> various directories from the extracted toolchain to the staging directory.
> The complexity mainly stems from the fact that we support multilib toolchain
> tarballs but only copy one of the multilib variants into staging.
> 
> Increase understandability of this logic by explicitly restricting the
> rsync excludes to the iteration of the for loop they are relevant for.
> Additionally, update the function comment.
> 
> Note: all attempts to reduce duplication between both rsync while keeping
> things nice and readable failed. One has to be extremely careful regarding
> line continuation, indentation, and single vs double quoting. In the end, a
> split up rsync seemed most clean.
> 
> Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
> ---
> v2: no changes
> 
>  toolchain/helpers.mk | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
> 
> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> index 72e7292..06dc197 100644
> --- a/toolchain/helpers.mk
> +++ b/toolchain/helpers.mk
> @@ -39,11 +39,16 @@ copy_toolchain_lib_root = \
>  # dir. The operation of this function is rendered a little bit
>  # complicated by the support for multilib toolchains.
>  #
> -# We start by copying etc, lib, sbin and usr from the sysroot of the
> -# selected architecture variant (as pointed by ARCH_SYSROOT_DIR). This
> -# allows to import into the staging directory the C library and
> -# companion libraries for the correct architecture variant. We
> -# explictly only copy etc, lib, sbin and usr since other directories
> +# We start by copying etc, 'lib', sbin, usr and usr/'lib' from the
> +# sysroot of the selected architecture variant (as pointed to by
> +# ARCH_SYSROOT_DIR). This allows to import into the staging directory
> +# the C library and companion libraries for the correct architecture
> +# variant. 'lib' may not be literally 'lib' but could be something else,
> +# e.g. lib32-fp (as determined by ARCH_LIB_DIR) and we only want to copy
> +# that lib directory and no other. When copying usr, we therefore need
> +# to be extra careful not to include usr/lib* but we _do_ want to
> +# include usr/libexec.
> +# We are selective in the directories we copy since other directories
>  # might exist for other architecture variants (on Codesourcery
>  # toolchain, the sysroot for the default architecture variant contains
>  # the armv4t and thumb2 subdirectories, which are the sysroot for the
> @@ -90,9 +95,14 @@ copy_toolchain_sysroot = \
>  	SUPPORT_LIB_DIR="$(strip $5)" ; \
>  	for i in etc $${ARCH_LIB_DIR} sbin usr usr/$${ARCH_LIB_DIR}; do \
>  		if [ -d $${ARCH_SYSROOT_DIR}/$$i ] ; then \
> -			rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale' \
> -				--include '/libexec*/' --exclude '/lib*/' \
> -				$${ARCH_SYSROOT_DIR}/$$i/ $(STAGING_DIR)/$$i/ ; \
> +			if [ "$$i" = "usr" ]; then \
> +				rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale'\
> +					--include '/libexec*/' --exclude '/lib*/' \
> +					$${ARCH_SYSROOT_DIR}/$$i/ $(STAGING_DIR)/$$i/ ; \
> +			else \
> +				rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale'\

Not related to your patch but I'm wondering if --exclude 'usr/lib/locale' is
still relevant. It was added to rsync with commit [1] but in seems to be already
excluded while extracting the toolchain (BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y case).
Maybe we still need it for BR2_TOOLCHAIN_EXTERNAL_PREINSTALLED=y case.

Also, in "$$i" != "usr" case it's seems not needed.

Best regards,
Romain

[1] 2c23e937660f43131f68f6cbf2ff3fe3d0417032

> +					$${ARCH_SYSROOT_DIR}/$$i/ $(STAGING_DIR)/$$i/ ; \
> +			fi ; \
>  		fi ; \
>  	done ; \
>  	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
> 

[Buildroot] [PATCHv2 02/10] toolchain-external: handle ld.so fixups centrally

[Romain Naour]

Hi Thomas,

Le 04/02/2017 ? 14:42, Thomas De Schampheleire a ?crit :
> From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
> 
> Normally, the Buildroot toolchain logic copies all required libraries from
> the external toolchain to the staging directory, including the dynamic
> loader ld-*.so.
> There are cases, however, where the dynamic loader is _not_ automatically
> copied to staging. This happens when the dynamic loader is not inside
> ARCH_LIB_DIR itself (e.g. lib64), but instead resides in 'lib' (assume, of
> course, that ARCH_LIB_DIR != 'lib').
> 
> Currently, this is fixed in a toolchain-specific fixup, e.g. by recreating a
> missing symlink or copying over a missing file. Such toolchain specific
> fixups are not very nice.

Indeed, like the one I introduced in [1]. The CodeSourcery AArch64 toolchain
(which is still in Buildroot) was broken for more than one year, then ThomasP
applied the patch...

[1] 4d39ca1c2ab19766abf0430abe9b65ba0e16602b

> 
> Moreover, in a subsequent patch, the value of ARCH_LIB_DIR changes for some
> toolchains, causing them to have the same problem of a missing dynamic
> loader. This used to be the case for older Linaro toolchains with libraries
> in 'lib/<tuple>': Buildroot used to set ARCH_LIB_DIR=lib but the mentioned
> patch changes it to 'lib/<tuple>' instead. As a result, the files directly
> under 'lib/' will no longer be copied. There should be none, but the dynamic
> loader is a notable exception.
> [Note: support for these older Linaro toolchain has been removed in 2016.11]
> 
> Instead, copy over the ld.so file(s)/link(s) from the extracted toolchain
> into staging, in the central copy_toolchain_sysroot function. The existing
> toolchain logic will then handle the copy of these files from staging to
> target.
> This means the toolchain-specific fixups can be removed.
> 
> Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
> ---
> v2: no changes
> 
>  toolchain/helpers.mk                                          | 11 +++++++++++
>  .../toolchain-external-codesourcery-aarch64.mk                | 10 ----------
>  2 files changed, 11 insertions(+), 10 deletions(-)
> 
> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> index 06dc197..6720629 100644
> --- a/toolchain/helpers.mk
> +++ b/toolchain/helpers.mk
> @@ -55,6 +55,11 @@ copy_toolchain_lib_root = \
>  # corresponding architecture variants), and we don't want to import
>  # them.
>  #
> +# It is possible that ARCH_LIB_DIR does not contain the dynamic loader
> +# (ld*.so or similar) because it (or the main symlink to it) normally
> +# resides in /lib while ARCH_LIB_DIR may be something else (e.g. lib64).
> +# Therefore, copy the dynamic loader separately.
> +#
>  # Then, if the selected architecture variant is not the default one
>  # (i.e, if SYSROOT_DIR != ARCH_SYSROOT_DIR), then we :
>  #
> @@ -105,6 +110,12 @@ copy_toolchain_sysroot = \
>  			fi ; \
>  		fi ; \
>  	done ; \
> +	if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so ]; then \
> +		cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so $(STAGING_DIR)/lib/ ; \
> +	fi ; \
> +	if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so.* ]; then \
> +		cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so.* $(STAGING_DIR)/lib/ ; \
> +	fi ; \

It's far better than having ld-*.so version hardcoded in the .mk...

Best regards,
Romain


>  	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
>  		if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
>  			cp -a $${SYSROOT_DIR}/usr/include $(STAGING_DIR)/usr ; \
> diff --git a/toolchain/toolchain-external/toolchain-external-codesourcery-aarch64/toolchain-external-codesourcery-aarch64.mk b/toolchain/toolchain-external/toolchain-external-codesourcery-aarch64/toolchain-external-codesourcery-aarch64.mk
> index bc58c44..5f4453a 100644
> --- a/toolchain/toolchain-external/toolchain-external-codesourcery-aarch64/toolchain-external-codesourcery-aarch64.mk
> +++ b/toolchain/toolchain-external/toolchain-external-codesourcery-aarch64/toolchain-external-codesourcery-aarch64.mk
> @@ -9,14 +9,4 @@ TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_VERSION = 2014.11-95
>  TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_SOURCE = aarch64-amd-$(TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_VERSION)-$(TOOLCHAIN_EXTERNAL_PREFIX)-i686-pc-linux-gnu.tar.bz2
>  TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_ACTUAL_SOURCE_TARBALL = aarch64-amd-$(TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_VERSION)-$(TOOLCHAIN_EXTERNAL_PREFIX).src.tar.bz2
>  
> -define TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_STAGING_FIXUP
> -	ln -sf ld-2.20.so $(STAGING_DIR)/lib/ld-linux-aarch64.so.1
> -endef
> -TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_POST_INSTALL_STAGING_HOOKS += TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_STAGING_FIXUP
> -
> -define TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_TARGET_FIXUP
> -	ln -sf ld-2.20.so $(TARGET_DIR)/lib/ld-linux-aarch64.so.1
> -endef
> -TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_POST_INSTALL_TARGET_HOOKS += TOOLCHAIN_EXTERNAL_CODESOURCERY_AARCH64_TARGET_FIXUP
> -
>  $(eval $(toolchain-external-package))
> 

[Buildroot] [PATCHv2 03/10] toolchain helpers: introduce function relpath_prefix

[Romain Naour]

Hi Thomas,

Le 04/02/2017 ? 14:42, Thomas De Schampheleire a ?crit :
> From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
> 
> The helper function copy_toolchain_sysroot has some logic to transform a
> path into a number of '../' components based on the depth of that path.
> 
> As this same logic will be needed in another place in a subsequent patch,
> extract it into a separate helper relpath_prefix.
> 
> Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
> ---
> v2: no changes
> 
>  toolchain/helpers.mk | 25 ++++++++++++++++++++-----
>  1 file changed, 20 insertions(+), 5 deletions(-)
> 
> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> index 6720629..ba14b38 100644
> --- a/toolchain/helpers.mk
> +++ b/toolchain/helpers.mk
> @@ -121,11 +121,7 @@ copy_toolchain_sysroot = \
>  			cp -a $${SYSROOT_DIR}/usr/include $(STAGING_DIR)/usr ; \
>  		fi ; \
>  		mkdir -p `dirname $(STAGING_DIR)/$${ARCH_SUBDIR}` ; \
> -		relpath="./" ; \
> -		nbslashs=`printf $${ARCH_SUBDIR} | sed 's%[^/]%%g' | wc -c` ; \
> -		for slash in `seq 1 $${nbslashs}` ; do \
> -			relpath=$${relpath}"../" ; \
> -		done ; \
> +		relpath="$(call relpath_prefix,$${ARCH_SUBDIR})./" ; \
>  		ln -s $${relpath} $(STAGING_DIR)/$${ARCH_SUBDIR} ; \
>  		echo "Symlinking $(STAGING_DIR)/$${ARCH_SUBDIR} -> $${relpath}" ; \
>  	fi ; \
> @@ -424,3 +420,22 @@ check_toolchain_ssp = \
>  gen_gdbinit_file = \
>  	mkdir -p $(STAGING_DIR)/usr/share/buildroot/ ; \
>  	echo "set sysroot $(STAGING_DIR)" > $(STAGING_DIR)/usr/share/buildroot/gdbinit
> +
> +# Given a path, determine the relative prefix (../) needed to return to the
> +# root level. Note that the last component is treated as a file component; use a
> +# trailing slash to force treating it as a directory. Examples:
> +#     relpath_prefix(lib32) = ""
> +#     relpath_prefix(lib32/octeon2) = "../"
> +#     relpath_prefix(lib32/octeon2/) = "../../"
> +#
> +# $1: input path
> +define relpath_prefix
> +$$( \
> +	prefix="" ; \
> +	nbslashs=`printf $1 | sed 's%[^/]%%g' | wc -c` ; \
> +	for slash in `seq 1 $${nbslashs}` ; do \
> +		prefix=$${prefix}"../" ; \
> +	done ; \
> +	printf "$$prefix" ;\

missing space after ";"

Best regards,
Romain


> +)
> +endef
> 

[Buildroot] [PATCHv2 01/10] toolchain-external: clarify rsync excludes in copy_toolchain_sysroot

[Thomas De Schampheleire]

On Tue, Feb 7, 2017 at 11:26 AM, Romain Naour <romain.naour@gmail.com> wrote:
> Hi Thomas,
>
> Le 04/02/2017 ? 14:42, Thomas De Schampheleire a ?crit :
>> From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
>>
>> The copy_toolchain_sysroot helper features a complex rsync loop that copies
>> various directories from the extracted toolchain to the staging directory.
>> The complexity mainly stems from the fact that we support multilib toolchain
>> tarballs but only copy one of the multilib variants into staging.
>>
>> Increase understandability of this logic by explicitly restricting the
>> rsync excludes to the iteration of the for loop they are relevant for.
>> Additionally, update the function comment.
>>
>> Note: all attempts to reduce duplication between both rsync while keeping
>> things nice and readable failed. One has to be extremely careful regarding
>> line continuation, indentation, and single vs double quoting. In the end, a
>> split up rsync seemed most clean.
>>
>> Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
>> ---
>> v2: no changes
>>
>>  toolchain/helpers.mk | 26 ++++++++++++++++++--------
>>  1 file changed, 18 insertions(+), 8 deletions(-)
>>
>> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
>> index 72e7292..06dc197 100644
>> --- a/toolchain/helpers.mk
>> +++ b/toolchain/helpers.mk
>> @@ -39,11 +39,16 @@ copy_toolchain_lib_root = \
>>  # dir. The operation of this function is rendered a little bit
>>  # complicated by the support for multilib toolchains.
>>  #
>> -# We start by copying etc, lib, sbin and usr from the sysroot of the
>> -# selected architecture variant (as pointed by ARCH_SYSROOT_DIR). This
>> -# allows to import into the staging directory the C library and
>> -# companion libraries for the correct architecture variant. We
>> -# explictly only copy etc, lib, sbin and usr since other directories
>> +# We start by copying etc, 'lib', sbin, usr and usr/'lib' from the
>> +# sysroot of the selected architecture variant (as pointed to by
>> +# ARCH_SYSROOT_DIR). This allows to import into the staging directory
>> +# the C library and companion libraries for the correct architecture
>> +# variant. 'lib' may not be literally 'lib' but could be something else,
>> +# e.g. lib32-fp (as determined by ARCH_LIB_DIR) and we only want to copy
>> +# that lib directory and no other. When copying usr, we therefore need
>> +# to be extra careful not to include usr/lib* but we _do_ want to
>> +# include usr/libexec.
>> +# We are selective in the directories we copy since other directories
>>  # might exist for other architecture variants (on Codesourcery
>>  # toolchain, the sysroot for the default architecture variant contains
>>  # the armv4t and thumb2 subdirectories, which are the sysroot for the
>> @@ -90,9 +95,14 @@ copy_toolchain_sysroot = \
>>       SUPPORT_LIB_DIR="$(strip $5)" ; \
>>       for i in etc $${ARCH_LIB_DIR} sbin usr usr/$${ARCH_LIB_DIR}; do \
>>               if [ -d $${ARCH_SYSROOT_DIR}/$$i ] ; then \
>> -                     rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale' \
>> -                             --include '/libexec*/' --exclude '/lib*/' \
>> -                             $${ARCH_SYSROOT_DIR}/$$i/ $(STAGING_DIR)/$$i/ ; \
>> +                     if [ "$$i" = "usr" ]; then \
>> +                             rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale'\
>> +                                     --include '/libexec*/' --exclude '/lib*/' \
>> +                                     $${ARCH_SYSROOT_DIR}/$$i/ $(STAGING_DIR)/$$i/ ; \
>> +                     else \
>> +                             rsync -au --chmod=u=rwX,go=rX --exclude 'usr/lib/locale'\
>
> Not related to your patch but I'm wondering if --exclude 'usr/lib/locale' is
> still relevant. It was added to rsync with commit [1] but in seems to be already
> excluded while extracting the toolchain (BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y case).
> Maybe we still need it for BR2_TOOLCHAIN_EXTERNAL_PREINSTALLED=y case.
>
> Also, in "$$i" != "usr" case it's seems not needed.

I wasn't too sure about this 'usr/lib/locale' thing so did not touch it.
Note that the exclusion does not contain a leading slash, which means
that 'usr/lib/locale' is matched anywhere in the path, so e.g.
usr/share/foo/usr/lib/locale would also be excluded. If we are sure
that the intention is to only match in the beginning, then the code
can indeed be simplified.