[Buildroot] [PATCH RFC 0/2] unify library locations in target/staging

[Buildroot] [PATCH RFC 0/2] unify library locations in target/staging

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

See second patch for full details, first patch is just a minor correction
in variable naming.

This is code has been tested for various toolchains, BUT is not yet correct
for Blackfin toolchains when FDPIC_SHARED is selected (hence: DO NOT APPLY)
For this family there is some specific code that needs to be fixed.

Another improvement is the simplification of the copy_toolchain_lib_root
arguments: only the LIB argument is used now.

Finally, a follow-up patch is expected to unify the variables LIB_EXTERNAL_LIBS
and USR_LIB_EXTERNAL_LIBS, as there is no need for the distinction anymore.

/Thomas



Thomas De Schampheleire (2):
  toolchain: copy_toolchain_lib_root: rename LIBSPATH to LIBPATHS
  toolchain-external: unify library locations in target and staging dir

 toolchain/helpers.mk                               | 18 ++++--------------
 toolchain/toolchain-external/toolchain-external.mk | 14 ++++----------
 2 files changed, 8 insertions(+), 24 deletions(-)

-- 
2.4.10

[Buildroot] [PATCH RFC 1/2] toolchain: copy_toolchain_lib_root: rename LIBSPATH to LIBPATHS

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

LIBSPATH is populated based on a find with a pattern that can look like:
    libfoo*.so
and thus the output of the find will contain all file paths that match this
pattern.

Unfortunately, the name LIBSPATH suggests that only one entry is returned,
rather than possibly multiple.

As this code is quite complex, use the more accurate name LIBPATHS iso
LIBSPATH.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
---
 toolchain/helpers.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 02cc0bb..ee878e8 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -55,13 +55,13 @@ copy_toolchain_lib_root = \
 		$${ARCH_SYSROOT_DIR}/$${ARCH_LIB_DIR} \
 		$${ARCH_SYSROOT_DIR}/usr/$${ARCH_LIB_DIR} \
 		$${SUPPORT_LIB_DIR} ; do \
-		LIBSPATH=`find $${dir} -maxdepth 1 -name "$${LIB}" 2>/dev/null` ; \
-		if test -n "$${LIBSPATH}" ; then \
+		LIBPATHS=`find $${dir} -maxdepth 1 -name "$${LIB}" 2>/dev/null` ; \
+		if test -n "$${LIBPATHS}" ; then \
 			break ; \
 		fi \
 	done ; \
 	mkdir -p $(TARGET_DIR)/$${DESTDIR}; \
-	for LIBPATH in $${LIBSPATH} ; do \
+	for LIBPATH in $${LIBPATHS} ; do \
 		while true ; do \
 			LIBNAME=`basename $${LIBPATH}`; \
 			LIBDIR=`dirname $${LIBPATH}` ; \
-- 
2.4.10

[Buildroot] [PATCH RFC 2/2] toolchain-external: unify library locations in target and staging dir

[Thomas De Schampheleire]

From: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

The toolchain-external logic is roughly:
- populate the staging dir by rsyncing the entire ${ARCH_LIB_DIR} and
  usr/${ARCH_LIB_DIR} from sysroot.
- populate the target dir by explictly copying some libraries from sysroot
  into target/lib and some other libraries in target/usr/lib, the split
  being hardcoded into buildroot regardless of the location in the sysroot.

This means that a library libfoo could be located in:
  staging/lib/libfoo.so
  target/usr/lib/libfoo.so

When debugging an application that links against this library, gdb will
fruitlessly search for 'usr/lib/libfoo.so' in staging, and then suggest to
use 'set solib-search-path' which is a hack, really.

To solve the problem, we need to make sure that libraries from the toolchain
are installed in the same relative location in staging and target.
Achieve this by:
- replacing the convoluted search for libraries using for+find in sysroot
  with a simple find in staging.
- determine DESTDIR for each library individually based on the location in
  staging.
- treating LIB_EXTERNAL_LIBS and USR_LIB_EXTERNAL_LIBS equivalently

Test procedure:
- set configuration for a given toolchain
- make clean toolchain
- find output/target | sort > /tmp/out-before
- apply patch
- make clean toolchain
- find output/target | sort > /tmp/out-after
- diff -u /tmp/out-before /tmp/out-after

The only changes should be some libraries moving from lib to usr/lib or vice
versa. Notable examples being libstdc++ and libatomic.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
---
 toolchain/helpers.mk                               | 16 +++-------------
 toolchain/toolchain-external/toolchain-external.mk | 14 ++++----------
 2 files changed, 7 insertions(+), 23 deletions(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index ee878e8..aa53c7c 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -47,21 +47,11 @@ copy_toolchain_lib_root = \
 	SUPPORT_LIB_DIR="$(strip $2)" ; \
 	ARCH_LIB_DIR="$(strip $3)" ; \
 	LIB="$(strip $4)"; \
-	DESTDIR="$(strip $5)" ; \
 \
-	for dir in \
-		$${ARCH_SYSROOT_DIR}/$${ARCH_LIB_DIR}/$(TOOLCHAIN_EXTERNAL_PREFIX) \
-		$${ARCH_SYSROOT_DIR}/usr/$(TOOLCHAIN_EXTERNAL_PREFIX)/$${ARCH_LIB_DIR} \
-		$${ARCH_SYSROOT_DIR}/$${ARCH_LIB_DIR} \
-		$${ARCH_SYSROOT_DIR}/usr/$${ARCH_LIB_DIR} \
-		$${SUPPORT_LIB_DIR} ; do \
-		LIBPATHS=`find $${dir} -maxdepth 1 -name "$${LIB}" 2>/dev/null` ; \
-		if test -n "$${LIBPATHS}" ; then \
-			break ; \
-		fi \
-	done ; \
-	mkdir -p $(TARGET_DIR)/$${DESTDIR}; \
+	LIBPATHS=`find $(STAGING_DIR) -follow -name "$${LIB}" 2>/dev/null` ; \
 	for LIBPATH in $${LIBPATHS} ; do \
+		DESTDIR=`echo $${LIBPATH} | sed "s,^$(STAGING_DIR)/,," | xargs dirname` ; \
+		mkdir -p $(TARGET_DIR)/$${DESTDIR}; \
 		while true ; do \
 			LIBNAME=`basename $${LIBPATH}`; \
 			LIBDIR=`dirname $${LIBPATH}` ; \
diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
index 518afd6..c2076d9 100644
--- a/toolchain/toolchain-external/toolchain-external.mk
+++ b/toolchain/toolchain-external/toolchain-external.mk
@@ -604,11 +604,8 @@ define TOOLCHAIN_EXTERNAL_INSTALL_TARGET_LIBS
 	ARCH_SUBDIR=`echo $${ARCH_SYSROOT_DIR} | sed -r -e "s:^$${SYSROOT_DIR}(.*)/$$:\1:"` ; \
 	if test -z "$(BR2_STATIC_LIBS)" ; then \
 		$(call MESSAGE,"Copying external toolchain libraries to target...") ; \
-		for libs in $(LIB_EXTERNAL_LIBS); do \
-			$(call copy_toolchain_lib_root,$${ARCH_SYSROOT_DIR},$${SUPPORT_LIB_DIR},$${ARCH_LIB_DIR},$$libs,/lib); \
-		done ; \
-		for libs in $(USR_LIB_EXTERNAL_LIBS); do \
-			$(call copy_toolchain_lib_root,$${ARCH_SYSROOT_DIR},$${SUPPORT_LIB_DIR},$${ARCH_LIB_DIR},$$libs,/usr/lib); \
+		for libs in $(LIB_EXTERNAL_LIBS) $(USR_LIB_EXTERNAL_LIBS); do \
+			$(call copy_toolchain_lib_root,$${ARCH_SYSROOT_DIR},$${SUPPORT_LIB_DIR},$${ARCH_LIB_DIR},$$libs); \
 		done ; \
 	fi ; \
 	if test "$(BR2_TOOLCHAIN_EXTERNAL_GDB_SERVER_COPY)" = "y"; then \
@@ -668,11 +665,8 @@ define TOOLCHAIN_EXTERNAL_INSTALL_BFIN_FDPIC
 	                FDPIC_SUPPORT_LIB_DIR=`readlink -f $${FDPIC_LIBSTDCPP_A_LOCATION} | sed -r -e 's:libstdc\+\+\.a::'` ; \
 	        fi ; \
 	fi ; \
-	for libs in $(LIB_EXTERNAL_LIBS); do \
-	        $(call copy_toolchain_lib_root,$${FDPIC_SYSROOT_DIR},$${FDPIC_SUPPORT_LIB_DIR},$${FDPIC_LIB_DIR},$$libs,/lib); \
-	done ; \
-	for libs in $(USR_LIB_EXTERNAL_LIBS); do \
-	        $(call copy_toolchain_lib_root,$${FDPIC_SYSROOT_DIR},$${FDPIC_SUPPORT_LIB_DIR},$${FDPIC_LIB_DIR},$$libs,/usr/lib); \
+	for libs in $(LIB_EXTERNAL_LIBS) $(USR_LIB_EXTERNAL_LIBS); do \
+	        $(call copy_toolchain_lib_root,$${FDPIC_SYSROOT_DIR},$${FDPIC_SUPPORT_LIB_DIR},$${FDPIC_LIB_DIR},$$libs); \
 	done
 endef
 endif
-- 
2.4.10

[Buildroot] [PATCH RFC 1/2] toolchain: copy_toolchain_lib_root: rename LIBSPATH to LIBPATHS

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Wed,  3 Feb 2016 22:45:28 +0100, Thomas De Schampheleire wrote:
> From: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
> 
> LIBSPATH is populated based on a find with a pattern that can look like:
>     libfoo*.so
> and thus the output of the find will contain all file paths that match this
> pattern.
> 
> Unfortunately, the name LIBSPATH suggests that only one entry is returned,
> rather than possibly multiple.
> 
> As this code is quite complex, use the more accurate name LIBPATHS iso
> LIBSPATH.
> 
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
> ---
>  toolchain/helpers.mk | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com