* [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr
@ 2022-02-15 12:46 Norbert Lange
2022-02-15 12:46 ` [Buildroot] [PATCH 2/2] toolchain: prevent infinite loop in copy_toolchain_lib_root Norbert Lange
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Norbert Lange @ 2022-02-15 12:46 UTC (permalink / raw)
To: buildroot
Cc: Romain Naour, Norbert Lange, Giulio Benetti,
Thomas De Schampheleire, Thomas Petazzoni
Look at the case where the source toolchain has non-merged usr,
yet the target will have merged usr.
sysroot/lib/ld-musl-x86_64.so.1 -> ../usr/lib/libc.so
sysroot/usr/lib/libc.so
What happens is that buildroot copies the ld-*so* symlink
into usr/lib, at which point it becomes broken.
We now detect these broken symlinks, then try to find the target
binary in the library directories and fix the link.
Fix the case where the lib directory is a symlink, and no ld-*so*
is installed by adding -H to find.
Also use `cp -t` instead of some rarely used xargs tricks.
Signed-off-by: Norbert Lange <nolange79@gmail.com>
---
toolchain/helpers.mk | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index ef8e9a5f64..aaf2aecd80 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -135,8 +135,17 @@ copy_toolchain_sysroot = \
$(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
done ; \
fi ; \
- if [[ ! $$(find $(STAGING_DIR)/lib -name 'ld*.so.*' -print -quit) ]]; then \
- find $${ARCH_SYSROOT_DIR}/lib -name 'ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
+ for i in $$(find -H $(STAGING_DIR)/lib -name 'ld*.so.*' -xtype l); do \
+ LINKTARGET=`readlink $$i`; \
+ rm $$i; \
+ NEWLINKTARGET=$$(find -H $(STAGING_DIR)/$${ARCH_LIB_DIR} $(STAGING_DIR)/lib $(STAGING_DIR)/usr/$${ARCH_LIB_DIR} $(STAGING_DIR)/usr/lib -name "`basename $${LINKTARGET}`" -print -quit); \
+ if [ -n "$${NEWLINKTARGET}" -a -e "$${NEWLINKTARGET}" ]; then \
+ ln -sr $${NEWLINKTARGET} $$i; \
+ echo "Symlinking $$i -> `readlink $$i`" ; \
+ fi; \
+ done; \
+ if [[ ! $$(find -H $(STAGING_DIR)/lib -name 'ld*.so.*' -print -quit) ]]; then \
+ find $${ARCH_SYSROOT_DIR}/lib -name 'ld*.so.*' -print0 | xargs -0 cp -t $(STAGING_DIR)/lib/; \
fi ; \
if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
--
2.34.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH 2/2] toolchain: prevent infinite loop in copy_toolchain_lib_root
2022-02-15 12:46 [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr Norbert Lange
@ 2022-02-15 12:46 ` Norbert Lange
2022-09-18 16:38 ` Arnout Vandecappelle
2022-03-05 15:09 ` [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr Arnout Vandecappelle
2022-09-18 16:29 ` Arnout Vandecappelle
2 siblings, 1 reply; 6+ messages in thread
From: Norbert Lange @ 2022-02-15 12:46 UTC (permalink / raw)
To: buildroot
Cc: Thomas Petazzoni, Romain Naour, Norbert Lange, Giulio Benetti,
Thomas De Schampheleire
This can happen when there are broken symlinks.
Signed-off-by: Norbert Lange <nolange79@gmail.com>
---
toolchain/helpers.mk | 1 +
1 file changed, 1 insertion(+)
diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index aaf2aecd80..7623d68123 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -20,6 +20,7 @@ copy_toolchain_lib_root = \
if test -h $${LIBPATH} ; then \
cp -d $${LIBPATH} $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
LIBPATH="`readlink -f $${LIBPATH}`"; \
+ [ -n "$${LIBPATH}" ] || break; \
elif test -f $${LIBPATH}; then \
$(INSTALL) -D -m0755 $${LIBPATH} $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
break ; \
--
2.34.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr
2022-02-15 12:46 [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr Norbert Lange
2022-02-15 12:46 ` [Buildroot] [PATCH 2/2] toolchain: prevent infinite loop in copy_toolchain_lib_root Norbert Lange
@ 2022-03-05 15:09 ` Arnout Vandecappelle
2022-03-09 13:52 ` Norbert Lange
2022-09-18 16:29 ` Arnout Vandecappelle
2 siblings, 1 reply; 6+ messages in thread
From: Arnout Vandecappelle @ 2022-03-05 15:09 UTC (permalink / raw)
To: Norbert Lange, buildroot
Cc: Thomas Petazzoni, Romain Naour, Giulio Benetti, Thomas De Schampheleire
On 15/02/2022 13:46, Norbert Lange wrote:
> Look at the case where the source toolchain has non-merged usr,
> yet the target will have merged usr.
>
> sysroot/lib/ld-musl-x86_64.so.1 -> ../usr/lib/libc.so
> sysroot/usr/lib/libc.so
Where do you get an external toolchain with such a weird layout? Normally libc
is in sysroot/lib and ld-musl*.so.1 is a symlink to either /lib/libc.so or
../lib/libc.so.
Regards,
Arnout
>
> What happens is that buildroot copies the ld-*so* symlink
> into usr/lib, at which point it becomes broken.
>
> We now detect these broken symlinks, then try to find the target
> binary in the library directories and fix the link.
>
> Fix the case where the lib directory is a symlink, and no ld-*so*
> is installed by adding -H to find.
>
> Also use `cp -t` instead of some rarely used xargs tricks.
>
> Signed-off-by: Norbert Lange <nolange79@gmail.com>
> ---
> toolchain/helpers.mk | 13 +++++++++++--
> 1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> index ef8e9a5f64..aaf2aecd80 100644
> --- a/toolchain/helpers.mk
> +++ b/toolchain/helpers.mk
> @@ -135,8 +135,17 @@ copy_toolchain_sysroot = \
> $(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
> done ; \
> fi ; \
> - if [[ ! $$(find $(STAGING_DIR)/lib -name 'ld*.so.*' -print -quit) ]]; then \
> - find $${ARCH_SYSROOT_DIR}/lib -name 'ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
> + for i in $$(find -H $(STAGING_DIR)/lib -name 'ld*.so.*' -xtype l); do \
> + LINKTARGET=`readlink $$i`; \
> + rm $$i; \
> + NEWLINKTARGET=$$(find -H $(STAGING_DIR)/$${ARCH_LIB_DIR} $(STAGING_DIR)/lib $(STAGING_DIR)/usr/$${ARCH_LIB_DIR} $(STAGING_DIR)/usr/lib -name "`basename $${LINKTARGET}`" -print -quit); \
> + if [ -n "$${NEWLINKTARGET}" -a -e "$${NEWLINKTARGET}" ]; then \
> + ln -sr $${NEWLINKTARGET} $$i; \
> + echo "Symlinking $$i -> `readlink $$i`" ; \
> + fi; \
> + done; \
> + if [[ ! $$(find -H $(STAGING_DIR)/lib -name 'ld*.so.*' -print -quit) ]]; then \
> + find $${ARCH_SYSROOT_DIR}/lib -name 'ld*.so.*' -print0 | xargs -0 cp -t $(STAGING_DIR)/lib/; \
> fi ; \
> if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
> if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr
2022-03-05 15:09 ` [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr Arnout Vandecappelle
@ 2022-03-09 13:52 ` Norbert Lange
0 siblings, 0 replies; 6+ messages in thread
From: Norbert Lange @ 2022-03-09 13:52 UTC (permalink / raw)
To: Arnout Vandecappelle
Cc: Thomas Petazzoni, Romain Naour, Giulio Benetti,
Thomas De Schampheleire, buildroot
Am Sa., 5. März 2022 um 16:09 Uhr schrieb Arnout Vandecappelle <arnout@mind.be>:
>
>
>
> On 15/02/2022 13:46, Norbert Lange wrote:
> > Look at the case where the source toolchain has non-merged usr,
> > yet the target will have merged usr.
> >
> > sysroot/lib/ld-musl-x86_64.so.1 -> ../usr/lib/libc.so
> > sysroot/usr/lib/libc.so
>
> Where do you get an external toolchain with such a weird layout? Normally libc
> is in sysroot/lib and ld-musl*.so.1 is a symlink to either /lib/libc.so or
> ../lib/libc.so.
>
>
> Regards,
> Arnout
>
the toolchain is built with crosstool-ng
> >
> > What happens is that buildroot copies the ld-*so* symlink
> > into usr/lib, at which point it becomes broken.
> >
> > We now detect these broken symlinks, then try to find the target
> > binary in the library directories and fix the link.
> >
> > Fix the case where the lib directory is a symlink, and no ld-*so*
> > is installed by adding -H to find.
> >
> > Also use `cp -t` instead of some rarely used xargs tricks.
> >
> > Signed-off-by: Norbert Lange <nolange79@gmail.com>
> > ---
> > toolchain/helpers.mk | 13 +++++++++++--
> > 1 file changed, 11 insertions(+), 2 deletions(-)
> >
> > diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> > index ef8e9a5f64..aaf2aecd80 100644
> > --- a/toolchain/helpers.mk
> > +++ b/toolchain/helpers.mk
> > @@ -135,8 +135,17 @@ copy_toolchain_sysroot = \
> > $(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
> > done ; \
> > fi ; \
> > - if [[ ! $$(find $(STAGING_DIR)/lib -name 'ld*.so.*' -print -quit) ]]; then \
> > - find $${ARCH_SYSROOT_DIR}/lib -name 'ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
> > + for i in $$(find -H $(STAGING_DIR)/lib -name 'ld*.so.*' -xtype l); do \
> > + LINKTARGET=`readlink $$i`; \
> > + rm $$i; \
> > + NEWLINKTARGET=$$(find -H $(STAGING_DIR)/$${ARCH_LIB_DIR} $(STAGING_DIR)/lib $(STAGING_DIR)/usr/$${ARCH_LIB_DIR} $(STAGING_DIR)/usr/lib -name "`basename $${LINKTARGET}`" -print -quit); \
> > + if [ -n "$${NEWLINKTARGET}" -a -e "$${NEWLINKTARGET}" ]; then \
> > + ln -sr $${NEWLINKTARGET} $$i; \
> > + echo "Symlinking $$i -> `readlink $$i`" ; \
> > + fi; \
> > + done; \
> > + if [[ ! $$(find -H $(STAGING_DIR)/lib -name 'ld*.so.*' -print -quit) ]]; then \
> > + find $${ARCH_SYSROOT_DIR}/lib -name 'ld*.so.*' -print0 | xargs -0 cp -t $(STAGING_DIR)/lib/; \
> > fi ; \
> > if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
> > if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr
2022-02-15 12:46 [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr Norbert Lange
2022-02-15 12:46 ` [Buildroot] [PATCH 2/2] toolchain: prevent infinite loop in copy_toolchain_lib_root Norbert Lange
2022-03-05 15:09 ` [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr Arnout Vandecappelle
@ 2022-09-18 16:29 ` Arnout Vandecappelle
2 siblings, 0 replies; 6+ messages in thread
From: Arnout Vandecappelle @ 2022-09-18 16:29 UTC (permalink / raw)
To: Norbert Lange, buildroot
Cc: Thomas Petazzoni, Romain Naour, Giulio Benetti, Thomas De Schampheleire
Hi Norbert,
Sorry that this has been laying around so long without reaction. The thing is
that this external toolchain sysroot stuff is incredibly fragile and even worse,
difficult to understand. So there isn't much motivation to accept patches. We
probably should have some tests for it...
On 15/02/2022 13:46, Norbert Lange wrote:
> Look at the case where the source toolchain has non-merged usr,
> yet the target will have merged usr.
>
> sysroot/lib/ld-musl-x86_64.so.1 -> ../usr/lib/libc.so
> sysroot/usr/lib/libc.so
>
> What happens is that buildroot copies the ld-*so* symlink
> into usr/lib, at which point it becomes broken.
>
> We now detect these broken symlinks, then try to find the target
> binary in the library directories and fix the link.
>
> Fix the case where the lib directory is a symlink, and no ld-*so*
> is installed by adding -H to find.
I don't understand what's the issue there. $(STAGING_DIR)/lib is always a
symlink in merged usr, and never a symlink in non-merged usr. Also, the -H
parameter has no effect on the paths supplied in the command line (i.e.
$(STAGING_DIR)/lib itself), only on the symlinks within it. So I don't
understand what you're trying to fix here. Maybe you have a tuple->. symlink
inside the lib directory? But even that won't be followed by find because it
sees that it's a directory it already encountered. Or maybe it's a
tuple->../usr/lib/tuple symlink? That also won't be followed by find because
it's a broken symlink.
So, can you explain better (in the commit message) in which situation this is
needed?
> Also use `cp -t` instead of some rarely used xargs tricks.
That's pretty much in the eye of the beholder. I regularly use xargs -I, I
never used cp -t. We use neither of these constructs anywhere else in Buildroot.
So if anything, it should probably be changed into find -exec which we do use
elsewhere (though I honestly find it much more annoying to use than xargs -I).
Anyway, such an unrelated change should be a separate patch, so we can easily
skip it if we don't agree (or apply it if we don't agree with the rest).
>
> Signed-off-by: Norbert Lange <nolange79@gmail.com>
> ---
> toolchain/helpers.mk | 13 +++++++++++--
> 1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> index ef8e9a5f64..aaf2aecd80 100644
> --- a/toolchain/helpers.mk
> +++ b/toolchain/helpers.mk
> @@ -135,8 +135,17 @@ copy_toolchain_sysroot = \
There's a huge comment above copy_toolchain_sysroot that explains in detail
what it does. It should really be interspersed with the code itself, but due to
make limitations that's not possible. Anyway, that comment should be extended
with an explanation of the why and the how of the new block.
> $(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
> done ; \
> fi ; \
> - if [[ ! $$(find $(STAGING_DIR)/lib -name 'ld*.so.*' -print -quit) ]]; then \
> - find $${ARCH_SYSROOT_DIR}/lib -name 'ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
> + for i in $$(find -H $(STAGING_DIR)/lib -name 'ld*.so.*' -xtype l); do \
Should there ever be more than one? IOW shouldn't there be a -quit?
Also, we usually use backticks instead of $$()
> + LINKTARGET=`readlink $$i`; \
> + rm $$i; \
> + NEWLINKTARGET=$$(find -H $(STAGING_DIR)/$${ARCH_LIB_DIR} $(STAGING_DIR)/lib $(STAGING_DIR)/usr/$${ARCH_LIB_DIR} $(STAGING_DIR)/usr/lib -name "`basename $${LINKTARGET}`" -print -quit); \
What you actually want here is to take the first one that matches, right?
Also, it should always be in one of those directories themselves, and not in one
of the subdirectories I think? In that case, I think it's more clear if it's a
loop rather than find. So something like:
for libdir in $(STAGING_DIR)/$${ARCH_LIB_DIR} $(STAGING_DIR)/lib
$(STAGING_DIR)/usr/$${ARCH_LIB_DIR} $(STAGING_DIR)/usr/lib; do \
NEWLINKTARGET="$${libdir}/`basename $${LINKTARGET}`"; \
if [ -n "$${NEWLINKTARGET}" -a -e "$${NEWLINKTARGET}" ]; then \
echo "Symlinking $$i -> `readlink $$i`" ; \
ln -sr $${NEWLINKTARGET} $$i; \
break;
fi; \
done; \
(this was just some quick coding, I probably missed a lot of things).
> + if [ -n "$${NEWLINKTARGET}" -a -e "$${NEWLINKTARGET}" ]; then \
If this is not true, was it a good idea to delete the symlink? Or maybe we
should even error out in that case?
> + ln -sr $${NEWLINKTARGET} $$i; \
> + echo "Symlinking $$i -> `readlink $$i`" ; \
> + fi; \
> + done; \
> + if [[ ! $$(find -H $(STAGING_DIR)/lib -name 'ld*.so.*' -print -quit) ]]; then \
> + find $${ARCH_SYSROOT_DIR}/lib -name 'ld*.so.*' -print0 | xargs -0 cp -t $(STAGING_DIR)/lib/; \
I wonder if this bit couldn't be merged with the above (skipping the -xtype in
the find of course).
Regards,
Arnout
> fi ; \
> if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
> if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH 2/2] toolchain: prevent infinite loop in copy_toolchain_lib_root
2022-02-15 12:46 ` [Buildroot] [PATCH 2/2] toolchain: prevent infinite loop in copy_toolchain_lib_root Norbert Lange
@ 2022-09-18 16:38 ` Arnout Vandecappelle
0 siblings, 0 replies; 6+ messages in thread
From: Arnout Vandecappelle @ 2022-09-18 16:38 UTC (permalink / raw)
To: Norbert Lange, buildroot
Cc: Thomas De Schampheleire, Romain Naour, Giulio Benetti, Thomas Petazzoni
On 15/02/2022 13:46, Norbert Lange wrote:
> This can happen when there are broken symlinks.
>
> Signed-off-by: Norbert Lange <nolange79@gmail.com>
> ---
> toolchain/helpers.mk | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> index aaf2aecd80..7623d68123 100644
> --- a/toolchain/helpers.mk
> +++ b/toolchain/helpers.mk
> @@ -20,6 +20,7 @@ copy_toolchain_lib_root = \
> if test -h $${LIBPATH} ; then \
> cp -d $${LIBPATH} $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
> LIBPATH="`readlink -f $${LIBPATH}`"; \
> + [ -n "$${LIBPATH}" ] || break; \
I again don't understand how this can happen. LIBPATH exists and is a symlink
(tested a few lines above), so readlink will always return the contents of that
symlink. I don't think that can ever be empty, but with the -f (full path)
option it can definitely not be empty: readlink doesn't care about broken
symlinks, and even if it has too many .. components it will just print something
based on /.
Can you give an example of such a broken symlink where this can happen?
Regards,
Arnout
> elif test -f $${LIBPATH}; then \
> $(INSTALL) -D -m0755 $${LIBPATH} $(TARGET_DIR)/$${DESTDIR}/$${LIBNAME}; \
> break ; \
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-09-18 16:38 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-15 12:46 [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr Norbert Lange
2022-02-15 12:46 ` [Buildroot] [PATCH 2/2] toolchain: prevent infinite loop in copy_toolchain_lib_root Norbert Lange
2022-09-18 16:38 ` Arnout Vandecappelle
2022-03-05 15:09 ` [Buildroot] [PATCH 1/2] toolchain: support mismatched merged usr Arnout Vandecappelle
2022-03-09 13:52 ` Norbert Lange
2022-09-18 16:29 ` Arnout Vandecappelle
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.