[Buildroot] [PATCH v2 1/1] toolchain: handle toolchains with multiple ld*.so.* files

[Buildroot] [PATCH v2 1/1] toolchain: handle toolchains with multiple ld*.so.* files

[Jonah Petri]

Some 3rd party vendor toolchains have multiple files which match
these glob patterns.  In this case, the shell script failed.
Switching to use find and xargs solves the issue.

Signed-off-by: Jonah Petri <jonah@petri.us>
---
Changes v1->v2:
	Add SOB

 toolchain/helpers.mk | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index dfb2581ed5..5ed149d57d 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -135,10 +135,8 @@ copy_toolchain_sysroot = \
 			$(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
 		done ; \
 	fi ; \
-	if [ ! -e $(STAGING_DIR)/lib/ld*.so.* ]; then \
-		if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so.* ]; then \
-			cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so.* $(STAGING_DIR)/lib/ ; \
-		fi ; \
+       if ! find -wholename $(STAGING_DIR)'/lib/ld*.so.*' > /dev/null; then \
+               find -wholename $${ARCH_SYSROOT_DIR}'/lib/ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
 	fi ; \
 	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
 		if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
-- 
2.30.2

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/1] toolchain: handle toolchains with multiple ld*.so.* files

[Arnout Vandecappelle]

On 24/08/2021 22:38, Jonah Petri wrote:
> Some 3rd party vendor toolchains have multiple files which match
> these glob patterns.  In this case, the shell script failed.
> Switching to use find and xargs solves the issue.
> 
> Signed-off-by: Jonah Petri <jonah@petri.us>

 Applied to master, thanks.

 Regards,
 Arnout

> ---
> Changes v1->v2:
> 	Add SOB
> 
>  toolchain/helpers.mk | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> index dfb2581ed5..5ed149d57d 100644
> --- a/toolchain/helpers.mk
> +++ b/toolchain/helpers.mk
> @@ -135,10 +135,8 @@ copy_toolchain_sysroot = \
>  			$(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
>  		done ; \
>  	fi ; \
> -	if [ ! -e $(STAGING_DIR)/lib/ld*.so.* ]; then \
> -		if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so.* ]; then \
> -			cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so.* $(STAGING_DIR)/lib/ ; \
> -		fi ; \
> +       if ! find -wholename $(STAGING_DIR)'/lib/ld*.so.*' > /dev/null; then \
> +               find -wholename $${ARCH_SYSROOT_DIR}'/lib/ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
>  	fi ; \
>  	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
>  		if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
> 
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/1] toolchain: handle toolchains with multiple ld*.so.* files

[Baruch Siach]

Hi Arnout, Jonah,

On Wed, Aug 25 2021, Arnout Vandecappelle wrote:
> On 24/08/2021 22:38, Jonah Petri wrote:
>> Some 3rd party vendor toolchains have multiple files which match
>> these glob patterns.  In this case, the shell script failed.
>> Switching to use find and xargs solves the issue.
>> 
>> Signed-off-by: Jonah Petri <jonah@petri.us>
>
>  Applied to master, thanks.

It looks like this breaks the ARM aarch64 external toolchain. Both ld.so
existence test and ld.so listing command are broken. See this libnl
failure:

  http://autobuild.buildroot.net/results/afe/afe44f4b6a3c53e5864cfb10b04529011e72cf5c/

I have no idea why we don't see much wider breakage in the autobuilder.

The patch below fixes the build here, but I'm not entirely sure it is
the correct fix.

Since this is a risky change and we are close to release, I'd suggest to
revert this change in master, and consider it again for next.

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index 5ed149d57d31..3aefd1d0096e 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -135,8 +135,8 @@ copy_toolchain_sysroot = \
 			$(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
 		done ; \
 	fi ; \
-       if ! find -wholename $(STAGING_DIR)'/lib/ld*.so.*' > /dev/null; then \
-               find -wholename $${ARCH_SYSROOT_DIR}'/lib/ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
+       if [ ! -e $(STAGING_DIR)/lib/ld*.so.* ]; then \
+               find $${ARCH_SYSROOT_DIR} -wholename '*/lib/ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
 	fi ; \
 	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
 		if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \

baruch

>> ---
>> Changes v1->v2:
>> 	Add SOB
>> 
>>  toolchain/helpers.mk | 6 ++----
>>  1 file changed, 2 insertions(+), 4 deletions(-)
>> 
>> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
>> index dfb2581ed5..5ed149d57d 100644
>> --- a/toolchain/helpers.mk
>> +++ b/toolchain/helpers.mk
>> @@ -135,10 +135,8 @@ copy_toolchain_sysroot = \
>>  			$(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
>>  		done ; \
>>  	fi ; \
>> -	if [ ! -e $(STAGING_DIR)/lib/ld*.so.* ]; then \
>> -		if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so.* ]; then \
>> -			cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so.* $(STAGING_DIR)/lib/ ; \
>> -		fi ; \
>> +       if ! find -wholename $(STAGING_DIR)'/lib/ld*.so.*' > /dev/null; then \
>> +               find -wholename $${ARCH_SYSROOT_DIR}'/lib/ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
>>  	fi ; \
>>  	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
>>  		if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
>> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot


-- 
                                                     ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch@tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/1] toolchain: handle toolchains with multiple ld*.so.* files

[Arnout Vandecappelle]

On 26/08/2021 13:24, Baruch Siach wrote:
> Hi Arnout, Jonah,
> 
> On Wed, Aug 25 2021, Arnout Vandecappelle wrote:
>> On 24/08/2021 22:38, Jonah Petri wrote:
>>> Some 3rd party vendor toolchains have multiple files which match
>>> these glob patterns.  In this case, the shell script failed.
>>> Switching to use find and xargs solves the issue.
>>>
>>> Signed-off-by: Jonah Petri <jonah@petri.us>
>>
>>  Applied to master, thanks.
> 
> It looks like this breaks the ARM aarch64 external toolchain. Both ld.so
> existence test and ld.so listing command are broken. See this libnl
> failure:
> 
>   http://autobuild.buildroot.net/results/afe/afe44f4b6a3c53e5864cfb10b04529011e72cf5c/
> 
> I have no idea why we don't see much wider breakage in the autobuilder.
> 
> The patch below fixes the build here, but I'm not entirely sure it is
> the correct fix.
> 
> Since this is a risky change and we are close to release, I'd suggest to
> revert this change in master, and consider it again for next.

 Reverted.

 There's indeed a lot wrong with this...

> 
> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> index 5ed149d57d31..3aefd1d0096e 100644
> --- a/toolchain/helpers.mk
> +++ b/toolchain/helpers.mk
> @@ -135,8 +135,8 @@ copy_toolchain_sysroot = \
>  			$(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
>  		done ; \
>  	fi ; \
> -       if ! find -wholename $(STAGING_DIR)'/lib/ld*.so.*' > /dev/null; then \
> -               find -wholename $${ARCH_SYSROOT_DIR}'/lib/ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
> +       if [ ! -e $(STAGING_DIR)/lib/ld*.so.* ]; then \
> +               find $${ARCH_SYSROOT_DIR} -wholename '*/lib/ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
>  	fi ; \
>  	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
>  		if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
> 
> baruch
> 
>>> ---
>>> Changes v1->v2:
>>> 	Add SOB
>>>
>>>  toolchain/helpers.mk | 6 ++----
>>>  1 file changed, 2 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
>>> index dfb2581ed5..5ed149d57d 100644
>>> --- a/toolchain/helpers.mk
>>> +++ b/toolchain/helpers.mk
>>> @@ -135,10 +135,8 @@ copy_toolchain_sysroot = \
>>>  			$(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
>>>  		done ; \
>>>  	fi ; \
>>> -	if [ ! -e $(STAGING_DIR)/lib/ld*.so.* ]; then \
>>> -		if [ -e $${ARCH_SYSROOT_DIR}/lib/ld*.so.* ]; then \
>>> -			cp -a $${ARCH_SYSROOT_DIR}/lib/ld*.so.* $(STAGING_DIR)/lib/ ; \
>>> -		fi ; \
>>> +       if ! find -wholename $(STAGING_DIR)'/lib/ld*.so.*' > /dev/null; then \

 With no path, find defaults to the current directory. However, STAGING_DIR is
the full absolute directory, so wholename is never going to match it. It should
probably be

find $(STAGINGDIR) -wholename $(STAGINGDIR)/lib/ld*.so.*

 In addition, however, find's exit code doesn't say anything about whether it
was found or not. So we instead have to use the printed output and check if it's
non-empty.

 Finally, as an optimisation, we could use -quit to make sure it exits after the
first match instead of needlessly continuing.


 As Baruch said, however, this is too risky for master, so if a v3 comes it will
be applied to next.


 Regards,
 Arnout

>>> +               find -wholename $${ARCH_SYSROOT_DIR}'/lib/ld*.so.*' -print0 | xargs -0 -I % cp % $(STAGING_DIR)/lib/; \
>>>  	fi ; \
>>>  	if [ `readlink -f $${SYSROOT_DIR}` != `readlink -f $${ARCH_SYSROOT_DIR}` ] ; then \
>>>  		if [ ! -d $${ARCH_SYSROOT_DIR}/usr/include ] ; then \
>>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot@busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
> 
> 
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/1] toolchain: handle toolchains with multiple ld*.so.* files

[Jonah Petri]

On 8/26/21 8:24 AM, Arnout Vandecappelle wrote:
>>>> +       if ! find -wholename $(STAGING_DIR)'/lib/ld*.so.*' > /dev/null; then \
>   With no path, find defaults to the current directory. However, STAGING_DIR is
> the full absolute directory, so wholename is never going to match it. It should
> probably be
>
> find $(STAGINGDIR) -wholename $(STAGINGDIR)/lib/ld*.so.*
>
>   In addition, however, find's exit code doesn't say anything about whether it
> was found or not. So we instead have to use the printed output and check if it's
> non-empty.
>
>   Finally, as an optimisation, we could use -quit to make sure it exits after the
> first match instead of needlessly continuing.
>
>
>   As Baruch said, however, this is too risky for master, so if a v3 comes it will
> be applied to next.


Hi Arnout & Baruch,

Thanks for the feedback, and I apologize for the breakage.  I'm 
embarrassed by my incorrect assumption about the return value of find 
(wow, I've been using that incorrectly for a lot of years!) and I 
understand and appreciate the rest of the comments above.

I'll wait for the next release to complete and then put in a v3.

Thanks for buildroot!

Jonah

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/1] toolchain: handle toolchains with multiple ld*.so.* files

[Arnout Vandecappelle]

On 26/08/2021 15:24, Jonah Petri wrote:
> On 8/26/21 8:24 AM, Arnout Vandecappelle wrote:
>>>>> +       if ! find -wholename $(STAGING_DIR)'/lib/ld*.so.*' > /dev/null; then \
>>   With no path, find defaults to the current directory. However, STAGING_DIR is
>> the full absolute directory, so wholename is never going to match it. It should
>> probably be
>>
>> find $(STAGINGDIR) -wholename $(STAGINGDIR)/lib/ld*.so.*
>>
>>   In addition, however, find's exit code doesn't say anything about whether it
>> was found or not. So we instead have to use the printed output and check if it's
>> non-empty.
>>
>>   Finally, as an optimisation, we could use -quit to make sure it exits after the
>> first match instead of needlessly continuing.
>>
>>
>>   As Baruch said, however, this is too risky for master, so if a v3 comes it will
>> be applied to next.
> 
> 
> Hi Arnout & Baruch,
> 
> Thanks for the feedback, and I apologize for the breakage.  I'm embarrassed by
> my incorrect assumption about the return value of find (wow, I've been using
> that incorrectly for a lot of years!) and I understand and appreciate the rest
> of the comments above.
> 
> I'll wait for the next release to complete and then put in a v3.

 You don't need to wait. We apply stuff to the next branch while master is in
stabilization.

 Regards,
 Arnout
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot