[Buildroot] Bug in RPATH fixing logic

[Buildroot] Bug in RPATH fixing logic

[Thomas Petazzoni]

Hello,

I found a bug in the RPATH fixing logic. On ARC, some of the binutils
programs (ar and ranlib at least) are linked with libfl.so.2, which
is provided by Buildroot in $(HOST_DIR)/lib.

However, after "make sdk" has been called, ar and ranlib don't have any
RPATH encoded. I debugged this and here is what happens:

First, fix-rpath fixes the RPATH of $(HOST_DIR)/bin/<tuple>-ar:

/opt/br-arcle-hs38-full-2017.11-rc1/bin/patchelf --debug --make-rpath-relative /opt/br-arcle-hs38-full-2017.11-rc1 --relative-to-file /opt/br-arcle-hs38-full-2017.11-rc1/bin/arc-buildroot-linux-uclibc-ar
patching ELF file `/opt/br-arcle-hs38-full-2017.11-rc1/bin/arc-buildroot-linux-uclibc-ar'
Kernel page size is 4096 bytes
keeping relative path of /opt/br-arcle-hs38-full-2017.11-rc1/lib
new rpath is `$ORIGIN/../lib'

Here, everything is fine, the rpath is $ORIGIN/../lib.

However, fix-rpath moves on and fixes $(HOST_DIR)/<tuple>/bin/ar:

/opt/br-arcle-hs38-full-2017.11-rc1/bin/patchelf --debug --make-rpath-relative /opt/br-arcle-hs38-full-2017.11-rc1 --relative-to-file /opt/br-arcle-hs38-full-2017.11-rc1/arc-buildroot-linux-uclibc/bin/ar
patching ELF file `/opt/br-arcle-hs38-full-2017.11-rc1/arc-buildroot-linux-uclibc/bin/ar'
Kernel page size is 4096 bytes
removing directory '$ORIGIN/../lib' from RPATH because it does not contain needed libs
new rpath is `'

The gotcha is that $(HOST_DIR)/<tuple>/bin/ar and
$(HOST_DIR)/bin/<tuple>-ar are the same files: they are hard links!

So, when $(HOST_DIR)/bin/<tuple>-ar is fixed, everything is fine
because the library we depend on is indeed located
in /opt/br-arcle-hs38-full-2017.11-rc1/lib/, which is the hardcoded
RPATH. It gets fixed to $ORIGIN/../lib, as should be the case.

However, when we get to $(HOST_DIR)/<tuple>/bin/ar, libfl.so.2 is not
found in $ORIGIN/../lib (because it should be $ORIGIN/../../lib), and
patchelf decides to remove the rpath entirely.

So, we have two things that are a bit weird here:

 - Why are the binutils linked to libfl ? This happens only on the ARC
   architecture, where we use a special version of binutils that gets
   autoreconf'ed.

 - The absolute rpath in $(HOST_DIR)/<tuple>/bin/ar is wrong in the
   first place, but I'm not sure how to fix this.

Wolfgang, Arnout, do you have any idea?

You can reproduce this by building an ARC toolchain, and look at the
(absence of) rpath on bin/<tuple>-ar.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] Bug in RPATH fixing logic

[Thomas Petazzoni]

Hello,

On Sun, 12 Nov 2017 17:40:13 +0100, Thomas Petazzoni wrote:

> So, we have two things that are a bit weird here:
> 
>  - Why are the binutils linked to libfl ? This happens only on the ARC
>    architecture, where we use a special version of binutils that gets
>    autoreconf'ed.

So this happens because the ARC binutils version is fetched from Git,
so we need to have host-flex installed, and binutils detects the flex
library and decides to use it.

>  - The absolute rpath in $(HOST_DIR)/<tuple>/bin/ar is wrong in the
>    first place, but I'm not sure how to fix this.

I'm wrong on this: $(HOST_DIR)/<tuple>/bin/ar RPATH is totally fine:

$ readelf -d arc-buildroot-linux-uclibc/bin/ar  | grep rpath
 0x000000000000000f (RPATH)              Library rpath: [/opt/br-arcle-hs38-full-2017.11-rc1/lib]

The problem is that this gets turned into $ORIGIN/../lib by patchelf.

I don't see any other solution than de-duplicating such binaries.

Do you see another option ?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] Bug in RPATH fixing logic

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Hello,
 > On Sun, 12 Nov 2017 17:40:13 +0100, Thomas Petazzoni wrote:

 >> So, we have two things that are a bit weird here:
 >> 
 >> - Why are the binutils linked to libfl ? This happens only on the ARC
 >> architecture, where we use a special version of binutils that gets
 >> autoreconf'ed.

 > So this happens because the ARC binutils version is fetched from Git,
 > so we need to have host-flex installed, and binutils detects the flex
 > library and decides to use it.

And there is no configure flag to disable that?


 >> - The absolute rpath in $(HOST_DIR)/<tuple>/bin/ar is wrong in the
 >> first place, but I'm not sure how to fix this.

 > I'm wrong on this: $(HOST_DIR)/<tuple>/bin/ar RPATH is totally fine:

 > $ readelf -d arc-buildroot-linux-uclibc/bin/ar  | grep rpath
 >  0x000000000000000f (RPATH)              Library rpath: [/opt/br-arcle-hs38-full-2017.11-rc1/lib]

 > The problem is that this gets turned into $ORIGIN/../lib by patchelf.

 > I don't see any other solution than de-duplicating such binaries.

 > Do you see another option ?

Not if we *need* both binaries (do we?) Is there a way to get binutils
to use soft links instead of hard links?

-- 
Bye, Peter Korsgaard

[Buildroot] Bug in RPATH fixing logic

[Thomas Petazzoni]

Hello,

On Sun, 12 Nov 2017 18:15:12 +0100, Peter Korsgaard wrote:

>  > So this happens because the ARC binutils version is fetched from Git,
>  > so we need to have host-flex installed, and binutils detects the flex
>  > library and decides to use it.  
> 
> And there is no configure flag to disable that?

Not that I can see.

>  >> - The absolute rpath in $(HOST_DIR)/<tuple>/bin/ar is wrong in the
>  >> first place, but I'm not sure how to fix this.  
> 
>  > I'm wrong on this: $(HOST_DIR)/<tuple>/bin/ar RPATH is totally fine:  
> 
>  > $ readelf -d arc-buildroot-linux-uclibc/bin/ar  | grep rpath
>  >  0x000000000000000f (RPATH)              Library rpath: [/opt/br-arcle-hs38-full-2017.11-rc1/lib]  
> 
>  > The problem is that this gets turned into $ORIGIN/../lib by patchelf.  
> 
>  > I don't see any other solution than de-duplicating such binaries.  
> 
>  > Do you see another option ?  
> 
> Not if we *need* both binaries (do we?)

I think we do need both binaries. <tuple>-<tool> is the "publicly"
visible copy, while <tuple>/bin/<tool> is the one called internally by
gcc (for example when it calls as or ld). Yann can explain more about
this perhaps.

> Is there a way to get binutils to use soft links instead of hard links?

How would this solve the problem?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] Bug in RPATH fixing logic

[Yann E. MORIN]

On 2017-11-12 18:24 +0100, Thomas Petazzoni spake thusly:
> Hello,
> 
> On Sun, 12 Nov 2017 18:15:12 +0100, Peter Korsgaard wrote:
> 
> >  > So this happens because the ARC binutils version is fetched from Git,
> >  > so we need to have host-flex installed, and binutils detects the flex
> >  > library and decides to use it.  
> > 
> > And there is no configure flag to disable that?
> 
> Not that I can see.

A long time ago (when I was too young to have a beard), libfl was only
available as a static library.

That library contained (and stil contains) a 'main()' symbol (is it even
a weak one?), so it really is a weird library to begin with... I would
not mind building that host library with --disable-shared --enable-static
and only install the static library...

That would solve this specific issue.

Until we need another library...

> >  >> - The absolute rpath in $(HOST_DIR)/<tuple>/bin/ar is wrong in the
> >  >> first place, but I'm not sure how to fix this.  
> > 
> >  > I'm wrong on this: $(HOST_DIR)/<tuple>/bin/ar RPATH is totally fine:  
> > 
> >  > $ readelf -d arc-buildroot-linux-uclibc/bin/ar  | grep rpath
> >  >  0x000000000000000f (RPATH)              Library rpath: [/opt/br-arcle-hs38-full-2017.11-rc1/lib]  
> > 
> >  > The problem is that this gets turned into $ORIGIN/../lib by patchelf.  
> > 
> >  > I don't see any other solution than de-duplicating such binaries.  
> > 
> >  > Do you see another option ?  
> > 
> > Not if we *need* both binaries (do we?)
> 
> I think we do need both binaries. <tuple>-<tool> is the "publicly"
> visible copy, while <tuple>/bin/<tool> is the one called internally by
> gcc (for example when it calls as or ld). Yann can explain more about
> this perhaps.

I can't explain much more than you did, sorry...

> > Is there a way to get binutils to use soft links instead of hard links?
> 
> How would this solve the problem?

I guess the reasoning of Peter was that maybe $ORIGIN is relative to the
real file not the symlink.

Thus by having <tuple>/bin/<tool> a symlink to ../../bin/<tuple>-<tool>
would solve the issue.

But I am not sure of it either, and the manpage for ld.so does not
mention that. It only states:

    $ORIGIN (or equivalently ${ORIGIN})
        This expands to the directory containing the program or
        shared object.

No mention of the behaviour when the program is being called through a
symlink in another directory... :-/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Bug in RPATH fixing logic

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Hello,
 > On Sun, 12 Nov 2017 18:15:12 +0100, Peter Korsgaard wrote:

 >> > So this happens because the ARC binutils version is fetched from Git,
 >> > so we need to have host-flex installed, and binutils detects the flex
 >> > library and decides to use it.  
 >> 
 >> And there is no configure flag to disable that?

 > Not that I can see.

Ok.

 >> Is there a way to get binutils to use soft links instead of hard links?

 > How would this solve the problem?

Well, the problem is that the binary gets processed twice in different
directories because of the hard link, so if it instead was a softlink it
woud get ignored by the RPATH logic and only get fixed up once - Right?

-- 
Bye, Peter Korsgaard

[Buildroot] Bug in RPATH fixing logic

[Wolfgang Grandegger]

Am 12.11.2017 um 19:03 schrieb Peter Korsgaard:
>>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> 
>   > Hello,
>   > On Sun, 12 Nov 2017 18:15:12 +0100, Peter Korsgaard wrote:
> 
>   >> > So this happens because the ARC binutils version is fetched from Git,
>   >> > so we need to have host-flex installed, and binutils detects the flex
>   >> > library and decides to use it.
>   >>
>   >> And there is no configure flag to disable that?
> 
>   > Not that I can see.
> 
> Ok.
> 
>   >> Is there a way to get binutils to use soft links instead of hard links?
> 
>   > How would this solve the problem?
> 
> Well, the problem is that the binary gets processed twice in different
> directories because of the hard link, so if it instead was a softlink it
> woud get ignored by the RPATH logic and only get fixed up once - Right?

Yes, soft links are ignored (only find -type f). $ORIGIN will not work 
with hard-linked binary files in different levels of the file hierarchy. 
Never thought about such cases.

Wolfgang.

[Buildroot] Bug in RPATH fixing logic

[Wolfgang Grandegger]

Am 12.11.2017 um 20:04 schrieb Wolfgang Grandegger:
> Am 12.11.2017 um 19:03 schrieb Peter Korsgaard:
>>>>>>> "Thomas" == Thomas Petazzoni 
>>>>>>> <thomas.petazzoni@free-electrons.com> writes:
>>
>> ? > Hello,
>> ? > On Sun, 12 Nov 2017 18:15:12 +0100, Peter Korsgaard wrote:
>>
>> ? >> > So this happens because the ARC binutils version is fetched 
>> from Git,
>> ? >> > so we need to have host-flex installed, and binutils detects 
>> the flex
>> ? >> > library and decides to use it.
>> ? >>
>> ? >> And there is no configure flag to disable that?
>>
>> ? > Not that I can see.
>>
>> Ok.
>>
>> ? >> Is there a way to get binutils to use soft links instead of hard 
>> links?
>>
>> ? > How would this solve the problem?
>>
>> Well, the problem is that the binary gets processed twice in different
>> directories because of the hard link, so if it instead was a softlink it
>> woud get ignored by the RPATH logic and only get fixed up once - Right?
> 
> Yes, soft links are ignored (only find -type f). $ORIGIN will not work 
> with hard-linked binary files in different levels of the file hierarchy. 

Well, adding a proper $ORIGIN/... for each link to the RPATH would work 
but detecting if other hard links exist is a bit tricky and would take 
time. If the references count (number of hardlinks to a particular 
inode) is greater than 1, we could process them in a special way. Not 
sure if it's worth the effort.

Wolfgang.

[Buildroot] Bug in RPATH fixing logic

[Thomas Petazzoni]

Hello,

On Sun, 12 Nov 2017 20:35:39 +0100, Wolfgang Grandegger wrote:

> >> Well, the problem is that the binary gets processed twice in different
> >> directories because of the hard link, so if it instead was a softlink it
> >> woud get ignored by the RPATH logic and only get fixed up once - Right?  
> > 
> > Yes, soft links are ignored (only find -type f). $ORIGIN will not work 
> > with hard-linked binary files in different levels of the file hierarchy.   
> 
> Well, adding a proper $ORIGIN/... for each link to the RPATH would work 
> but detecting if other hard links exist is a bit tricky and would take 
> time. If the references count (number of hardlinks to a particular 
> inode) is greater than 1, we could process them in a special way. Not 
> sure if it's worth the effort.

Well, we're not at the point of deciding if it is worth the effort or
not: we have a bug, and we need to fix it somehow :)

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] Bug in RPATH fixing logic

[Wolfgang Grandegger]

Hello,

Am 12.11.2017 um 18:42 schrieb Yann E. MORIN:
> On 2017-11-12 18:24 +0100, Thomas Petazzoni spake thusly:
>> Hello,
>>
>> On Sun, 12 Nov 2017 18:15:12 +0100, Peter Korsgaard wrote:
>>
>>>   > So this happens because the ARC binutils version is fetched from Git,
>>>   > so we need to have host-flex installed, and binutils detects the flex
>>>   > library and decides to use it.
>>>
>>> And there is no configure flag to disable that?
>>
>> Not that I can see.
> 
> A long time ago (when I was too young to have a beard), libfl was only
> available as a static library.
> 
> That library contained (and stil contains) a 'main()' symbol (is it even
> a weak one?), so it really is a weird library to begin with... I would
> not mind building that host library with --disable-shared --enable-static
> and only install the static library...
> 
> That would solve this specific issue.
> 
> Until we need another library...
> 
>>>   >> - The absolute rpath in $(HOST_DIR)/<tuple>/bin/ar is wrong in the
>>>   >> first place, but I'm not sure how to fix this.
>>>
>>>   > I'm wrong on this: $(HOST_DIR)/<tuple>/bin/ar RPATH is totally fine:
>>>
>>>   > $ readelf -d arc-buildroot-linux-uclibc/bin/ar  | grep rpath
>>>   >  0x000000000000000f (RPATH)              Library rpath: [/opt/br-arcle-hs38-full-2017.11-rc1/lib]
>>>
>>>   > The problem is that this gets turned into $ORIGIN/../lib by patchelf.
>>>
>>>   > I don't see any other solution than de-duplicating such binaries.
>>>
>>>   > Do you see another option ?
>>>
>>> Not if we *need* both binaries (do we?)
>>
>> I think we do need both binaries. <tuple>-<tool> is the "publicly"
>> visible copy, while <tuple>/bin/<tool> is the one called internally by
>> gcc (for example when it calls as or ld). Yann can explain more about
>> this perhaps.
> 
> I can't explain much more than you did, sorry...
> 
>>> Is there a way to get binutils to use soft links instead of hard links?
>>
>> How would this solve the problem?
> 
> I guess the reasoning of Peter was that maybe $ORIGIN is relative to the
> real file not the symlink.
> 
> Thus by having <tuple>/bin/<tool> a symlink to ../../bin/<tuple>-<tool>
> would solve the issue.
> 
> But I am not sure of it either, and the manpage for ld.so does not
> mention that. It only states:
> 
>      $ORIGIN (or equivalently ${ORIGIN})
>          This expands to the directory containing the program or
>          shared object.
> 
> No mention of the behaviour when the program is being called through a
> symlink in another directory... :-/

A symlink seem to be resolved before using $ORIGIN:

  $ cd <buildroot-build-dir>

  $ ldd host/bin/xgettext
  ...
	libgettextsrc-0.19.8.1.so => /work/bdo/dcu/host/bin/../lib/libgettextsrc-0.19.8.1.so (0x00007f2ae6832000)
	libgettextlib-0.19.8.1.so => /work/bdo/dcu/host/bin/../lib/libgettextlib-0.19.8.1.so (0x00007f2ae652f000)

  # Hard link
  $ ln host/bin/xgettext xgettext
  $ ./xgettext
  ./xgettext: error while loading shared libraries: libgettextsrc-0.19.8.1.so: cannot open shared object file: No such file or directory

  # Soft link:
  $ rm xgettext
  $ ln -s ost/bin/xgettext xgettext
  $ ./xgettext
  ./xgettext: Eingabedatei fehlt

But:

  $ ldd xgettext
  ... 
	libgettextsrc-0.19.8.1.so => not found
	libgettextlib-0.19.8.1.so => not found

This means: A symbolic link would work.

Wofgang.

[Buildroot] Bug in RPATH fixing logic

[Peter Korsgaard]

>>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:

Hi,

 > This means: A symbolic link would work.

Ok, great - Makes sense. I wonder how much effort it wil be to get
binutils to generate symbolic links instead of hard links.

-- 
Bye, Peter Korsgaard

[Buildroot] Bug in RPATH fixing logic

[Wolfgang Grandegger]

Hello,

Am 13.11.2017 um 14:50 schrieb Peter Korsgaard:
>>>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:
> 
> Hi,
> 
>   > This means: A symbolic link would work.
> 
> Ok, great - Makes sense. I wonder how much effort it wil be to get
> binutils to generate symbolic links instead of hard links.
> 

If I look into the HOST_DIR of my X86_64 build, I find the attached files
with hard links. For example "ar" is in:

./bin/x86_64-buildroot-linux-gnu-ar: Device: 811h/2065d	Inode: 52043071    Links: 2
./x86_64-buildroot-linux-gnu/bin/ar: Device: 811h/2065d	Inode: 52043071    Links: 2

$ patchelf --print-rpath ./bin/x86_64-buildroot-linux-gnu-ar
/work/agco/bdo/dcu/host/lib

And this RPATH will be removed for the known reasons. Trying to use
symbolic instead of hard links in the binutils now...

Wolfgang.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hard-links-x86_64-build.log
Type: text/x-log
Size: 3092 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20171113/fbbb79e4/attachment.bin>

[Buildroot] Bug in RPATH fixing logic

[Yann E. MORIN]

Peter, All,

On 2017-11-13 14:50 +0100, Peter Korsgaard spake thusly:
> >>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:
>  > This means: A symbolic link would work.
> Ok, great - Makes sense. I wonder how much effort it wil be to get
> binutils to generate symbolic links instead of hard links.

No way.

We'll have to provide post-install hooks to do the de-hardlinking step
ourselves.

Also, as Wolfgang noticed, gcc is also affected by the hardlink issue.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Bug in RPATH fixing logic

[Wolfgang Grandegger]

Hello,

Am 13.11.2017 um 18:36 schrieb Yann E. MORIN:
> Peter, All,
> 
> On 2017-11-13 14:50 +0100, Peter Korsgaard spake thusly:
>>>>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:
>>   > This means: A symbolic link would work.
>> Ok, great - Makes sense. I wonder how much effort it wil be to get
>> binutils to generate symbolic links instead of hard links.
> 
> No way.

Well, the attached patch fixes the issue with the binary utilities (ld,
ar, objdump, as, ...). I think only the fixes in "binutils/Makefile.in" and
"gas/Makefile.in" are need, IIRC. And in the fixes did not break the
build, at least. Will do more checks...

> We'll have to provide post-install hooks to do the de-hardlinking step
> ourselves.

> Also, as Wolfgang noticed, gcc is also affected by the hardlink issue.

No, with the patch above I get:

./bin/x86_64-buildroot-linux-gnu-gcc-6.4.0.br_real: Device: 811h/2065d	Inode: 64370910    Links: 3
./bin/x86_64-buildroot-linux-gnu-g++.br_real: Device: 811h/2065d	Inode: 64370887    Links: 2
./bin/x86_64-buildroot-linux-gnu-gcc.br_real: Device: 811h/2065d	Inode: 64370910    Links: 3
./bin/x86_64-buildroot-linux-gnu-c++.br_real: Device: 811h/2065d	Inode: 64370887    Links: 2
./bin/x86_64-buildroot-linux-gnu-cc.br_real: Device: 811h/2065d	Inode: 64370910    Links: 3
./x86_64-buildroot-linux-gnu/sysroot/usr/libexec/getconf/XBS5_LP64_OFF64: Device: 811h/2065d	Inode: 72747785    Links: 4
./x86_64-buildroot-linux-gnu/sysroot/usr/libexec/getconf/POSIX_V6_LP64_OFF64: Device: 811h/2065d	Inode: 72747785    Links: 4
./x86_64-buildroot-linux-gnu/sysroot/usr/libexec/getconf/POSIX_V7_LP64_OFF64: Device: 811h/2065d	Inode: 72747785    Links: 4
./x86_64-buildroot-linux-gnu/sysroot/usr/bin/getconf: Device: 811h/2065d	Inode: 72747785    Links: 4
./x86_64-buildroot-linux-gnu/sysroot/usr/lib/dri/i965_dri.so: Device: 811h/2065d	Inode: 72748076    Links: 2
./x86_64-buildroot-linux-gnu/sysroot/usr/lib/dri/i915_dri.so: Device: 811h/2065d	Inode: 72748076    Links: 2

The gcc hard links are all in the same directory. Just "getconf" remains,
but the RPATH is already empty.

Wolfgang.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: binutils-use-symbolic-instead-of-hard-links.patch
Type: text/x-patch
Size: 6710 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20171113/d079cb1b/attachment.bin>

[Buildroot] Bug in RPATH fixing logic

[Peter Korsgaard]

>>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:

 > Hello,
 > Am 13.11.2017 um 18:36 schrieb Yann E. MORIN:
 >> Peter, All,
 >> 
 >> On 2017-11-13 14:50 +0100, Peter Korsgaard spake thusly:
 >>>>>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:
 >>> > This means: A symbolic link would work.
 >>> Ok, great - Makes sense. I wonder how much effort it wil be to get
 >>> binutils to generate symbolic links instead of hard links.
 >> 
 >> No way.

 > Well, the attached patch fixes the issue with the binary utilities (ld,
 > ar, objdump, as, ...). I think only the fixes in "binutils/Makefile.in" and
 > "gas/Makefile.in" are need, IIRC. And in the fixes did not break the
 > build, at least. Will do more checks...

Yes, either something like that or do something to cause the ln to fail
as it then falls back to use install (E.G. sed 's/\bln\b/false/' symlink
ln to false and stick that in the path).

The best solution would naturally be something that we could upstream,
E.G. way to configure the ln command to be used, but I don't know how
open upstream would be to that.

-- 
Bye, Peter Korsgaard

[Buildroot] Bug in RPATH fixing logic

[Yann E. MORIN]

On 2017-11-13 19:04 +0100, Wolfgang Grandegger spake thusly:
> Hello,
> 
> Am 13.11.2017 um 18:36 schrieb Yann E. MORIN:
> > Peter, All,
> > 
> > On 2017-11-13 14:50 +0100, Peter Korsgaard spake thusly:
> >>>>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:
> >>   > This means: A symbolic link would work.
> >> Ok, great - Makes sense. I wonder how much effort it wil be to get
> >> binutils to generate symbolic links instead of hard links.
> > 
> > No way.
> 
> Well, the attached patch fixes the issue with the binary utilities (ld,
> ar, objdump, as, ...). I think only the fixes in "binutils/Makefile.in" and
> "gas/Makefile.in" are need, IIRC. And in the fixes did not break the
> build, at least. Will do more checks...

Well, what I meant is that there are no --disable-hardlinks on the
ocnfigure command line.

And don;t forget that we do have 4 versions of binutils to patch.

And as Peter said: if we patch, we should try to get that upstreamable.
Maybe upstream will be receptive to that. Or maybe not. Who knows...

I think providing a post-install hook is better than patching.

> > We'll have to provide post-install hooks to do the de-hardlinking step
> > ourselves.
> 
> > Also, as Wolfgang noticed, gcc is also affected by the hardlink issue.
> 
> No, with the patch above I get:
> 
> ./bin/x86_64-buildroot-linux-gnu-gcc-6.4.0.br_real: Device: 811h/2065d	Inode: 64370910    Links: 3
> ./bin/x86_64-buildroot-linux-gnu-g++.br_real: Device: 811h/2065d	Inode: 64370887    Links: 2
> ./bin/x86_64-buildroot-linux-gnu-gcc.br_real: Device: 811h/2065d	Inode: 64370910    Links: 3
> ./bin/x86_64-buildroot-linux-gnu-c++.br_real: Device: 811h/2065d	Inode: 64370887    Links: 2
> ./bin/x86_64-buildroot-linux-gnu-cc.br_real: Device: 811h/2065d	Inode: 64370910    Links: 3
> ./x86_64-buildroot-linux-gnu/sysroot/usr/libexec/getconf/XBS5_LP64_OFF64: Device: 811h/2065d	Inode: 72747785    Links: 4
> ./x86_64-buildroot-linux-gnu/sysroot/usr/libexec/getconf/POSIX_V6_LP64_OFF64: Device: 811h/2065d	Inode: 72747785    Links: 4
> ./x86_64-buildroot-linux-gnu/sysroot/usr/libexec/getconf/POSIX_V7_LP64_OFF64: Device: 811h/2065d	Inode: 72747785    Links: 4
> ./x86_64-buildroot-linux-gnu/sysroot/usr/bin/getconf: Device: 811h/2065d	Inode: 72747785    Links: 4
> ./x86_64-buildroot-linux-gnu/sysroot/usr/lib/dri/i965_dri.so: Device: 811h/2065d	Inode: 72748076    Links: 2
> ./x86_64-buildroot-linux-gnu/sysroot/usr/lib/dri/i915_dri.so: Device: 811h/2065d	Inode: 72748076    Links: 2
> 
> The gcc hard links are all in the same directory.

Right, that's OK.

> Just "getconf" remains,
> but the RPATH is already empty.

And those are binaries for the target, not the host.

Regards,
Yann E. MORIN.

> Wolfgang.

> Index: host-binutils-2.28.1/ld/Makefile.am
> ===================================================================
> --- host-binutils-2.28.1.orig/ld/Makefile.am
> +++ host-binutils-2.28.1/ld/Makefile.am
> @@ -2226,17 +2226,17 @@ install-exec-local: ld-new$(EXEEXT) inst
>  	n=`echo $(installed_linker) | sed '$(transform)'`; \
>  	if test "$(bindir)" != "$(tooldir)/bin"; then \
>  	  rm -f $(DESTDIR)$(tooldir)/bin/$(installed_linker)$(EXEEXT); \
> -	  ln $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$(installed_linker)$(EXEEXT) >/dev/null 2>/dev/null \
> +	  ln -s $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$(installed_linker)$(EXEEXT) >/dev/null 2>/dev/null \
>  	  || $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) ld-new$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$(installed_linker)$(EXEEXT); \
>  	fi; \
>  	if test "x$(install_as_default)" = "xyes"; then \
>  	  ld=`echo ld | sed '$(transform)'`; \
>  	  rm -f $(DESTDIR)$(bindir)/$$ld$(EXEEXT); \
> -	  ln $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(bindir)/$$ld$(EXEEXT) >/dev/null 2>/dev/null \
> +	  ln -s $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(bindir)/$$ld$(EXEEXT) >/dev/null 2>/dev/null \
>  	  || $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) ld-new$(EXEEXT) $(DESTDIR)$(bindir)/$$ld$(EXEEXT); \
>  	  if test "$(bindir)" != "$(tooldir)/bin"; then \
>  	    rm -f $(DESTDIR)$(tooldir)/bin/ld$(EXEEXT); \
> -	    ln $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/ld$(EXEEXT) >/dev/null 2>/dev/null \
> +	    ln -s $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/ld$(EXEEXT) >/dev/null 2>/dev/null \
>  	    || $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) ld-new$(EXEEXT) $(DESTDIR)$(tooldir)/bin/ld$(EXEEXT); \
>  	  fi; \
>  	fi
> Index: host-binutils-2.28.1/ld/Makefile.in
> ===================================================================
> --- host-binutils-2.28.1.orig/ld/Makefile.in
> +++ host-binutils-2.28.1/ld/Makefile.in
> @@ -3738,17 +3738,17 @@ install-exec-local: ld-new$(EXEEXT) inst
>  	n=`echo $(installed_linker) | sed '$(transform)'`; \
>  	if test "$(bindir)" != "$(tooldir)/bin"; then \
>  	  rm -f $(DESTDIR)$(tooldir)/bin/$(installed_linker)$(EXEEXT); \
> -	  ln $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$(installed_linker)$(EXEEXT) >/dev/null 2>/dev/null \
> +	  ln -s $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$(installed_linker)$(EXEEXT) >/dev/null 2>/dev/null \
>  	  || $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) ld-new$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$(installed_linker)$(EXEEXT); \
>  	fi; \
>  	if test "x$(install_as_default)" = "xyes"; then \
>  	  ld=`echo ld | sed '$(transform)'`; \
>  	  rm -f $(DESTDIR)$(bindir)/$$ld$(EXEEXT); \
> -	  ln $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(bindir)/$$ld$(EXEEXT) >/dev/null 2>/dev/null \
> +	  ln -s $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(bindir)/$$ld$(EXEEXT) >/dev/null 2>/dev/null \
>  	  || $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) ld-new$(EXEEXT) $(DESTDIR)$(bindir)/$$ld$(EXEEXT); \
>  	  if test "$(bindir)" != "$(tooldir)/bin"; then \
>  	    rm -f $(DESTDIR)$(tooldir)/bin/ld$(EXEEXT); \
> -	    ln $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/ld$(EXEEXT) >/dev/null 2>/dev/null \
> +	    ln -s $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/ld$(EXEEXT) >/dev/null 2>/dev/null \
>  	    || $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) ld-new$(EXEEXT) $(DESTDIR)$(tooldir)/bin/ld$(EXEEXT); \
>  	  fi; \
>  	fi
> Index: host-binutils-2.28.1/binutils/Makefile.in
> ===================================================================
> --- host-binutils-2.28.1.orig/binutils/Makefile.in
> +++ host-binutils-2.28.1/binutils/Makefile.in
> @@ -1512,7 +1512,7 @@ install-exec-local: install-binPROGRAMS
>  	    k=`echo $$j | sed '$(transform)'`; \
>  	    if [ "$(bindir)/$$k" != "$(tooldir)/bin/$$j" ]; then \
>  	      rm -f $(DESTDIR)$(tooldir)/bin/$$j$(EXEEXT); \
> -	      ln $(DESTDIR)$(bindir)/$$k$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$$j$(EXEEXT) >/dev/null 2>/dev/null \
> +	      ln -s $(DESTDIR)$(bindir)/$$k$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$$j$(EXEEXT) >/dev/null 2>/dev/null \
>  		|| $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) $$i$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$$j$(EXEEXT); \
>  	    fi; \
>  	  else true; \
> Index: host-binutils-2.28.1/gas/Makefile.in
> ===================================================================
> --- host-binutils-2.28.1.orig/gas/Makefile.in
> +++ host-binutils-2.28.1/gas/Makefile.in
> @@ -2698,7 +2698,7 @@ install-exec-tooldir: install-exec-bindi
>  	n=`echo as | sed '$(transform)'`; \
>  	if [ "$(bindir)/$$n$(EXEEXT)" != "$(tooldir)/bin/as$(EXEEXT)" ]; then \
>  	  rm -f $(DESTDIR)$(tooldir)/bin/as$(EXEEXT); \
> -	  ln $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/as$(EXEEXT) >/dev/null 2>/dev/null \
> +	  ln -s $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/as$(EXEEXT) >/dev/null 2>/dev/null \
>  	    || $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) as-new$(EXEEXT) $(DESTDIR)$(tooldir)/bin/as$(EXEEXT); \
>  	else \
>  	  true ; \
> Index: host-binutils-2.28.1/gas/Makefile.am
> ===================================================================
> --- host-binutils-2.28.1.orig/gas/Makefile.am
> +++ host-binutils-2.28.1/gas/Makefile.am
> @@ -617,7 +617,7 @@ install-exec-tooldir: install-exec-bindi
>  	n=`echo as | sed '$(transform)'`; \
>  	if [ "$(bindir)/$$n$(EXEEXT)" != "$(tooldir)/bin/as$(EXEEXT)" ]; then \
>  	  rm -f $(DESTDIR)$(tooldir)/bin/as$(EXEEXT); \
> -	  ln $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/as$(EXEEXT) >/dev/null 2>/dev/null \
> +	  ln -s $(DESTDIR)$(bindir)/$$n$(EXEEXT) $(DESTDIR)$(tooldir)/bin/as$(EXEEXT) >/dev/null 2>/dev/null \
>  	    || $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) as-new$(EXEEXT) $(DESTDIR)$(tooldir)/bin/as$(EXEEXT); \
>  	else \
>  	  true ; \
> Index: host-binutils-2.28.1/binutils/Makefile.am
> ===================================================================
> --- host-binutils-2.28.1.orig/binutils/Makefile.am
> +++ host-binutils-2.28.1/binutils/Makefile.am
> @@ -547,7 +547,7 @@ install-exec-local: install-binPROGRAMS
>  	    k=`echo $$j | sed '$(transform)'`; \
>  	    if [ "$(bindir)/$$k" != "$(tooldir)/bin/$$j" ]; then \
>  	      rm -f $(DESTDIR)$(tooldir)/bin/$$j$(EXEEXT); \
> -	      ln $(DESTDIR)$(bindir)/$$k$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$$j$(EXEEXT) >/dev/null 2>/dev/null \
> +	      ln -s $(DESTDIR)$(bindir)/$$k$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$$j$(EXEEXT) >/dev/null 2>/dev/null \
>  		|| $(INSTALL_PROGRAM_ENV) $(LIBTOOL) --mode=install $(INSTALL_PROGRAM) $$i$(EXEEXT) $(DESTDIR)$(tooldir)/bin/$$j$(EXEEXT); \
>  	    fi; \
>  	  else true; \


-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Bug in RPATH fixing logic

[Wolfgang Grandegger]

Hello,

Am 13.11.2017 um 21:33 schrieb Yann E. MORIN:
> On 2017-11-13 19:04 +0100, Wolfgang Grandegger spake thusly:
>> Hello,
>>
>> Am 13.11.2017 um 18:36 schrieb Yann E. MORIN:
>>> Peter, All,
>>>
>>> On 2017-11-13 14:50 +0100, Peter Korsgaard spake thusly:
>>>>>>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:
>>>>    > This means: A symbolic link would work.
>>>> Ok, great - Makes sense. I wonder how much effort it wil be to get
>>>> binutils to generate symbolic links instead of hard links.
>>>
>>> No way.
>>
>> Well, the attached patch fixes the issue with the binary utilities (ld,
>> ar, objdump, as, ...). I think only the fixes in "binutils/Makefile.in" and
>> "gas/Makefile.in" are need, IIRC. And in the fixes did not break the
>> build, at least. Will do more checks...
> 
> Well, what I meant is that there are no --disable-hardlinks on the
> ocnfigure command line.
> 
> And don;t forget that we do have 4 versions of binutils to patch.
> 
> And as Peter said: if we patch, we should try to get that upstreamable.
> Maybe upstream will be receptive to that. Or maybe not. Who knows...
> 
> I think providing a post-install hook is better than patching.

I agree.

> 
>>> We'll have to provide post-install hooks to do the de-hardlinking step
>>> ourselves.
>>
>>> Also, as Wolfgang noticed, gcc is also affected by the hardlink issue.
>>
>> No, with the patch above I get:
>>
>> ./bin/x86_64-buildroot-linux-gnu-gcc-6.4.0.br_real: Device: 811h/2065d	Inode: 64370910    Links: 3
>> ./bin/x86_64-buildroot-linux-gnu-g++.br_real: Device: 811h/2065d	Inode: 64370887    Links: 2
>> ./bin/x86_64-buildroot-linux-gnu-gcc.br_real: Device: 811h/2065d	Inode: 64370910    Links: 3
>> ./bin/x86_64-buildroot-linux-gnu-c++.br_real: Device: 811h/2065d	Inode: 64370887    Links: 2
>> ./bin/x86_64-buildroot-linux-gnu-cc.br_real: Device: 811h/2065d	Inode: 64370910    Links: 3
>> ./x86_64-buildroot-linux-gnu/sysroot/usr/libexec/getconf/XBS5_LP64_OFF64: Device: 811h/2065d	Inode: 72747785    Links: 4
>> ./x86_64-buildroot-linux-gnu/sysroot/usr/libexec/getconf/POSIX_V6_LP64_OFF64: Device: 811h/2065d	Inode: 72747785    Links: 4
>> ./x86_64-buildroot-linux-gnu/sysroot/usr/libexec/getconf/POSIX_V7_LP64_OFF64: Device: 811h/2065d	Inode: 72747785    Links: 4
>> ./x86_64-buildroot-linux-gnu/sysroot/usr/bin/getconf: Device: 811h/2065d	Inode: 72747785    Links: 4
>> ./x86_64-buildroot-linux-gnu/sysroot/usr/lib/dri/i965_dri.so: Device: 811h/2065d	Inode: 72748076    Links: 2
>> ./x86_64-buildroot-linux-gnu/sysroot/usr/lib/dri/i915_dri.so: Device: 811h/2065d	Inode: 72748076    Links: 2
>>
>> The gcc hard links are all in the same directory.
> 
> Right, that's OK.
> 
>> Just "getconf" remains,
>> but the RPATH is already empty.
> 
> And those are binaries for the target, not the host.

Well, but they are treated as host libraries.

Wolfgang.

[Buildroot] Bug in RPATH fixing logic

[Wolfgang Grandegger]

Am 14.11.2017 um 09:00 schrieb Wolfgang Grandegger:
> Hello,
> 
> Am 13.11.2017 um 21:33 schrieb Yann E. MORIN:
>> On 2017-11-13 19:04 +0100, Wolfgang Grandegger spake thusly:
>>> Hello,
>>>
>>> Am 13.11.2017 um 18:36 schrieb Yann E. MORIN:
>>>> Peter, All,
>>>>
>>>> On 2017-11-13 14:50 +0100, Peter Korsgaard spake thusly:
>>>>>>>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:
>>>>> ?? > This means: A symbolic link would work.
>>>>> Ok, great - Makes sense. I wonder how much effort it wil be to get
>>>>> binutils to generate symbolic links instead of hard links.
>>>>
>>>> No way.
>>>
>>> Well, the attached patch fixes the issue with the binary utilities (ld,
>>> ar, objdump, as, ...). I think only the fixes in 
>>> "binutils/Makefile.in" and
>>> "gas/Makefile.in" are need, IIRC. And in the fixes did not break the
>>> build, at least. Will do more checks...
>>
>> Well, what I meant is that there are no --disable-hardlinks on the
>> ocnfigure command line.
>>
>> And don;t forget that we do have 4 versions of binutils to patch.
>>
>> And as Peter said: if we patch, we should try to get that upstreamable.
>> Maybe upstream will be receptive to that. Or maybe not. Who knows...
>>
>> I think providing a post-install hook is better than patching.
> 
> I agree.

You mean post-install for the "binutil" packages only? A generic 
solution would be better, but not straight-forward, unfortunately.

Do we want to have symbolic instead of hard links or is a special RPATH 
with directories for all hard linked ELF files acceptable?

Wolfgang.

[Buildroot] Bug in RPATH fixing logic

[Peter Korsgaard]

>>>>> "Wolfgang" == Wolfgang Grandegger <wg@grandegger.com> writes:

Hi,

 >>> I think providing a post-install hook is better than patching.
 >> 
 >> I agree.

 > You mean post-install for the "binutil" packages only? A generic
 > solution would be better, but not straight-forward, unfortunately.

I guess we could do something with find -type f -links +1 and then stat
those files to find the inode number and search for the other links with
find -inum <inode>, but yeah, it is not going to be pretty.

A package specific post-install hook is most likely going to be a lot
simpler/cleaner.

 > Do we want to have symbolic instead of hard links or is a special
 > RPATH with directories for all hard linked ELF files acceptable?

I think using symlinks is the simplest solution.

-- 
Bye, Peter Korsgaard