glibc binary reproducibility

glibc binary reproducibility

[Douglas Royds]

[-- Attachment #1: Type: text/plain, Size: 3529 bytes --]

When I build glibc in two different places, I get non-reproducible 
results - a 4-byte difference:

    $ cmp -l
    ~/workspace/upstream[12]/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/package/lib/libc-2.28.so
    1259181  71 172
    1259182  27 304
    1259183 152  77
    1259184 363 243

These 4 bytes are the checksum that objcopy --add-gnu-debuglink adds to 
the binary when the .debug info is split out at do_package time, see 
package.bbclass +416

So why is the debug info different? We add -fdebug-prefix-map to our 
CFLAGS, which ensures that all our intra-component debug paths are 
prefixed with /usr/src/debug/glibc/, for instance, but this isn't 
working in this case. The difference is happening in csu/crt1.o, which 
is being linked into libc.so (and others):

    $
    ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump
    -t csu/crt1.o
    ...
    00000000 l    df *ABS*    00000000
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
    00000000 l    df *ABS*    00000000
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o

This abi-note.o symbol is finding its way into libc.so:

    $
    ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump
    -t libc.so
    ...
    00000000 l    df *ABS*    00000000
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o

So where does csu/crt1.o come from?

    $ arm-tait-linux-gnueabi-gcc  -march=armv5te -marm -mcpu=arm926ej-s
    --sysroot=/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/recipe-sysroot
    -nostdlib -nostartfiles *-r -o
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crt1.o*/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/init.o
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/static-reloc.o

Note the -r = --relocatable, an ld option, which, "Generate[s] 
relocatable output---i.e., generate[s] an output file that can in turn 
serve as input to ld.  This is often called partial linking", ie. the 
glibc build is just putting these .o files together for later convenience.

Regrettably, this command both ignores -fdebug-prefix-map (which ld 
doesn't accept) and puts the fully-qualified path to some of the input 
.o files in the resulting crt1.o. At package splitdebuginfo() time, 
although the fully-qualified path info is split off into the .debug 
files, a (relative) path to the .debug files plus a checksum is tacked 
onto libc.so by objcopy --add-gnu-debuglink ... and the checksum depends 
on the path to the build.

There is a work-around: turn off the debug packaging:

    INHIBIT_PACKAGE_DEBUG_SPLIT_pn-glibc = "1"

I don't have a solution for this. Suggestions?


[-- Attachment #2: Type: text/html, Size: 4472 bytes --]

Re: glibc binary reproducibility

[Douglas Royds]

[-- Attachment #1: Type: text/plain, Size: 3769 bytes --]

The relocatable link is defined in glibc csu/Makefile line 125


On 27/09/18 12:18, Douglas Royds wrote:

> When I build glibc in two different places, I get non-reproducible 
> results - a 4-byte difference:
>
>     $ cmp -l
>     ~/workspace/upstream[12]/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/package/lib/libc-2.28.so
>     1259181  71 172
>     1259182  27 304
>     1259183 152  77
>     1259184 363 243
>
> These 4 bytes are the checksum that objcopy --add-gnu-debuglink adds 
> to the binary when the .debug info is split out at do_package time, 
> see package.bbclass +416
>
> So why is the debug info different? We add -fdebug-prefix-map to our 
> CFLAGS, which ensures that all our intra-component debug paths are 
> prefixed with /usr/src/debug/glibc/, for instance, but this isn't 
> working in this case. The difference is happening in csu/crt1.o, which 
> is being linked into libc.so (and others):
>
>     $
>     ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump
>     -t csu/crt1.o
>     ...
>     00000000 l    df *ABS*    00000000
>     /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
>     00000000 l    df *ABS*    00000000
>     /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
>
> This abi-note.o symbol is finding its way into libc.so:
>
>     $
>     ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump
>     -t libc.so
>     ...
>     00000000 l    df *ABS*    00000000
>     /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
>
> So where does csu/crt1.o come from?
>
>     $ arm-tait-linux-gnueabi-gcc  -march=armv5te -marm
>     -mcpu=arm926ej-s
>     --sysroot=/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/recipe-sysroot
>     -nostdlib -nostartfiles *-r -o
>     /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crt1.o*/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
>     /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
>     /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/init.o
>     /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/static-reloc.o
>
> Note the -r = --relocatable, an ld option, which, "Generate[s] 
> relocatable output---i.e., generate[s] an output file that can in turn 
> serve as input to ld.  This is often called partial linking", ie. the 
> glibc build is just putting these .o files together for later convenience.
>
> Regrettably, this command both ignores -fdebug-prefix-map (which ld 
> doesn't accept) and puts the fully-qualified path to some of the input 
> .o files in the resulting crt1.o. At package splitdebuginfo() time, 
> although the fully-qualified path info is split off into the .debug 
> files, a (relative) path to the .debug files plus a checksum is tacked 
> onto libc.so by objcopy --add-gnu-debuglink ... and the checksum 
> depends on the path to the build.
>
> There is a work-around: turn off the debug packaging:
>
>     INHIBIT_PACKAGE_DEBUG_SPLIT_pn-glibc = "1"
>
> I don't have a solution for this. Suggestions?
>


[-- Attachment #2: Type: text/html, Size: 5047 bytes --]

Re: glibc binary reproducibility

[Richard Purdie]

On Thu, 2018-09-27 at 12:18 +1200, Douglas Royds wrote:
> > $ arm-tait-linux-gnueabi-gcc  -march=armv5te -marm -mcpu=arm926ej-s 
> > --sysroot=/home/douglas/workspace/upstream1/build/tmp/work/armv5e-
> > tait-linux-gnueabi/glibc/2.28-r0/recipe-sysroot -nostdlib
> > -nostartfiles -r -o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crt1.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/init.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/static-
> > reloc.o
> 
> Note the -r = --relocatable, an ld option, which, "Generate[s]
> relocatable output---i.e., generate[s] an output file that can in
> turn serve as input to ld.  This is often called partial linking",
> ie. the glibc build is just putting these .o files together for later
> convenience.
> Regrettably, this command both ignores -fdebug-prefix-map (which ld
> doesn't accept) and puts the fully-qualified path to some of the
> input .o files in the resulting crt1.o. At package splitdebuginfo()
> time, although the fully-qualified path info is split off into the
> .debug files, a (relative) path to the .debug files plus a checksum
> is tacked onto libc.so by objcopy --add-gnu-debuglink ... and the
> checksum depends on the path to the build.
> There is a work-around: turn off the debug packaging:
> > INHIBIT_PACKAGE_DEBUG_SPLIT_pn-glibc = "1"
> 
> I don't have a solution for this. Suggestions?

Good work in tracking it down so far.

Going off a bit of a random memory fragment, would it help to use
relative paths in the compile/link command?

Cheers,

Richard

Re: glibc binary reproducibility

[Richard Purdie]

On Thu, 2018-09-27 at 12:18 +1200, Douglas Royds wrote:
> When I build glibc in two different places, I get non-reproducible
> results - a 4-byte difference:
> > $ cmp -l ~/workspace/upstream[12]/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/package/lib/libc-2.28.so
> > 1259181  71 172
> > 1259182  27 304
> > 1259183 152  77
> > 1259184 363 243
> 
> These 4 bytes are the checksum that objcopy --add-gnu-debuglink adds
> to the binary when the .debug info is split out at do_package time,
> see package.bbclass +416
> So why is the debug info different? We add -fdebug-prefix-map to our
> CFLAGS, which ensures that all our intra-component debug paths are
> prefixed with /usr/src/debug/glibc/, for instance, but this isn't
> working in this case. The difference is happening in csu/crt1.o,
> which is being linked into libc.so (and others):
> > $ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
> > linux-gnueabi-objdump -t csu/crt1.o
> > ...
> > 00000000 l    df *ABS*    00000000
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
> > 00000000 l    df *ABS*    00000000
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
>  This abi-note.o symbol is finding its way into libc.so: 
> > $ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
> > linux-gnueabi-objdump -t libc.so
> > ...
> > 00000000 l    df *ABS*    00000000             
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
>  So where does csu/crt1.o come from?
> > 
> > $ arm-tait-linux-gnueabi-gcc  -march=armv5te -marm -mcpu=arm926ej-s 
> > --sysroot=/home/douglas/workspace/upstream1/build/tmp/work/armv5e-
> > tait-linux-gnueabi/glibc/2.28-r0/recipe-sysroot -nostdlib
> > -nostartfiles -r -o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crt1.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/init.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/static-
> > reloc.o
> 
> Note the -r = --relocatable, an ld option, which, "Generate[s]
> relocatable output---i.e., generate[s] an output file that can in
> turn serve as input to ld.  This is often called partial linking",
> ie. the glibc build is just putting these .o files together for later
> convenience.
> Regrettably, this command both ignores -fdebug-prefix-map (which ld
> doesn't accept) and puts the fully-qualified path to some of the
> input .o files in the resulting crt1.o. At package splitdebuginfo()
> time, although the fully-qualified path info is split off into the
> .debug files, a (relative) path to the .debug files plus a checksum
> is tacked onto libc.so by objcopy --add-gnu-debuglink ... and the
> checksum depends on the path to the build.
> There is a work-around: turn off the debug packaging:
> > INHIBIT_PACKAGE_DEBUG_SPLIT_pn-glibc = "1"
> 
> I don't have a solution for this. Suggestions?

I did some digging.

I tried what I suggested using relative paths:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=22189be4bf508851950f72654870c4eebd4b73d9

and whilst it helps remove some references, there is a much wider
problem. I therefore went and had a look at the linker itself to
understand why its injecting this path. I found this code:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=f9aca51204990fbdbdfa7442f1dcc938e97bf782

which if disabled as per this hack, makes the binaries reproducible and
drops the problematic references.

We're swapping some misleading debug output for reproducibility with
that hack.

At this point we probably need to talk to some toolchain people about
what 'real' fixes may be possible...

Cheers,

Richard

Re: glibc binary reproducibility

[Douglas Royds]

[-- Attachment #1: Type: text/plain, Size: 6814 bytes --]

On 03/10/18 05:58, Richard Purdie wrote:

> On Thu, 2018-09-27 at 12:18 +1200, Douglas Royds wrote:
>> When I build glibc in two different places, I get non-reproducible
>> results - a 4-byte difference:
>>> $ cmp -l ~/workspace/upstream[12]/build/tmp/work/armv5e-tait-linux-
>>> gnueabi/glibc/2.28-r0/package/lib/libc-2.28.so
>>> 1259181  71 172
>>> 1259182  27 304
>>> 1259183 152  77
>>> 1259184 363 243
>> These 4 bytes are the checksum that objcopy --add-gnu-debuglink adds
>> to the binary when the .debug info is split out at do_package time,
>> see package.bbclass +416
>> So why is the debug info different? We add -fdebug-prefix-map to our
>> CFLAGS, which ensures that all our intra-component debug paths are
>> prefixed with /usr/src/debug/glibc/, for instance, but this isn't
>> working in this case. The difference is happening in csu/crt1.o,
>> which is being linked into libc.so (and others):
>>> $ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
>>> linux-gnueabi-objdump -t csu/crt1.o
>>> ...
>>> 00000000 l    df *ABS*    00000000
>>> /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
>>> gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
>>> 00000000 l    df *ABS*    00000000
>>> /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
>>> gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
>>   This abi-note.o symbol is finding its way into libc.so:
>>> $ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
>>> linux-gnueabi-objdump -t libc.so
>>> ...
>>> 00000000 l    df *ABS*    00000000
>>> /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
>>> gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o


No it isn't, it turns out. The non-reproducibility of crt1.o and friends 
is a problem, as they get copied into the recipe-sysroot of all other 
components, causing the 780ish different packages observed by Juro in 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12743


>> There is a work-around: turn off the debug packaging:
>>> INHIBIT_PACKAGE_DEBUG_SPLIT_pn-glibc = "1"
>> I don't have a solution for this. Suggestions?
> I did some digging.
>
> I tried what I suggested using relative paths:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=22189be4bf508851950f72654870c4eebd4b73d9
>
> and whilst it helps remove some references, there is a much wider
> problem. I therefore went and had a look at the linker itself to
> understand why its injecting this path. I found this code:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=f9aca51204990fbdbdfa7442f1dcc938e97bf782
>
> which if disabled as per this hack, makes the binaries reproducible and
> drops the problematic references.
>
> We're swapping some misleading debug output for reproducibility with
> that hack.
>
> At this point we probably need to talk to some toolchain people about
> what 'real' fixes may be possible...


Excellent. Your patch in 22189be is much tidier than my own Makefile 
hack that I was too embarassed to publish on this list (it may or may 
not have involved sedding a binary file).

It turns out that we have two binary difference issues: The crt1.o (and 
friends) one discussed above, and a similar problem linking all the 
shared libs, eg. libm.so

    $
    ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump
    -t math/libm.so | grep armv5e
    00000000 l    df *ABS*    00000000
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
    00000000 l    df *ABS*    00000000
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crti.o
    00000000 l    df *ABS*    00000000
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crtn.o

 From temp/log.do_compile:

    arm-tait-linux-gnueabi-gcc
         -march=armv5e -marm
         ...
    -B/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/
         ...
         -o
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/math/libm.so
         ...
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
         ...

This link command ignores -fdebug-prefix-map even if we add it (ld 
doesn't accept it). The fully-qualified paths to crti.o and crtn.o come 
from the -B option. If we use relative paths to these files and to 
abi-note.o, the problem goes away:

    $ cd git/math/
    $ arm-tait-linux-gnueabi-gcc \
         -march=armv5e -marm \
         ...
         -B../../build-arm-tait-linux-gnueabi/csu/ \
         ...
         -o
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/math/libm.so
    \
         ...
    ../../build-arm-tait-linux-gnueabi/csu/abi-note.o \
         ...

    $ cd ../../build-arm-tait-linux-gnueabi/
    $
    ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump
    -t math/libm.so | grep csu
    00000000 l    df *ABS*    00000000
    ../../build-arm-tait-linux-gnueabi/csu/abi-note.o
    00000000 l    df *ABS*    00000000
    ../../build-arm-tait-linux-gnueabi/csu/crti.o
    00000000 l    df *ABS*    00000000
    ../../build-arm-tait-linux-gnueabi/csu/crtn.o

Make is invoked from the build-arm-tait-linux-gnueabi/ directory, whose 
Makefile recursively invokes the Makefile in ../git/, passing in a 
fully-qualified "objdir":

    all .DEFAULT:
             $(MAKE) -r PARALLELMFLAGS="$(PARALLELMFLAGS)" -C $(srcdir)
    *objdir=`pwd`* $@

Unfortunately, the makefiles are written on the assumption that the 
objdir is fully-qualified, so simply hacking this line doesn't work.

The -B path comes from git/Makerules +612, and the fully-qualified path 
to abi-note.o from line +665, in both cases via the csu-objpfx variable.

    $(extra-B-$(@F:lib%.so=%).so) -B$(csu-objpfx) \
    ...

    $(csu-objpfx)abi-note.o $(build-shlib-objlist)

I added "realpath" to the HOSTTOOLS, and precomputed a relative path to 
the csu/ dir in git/Makerules.

    csu-objpfx-relative = $(shell realpath --relative-to=`pwd`
    $(csu-objpfx))/

Seems to do the trick. I'll send a patch(ish) separately.

[-- Attachment #2: Type: text/html, Size: 9194 bytes --]

Re: glibc binary reproducibility

[Douglas Royds]

[-- Attachment #1: Type: text/plain, Size: 5095 bytes --]

On 03/10/18 15:37, Douglas Royds wrote:

> I added "realpath" to the HOSTTOOLS, and precomputed a relative path 
> to the csu/ dir in git/Makerules.
>
>     csu-objpfx-relative = $(shell realpath --relative-to=`pwd`
>     $(csu-objpfx))/
>
> Seems to do the trick. I'll send a patch(ish) separately. 


Richard, this patch (perhaps) applies on top of your poky-contrib 
branch. I've used `realpath` instead of your python oe-relpaths script, 
just because it was there. Requires coreutils 8.23 or later, I believe 
(Ubuntu 16.04 or later).


Subject: [PATCH] glibc: Fix non-IA reproducibility for shared libraries

Compute a relative path to build-arm-tait-linux-gnueabi/csu/
for each library subdirectory.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
  meta/conf/bitbake.conf                        |  2 +-
  ...34-reproducible-makehack-shared-libs.patch | 48 +++++++++++++++++++
  meta/recipes-core/glibc/glibc_2.28.bb         |  1 +
  3 files changed, 50 insertions(+), 1 deletion(-)
  create mode 100644 
meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 1941633a54..aa759b19a3 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -482,7 +482,7 @@ HOSTTOOLS += " \
      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip 
gzip \
      head hostname id install ld ldd ln ls make makeinfo md5sum mkdir 
mknod \
      mktemp mv nm objcopy objdump od patch perl pod2man pr printf pwd 
python python2 \
-    python2.7 python3 ranlib readelf readlink rm rmdir rpcgen sed sh 
sha256sum \
+    python2.7 python3 ranlib readelf readlink realpath rm rmdir rpcgen 
sed sh sha256sum \
      sleep sort split stat strings strip tail tar tee test touch tr 
true uname \
      uniq wc wget which xargs \
  "
diff --git 
a/meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch 
b/meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch
new file mode 100644
index 0000000000..11902be523
--- /dev/null
+++ 
b/meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch
@@ -0,0 +1,48 @@
+diff --git a/Makerules b/Makerules
+index a10a0b4d70..cacddb8784 100644
+--- a/Makerules
++++ b/Makerules
+@@ -606,10 +606,11 @@ lib%.so: lib%_pic.a $(+preinit) $(+postinit) 
$(link-libc-deps)
+     $(build-shlib) $(link-libc-args)
+     $(call after-link,$@)
+
++csu-objpfx-relative = $(shell realpath --relative-to=`pwd` $(csu-objpfx))/
+ define build-shlib-helper
+ $(LINK.o) -shared -static-libgcc -Wl,-O1 $(sysdep-LDFLAGS) \
+       $(if $($(@F)-no-z-defs)$(no-z-defs),,-Wl,-z,defs) $(rtld-LDFLAGS) \
+-      $(extra-B-$(@F:lib%.so=%).so) -B$(csu-objpfx) \
++      $(extra-B-$(@F:lib%.so=%).so) -B$(csu-objpfx-relative) \
+       $(extra-B-$(@F:lib%.so=%).so) $(load-map-file) \
+ -Wl,-soname=lib$(libprefix)$(@F:lib%.so=%).so$($(@F)-version) \
+       $(LDFLAGS.so) $(LDFLAGS-lib.so) $(LDFLAGS-$(@F:lib%.so=%).so) \
+@@ -662,13 +663,13 @@ endif
+
+ define build-shlib
+ $(build-shlib-helper) -o $@ $(shlib-lds-flags) \
+-      $(csu-objpfx)abi-note.o $(build-shlib-objlist)
++      $(csu-objpfx-relative)abi-note.o $(build-shlib-objlist)
+ endef
+
+ define build-module-helper
+ $(LINK.o) -shared -static-libgcc $(sysdep-LDFLAGS) $(rtld-LDFLAGS) \
+       $(if $($(@F)-no-z-defs)$(no-z-defs),,-Wl,-z,defs) \
+-      -B$(csu-objpfx) $(load-map-file) \
++      -B$(csu-objpfx-relative) $(load-map-file) \
+       $(LDFLAGS.so) $(LDFLAGS-$(@F:%.so=%).so) \
+       -L$(subst :, -L,$(rpath-link)) -Wl,-rpath-link=$(rpath-link)
+ endef
+@@ -679,12 +680,12 @@ endef
+ # not for shared objects
+ define build-module
+ $(build-module-helper) -o $@ $(shlib-lds-flags) \
+-      $(csu-objpfx)abi-note.o $(build-module-objlist) $(link-libc-args)
++      $(csu-objpfx-relative)abi-note.o $(build-module-objlist) 
$(link-libc-args)
+ $(call after-link,$@)
+ endef
+ define build-module-asneeded
+ $(build-module-helper) -o $@ $(shlib-lds-flags) \
+-      $(csu-objpfx)abi-note.o \
++      $(csu-objpfx-relative)abi-note.o \
+       -Wl,--as-needed $(build-module-objlist) -Wl,--no-as-needed \
+       $(link-libc-args)
+ $(call after-link,$@)
diff --git a/meta/recipes-core/glibc/glibc_2.28.bb 
b/meta/recipes-core/glibc/glibc_2.28.bb
index 6ca89cbe2c..9d1d46bf28 100644
--- a/meta/recipes-core/glibc/glibc_2.28.bb
+++ b/meta/recipes-core/glibc/glibc_2.28.bb
@@ -46,6 +46,7 @@ SRC_URI = 
"${GLIBC_GIT_URI};branch=${SRCBRANCH};name=glibc \
file://0029-localedef-add-to-archive-uses-a-hard-coded-locale-pa.patch \
file://0030-intl-Emit-no-lines-in-bison-generated-files.patch \
             file://0033-reproducible-makehack.patch \
+           file://0034-reproducible-makehack-shared-libs.patch \
  "

  NATIVESDKFIXES ?= ""
-- 
2.17.1



[-- Attachment #2: Type: text/html, Size: 7194 bytes --]

[PATCH] glibc: Fix non-IA reproducibility for shared libraries

[Douglas Royds]

Compute a relative path to build-arm-tait-linux-gnueabi/csu/
for each library subdirectory.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/conf/bitbake.conf                        |  2 +-
 ...34-reproducible-makehack-shared-libs.patch | 48 +++++++++++++++++++
 meta/recipes-core/glibc/glibc_2.28.bb         |  1 +
 3 files changed, 50 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 93aee1ae46..6c0b3c239f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -482,7 +482,7 @@ HOSTTOOLS += " \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
     head hostname id install ld ldd ln ls make makeinfo md5sum mkdir mknod \
     mktemp mv nm objcopy objdump od patch perl pod2man pr printf pwd python python2 \
-    python2.7 python3 ranlib readelf readlink rm rmdir rpcgen sed sh sha256sum \
+    python2.7 python3 ranlib readelf readlink realpath rm rmdir rpcgen sed sh sha256sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
     uniq wc wget which xargs \
 "
diff --git a/meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch b/meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch
new file mode 100644
index 0000000000..11902be523
--- /dev/null
+++ b/meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch
@@ -0,0 +1,48 @@
+diff --git a/Makerules b/Makerules
+index a10a0b4d70..cacddb8784 100644
+--- a/Makerules
++++ b/Makerules
+@@ -606,10 +606,11 @@ lib%.so: lib%_pic.a $(+preinit) $(+postinit) $(link-libc-deps)
+ 	$(build-shlib) $(link-libc-args)
+ 	$(call after-link,$@)
+ 
++csu-objpfx-relative = $(shell realpath --relative-to=`pwd` $(csu-objpfx))/
+ define build-shlib-helper
+ $(LINK.o) -shared -static-libgcc -Wl,-O1 $(sysdep-LDFLAGS) \
+ 	  $(if $($(@F)-no-z-defs)$(no-z-defs),,-Wl,-z,defs) $(rtld-LDFLAGS) \
+-	  $(extra-B-$(@F:lib%.so=%).so) -B$(csu-objpfx) \
++	  $(extra-B-$(@F:lib%.so=%).so) -B$(csu-objpfx-relative) \
+ 	  $(extra-B-$(@F:lib%.so=%).so) $(load-map-file) \
+ 	  -Wl,-soname=lib$(libprefix)$(@F:lib%.so=%).so$($(@F)-version) \
+ 	  $(LDFLAGS.so) $(LDFLAGS-lib.so) $(LDFLAGS-$(@F:lib%.so=%).so) \
+@@ -662,13 +663,13 @@ endif
+ 
+ define build-shlib
+ $(build-shlib-helper) -o $@ $(shlib-lds-flags) \
+-	  $(csu-objpfx)abi-note.o $(build-shlib-objlist)
++	  $(csu-objpfx-relative)abi-note.o $(build-shlib-objlist)
+ endef
+ 
+ define build-module-helper
+ $(LINK.o) -shared -static-libgcc $(sysdep-LDFLAGS) $(rtld-LDFLAGS) \
+ 	  $(if $($(@F)-no-z-defs)$(no-z-defs),,-Wl,-z,defs) \
+-	  -B$(csu-objpfx) $(load-map-file) \
++	  -B$(csu-objpfx-relative) $(load-map-file) \
+ 	  $(LDFLAGS.so) $(LDFLAGS-$(@F:%.so=%).so) \
+ 	  -L$(subst :, -L,$(rpath-link)) -Wl,-rpath-link=$(rpath-link)
+ endef
+@@ -679,12 +680,12 @@ endef
+ # not for shared objects
+ define build-module
+ $(build-module-helper) -o $@ $(shlib-lds-flags) \
+-	  $(csu-objpfx)abi-note.o $(build-module-objlist) $(link-libc-args)
++	  $(csu-objpfx-relative)abi-note.o $(build-module-objlist) $(link-libc-args)
+ $(call after-link,$@)
+ endef
+ define build-module-asneeded
+ $(build-module-helper) -o $@ $(shlib-lds-flags) \
+-	  $(csu-objpfx)abi-note.o \
++	  $(csu-objpfx-relative)abi-note.o \
+ 	  -Wl,--as-needed $(build-module-objlist) -Wl,--no-as-needed \
+ 	  $(link-libc-args)
+ $(call after-link,$@)
diff --git a/meta/recipes-core/glibc/glibc_2.28.bb b/meta/recipes-core/glibc/glibc_2.28.bb
index 804b7891ec..5508be13f7 100644
--- a/meta/recipes-core/glibc/glibc_2.28.bb
+++ b/meta/recipes-core/glibc/glibc_2.28.bb
@@ -48,6 +48,7 @@ SRC_URI = "${GLIBC_GIT_URI};branch=${SRCBRANCH};name=glibc \
            file://0031-sysdeps-ieee754-prevent-maybe-uninitialized-errors.patch \
            file://0032-soft-fp-ignore-maybe-uninitialized.patch \
            file://0001-Linux-gethostid-Check-for-NULL-value-from-gethostbyn.patch \
+           file://0034-reproducible-makehack-shared-libs.patch \
 "
 
 NATIVESDKFIXES ?= ""
-- 
2.17.1

✗ patchtest: failure for glibc: Fix non-IA reproducibility for shared libraries

[Patchwork]

== Series Details ==

Series: glibc: Fix non-IA reproducibility for shared libraries
Revision: 1
URL   : https://patchwork.openembedded.org/series/14338/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             A patch file has been added, but does not have a Signed-off-by tag [test_signed_off_by_presence] 
  Suggested fix    Sign off the added patch file (meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch)

* Issue             Added patch file is missing Upstream-Status in the header [test_upstream_status_presence_format] 
  Suggested fix    Add Upstream-Status: <Valid status> to the header of meta/recipes-core/glibc/glibc/0034-reproducible-makehack-shared-libs.patch
  Standard format  Upstream-Status: <Valid status>
  Valid status     Pending, Accepted, Backport, Denied, Inappropriate [reason], Submitted [where]



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: glibc binary reproducibility

[Richard Purdie]

I don't think your patch will fix all the cases sadly. I did find
another technique that works:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=21bc0c2fe6e9d2ae1d08d3de5f2a6d75d4ba108b

basically by injecting a STT_FILE entry into the underlying assembler
files, the linker then doesn't need to add its own.

The patch above fixes aarch64 but will need additions for the other
arches which break...

Khem: Any thoughts on this?

Cheers,

Richard

Re: glibc binary reproducibility

[Khem Raj]

On Wed, Oct 3, 2018 at 4:25 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> I don't think your patch will fix all the cases sadly. I did find
> another technique that works:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=21bc0c2fe6e9d2ae1d08d3de5f2a6d75d4ba108b
>
> basically by injecting a STT_FILE entry into the underlying assembler
> files, the linker then doesn't need to add its own.
>
> The patch above fixes aarch64 but will need additions for the other
> arches which break...
>
> Khem: Any thoughts on this?

this is right fix

>
> Cheers,
>
> Richard

Re: glibc binary reproducibility

[Douglas Royds]

I had found and updated my patch for the ld.so and libpthread.so cases 
as well, but you beat me to it ... Much better solution, thanks.


On 04/10/18 00:25, Richard Purdie wrote:

> I don't think your patch will fix all the cases sadly. I did find
> another technique that works:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=21bc0c2fe6e9d2ae1d08d3de5f2a6d75d4ba108b
>
> basically by injecting a STT_FILE entry into the underlying assembler
> files, the linker then doesn't need to add its own.
>
> The patch above fixes aarch64 but will need additions for the other
> arches which break...
>
> Khem: Any thoughts on this?
>
> Cheers,
>
> Richard