On 03/10/18 05:58, Richard Purdie wrote: 

On Thu, 2018-09-27 at 12:18 +1200, Douglas Royds wrote:

When I build glibc in two different places, I get non-reproducible
results - a 4-byte difference:

$ cmp -l ~/workspace/upstream[12]/build/tmp/work/armv5e-tait-linux-
gnueabi/glibc/2.28-r0/package/lib/libc-2.28.so
1259181  71 172
1259182  27 304
1259183 152  77
1259184 363 243

These 4 bytes are the checksum that objcopy --add-gnu-debuglink adds
to the binary when the .debug info is split out at do_package time,
see package.bbclass +416
So why is the debug info different? We add -fdebug-prefix-map to our
CFLAGS, which ensures that all our intra-component debug paths are
prefixed with /usr/src/debug/glibc/, for instance, but this isn't
working in this case. The difference is happening in csu/crt1.o,
which is being linked into libc.so (and others):

$ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
linux-gnueabi-objdump -t csu/crt1.o
...
00000000 l    df *ABS*    00000000
/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
00000000 l    df *ABS*    00000000
/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o

 This abi-note.o symbol is finding its way into libc.so: 

$ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
linux-gnueabi-objdump -t libc.so
...
00000000 l    df *ABS*    00000000             
/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o


No it isn't, it turns out. The non-reproducibility of crt1.o and friends is a problem, as they get copied into the recipe-sysroot of all other components, causing the 780ish different packages observed by Juro in https://bugzilla.yoctoproject.org/show_bug.cgi?id=12743


There is a work-around: turn off the debug packaging:

INHIBIT_PACKAGE_DEBUG_SPLIT_pn-glibc = "1"

I don't have a solution for this. Suggestions?

I did some digging.

I tried what I suggested using relative paths:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=22189be4bf508851950f72654870c4eebd4b73d9

and whilst it helps remove some references, there is a much wider
problem. I therefore went and had a look at the linker itself to
understand why its injecting this path. I found this code:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=f9aca51204990fbdbdfa7442f1dcc938e97bf782

which if disabled as per this hack, makes the binaries reproducible and
drops the problematic references.

We're swapping some misleading debug output for reproducibility with
that hack.

At this point we probably need to talk to some toolchain people about
what 'real' fixes may be possible...


Excellent. Your patch in 22189be is much tidier than my own Makefile hack that I was too embarassed to publish on this list (it may or may not have involved sedding a binary file).

It turns out that we have two binary difference issues: The crt1.o (and friends) one discussed above, and a similar problem linking all the shared libs, eg. libm.so
$ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump -t math/libm.so | grep armv5e
00000000 l    df *ABS*    00000000              /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
00000000 l    df *ABS*    00000000              /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crti.o
00000000 l    df *ABS*    00000000              /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crtn.o

From temp/log.do_compile:

arm-tait-linux-gnueabi-gcc
    -march=armv5e -marm
    ...
    -B/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/
    ...
    -o /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/math/libm.so
    ...
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
    ...

This link command ignores -fdebug-prefix-map even if we add it (ld doesn't accept it). The fully-qualified paths to crti.o and crtn.o come from the -B option. If we use relative paths to these files and to abi-note.o, the problem goes away:
$ cd git/math/
$ arm-tait-linux-gnueabi-gcc \
    -march=armv5e -marm \
    ...
    -B../../build-arm-tait-linux-gnueabi/csu/ \
    ...
    -o /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/math/libm.so \
    ...
    ../../build-arm-tait-linux-gnueabi/csu/abi-note.o \
    ...

$ cd ../../build-arm-tait-linux-gnueabi/
$ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump -t math/libm.so | grep csu
00000000 l    df *ABS*    00000000              ../../build-arm-tait-linux-gnueabi/csu/abi-note.o
00000000 l    df *ABS*    00000000              ../../build-arm-tait-linux-gnueabi/csu/crti.o
00000000 l    df *ABS*    00000000              ../../build-arm-tait-linux-gnueabi/csu/crtn.o
Make is invoked from the build-arm-tait-linux-gnueabi/ directory, whose Makefile recursively invokes the Makefile in ../git/, passing in a fully-qualified "objdir":
all .DEFAULT:
        $(MAKE) -r PARALLELMFLAGS="$(PARALLELMFLAGS)" -C $(srcdir) objdir=`pwd` $@
Unfortunately, the makefiles are written on the assumption that the objdir is fully-qualified, so simply hacking this line doesn't work.

The -B path comes from git/Makerules +612, and the fully-qualified path to abi-note.o from line +665, in both cases via the csu-objpfx variable.
$(extra-B-$(@F:lib%.so=%).so) -B$(csu-objpfx) \
...

$(csu-objpfx)abi-note.o $(build-shlib-objlist)
I added "realpath" to the HOSTTOOLS, and precomputed a relative path to the csu/ dir in git/Makerules.
csu-objpfx-relative = $(shell realpath --relative-to=`pwd` $(csu-objpfx))/
Seems to do the trick. I'll send a patch(ish) separately.