Re: [OE-core] [master][PATCH 0/4] Reproducible LTO builds

["Khem Raj" <raj.khem@gmail.com>]

I am seeing bunch of failures on meta-oe and meta-atmel

https://errors.yoctoproject.org/Errors/Details/600064/
https://errors.yoctoproject.org/Errors/Details/600065/
https://errors.yoctoproject.org/Errors/Details/600071/

On Fri, Aug 6, 2021 at 2:04 PM Tony Battersby <tonyb@cybernetics.com> wrote:
>
> On 8/6/21 12:44 PM, Tony Battersby wrote:
> > On 8/6/21 1:33 AM, Richard Purdie wrote:
> >> On Thu, 2021-08-05 at 17:54 -0400, Tony Battersby wrote:
> >>> On 8/5/21 6:21 AM, Richard Purdie wrote:
> >>>> On Wed, 2021-08-04 at 20:13 -0400, Randy MacLeod wrote:
> >>>>> On 2021-07-27 9:34 a.m., Tony Battersby wrote:
> >>>>>> Currently, enabling Link Time Optimization via:
> >>>>>>
> >>>>>> require conf/distro/include/lto.inc
> >>>>>> DISTRO_FEATURES_append = " lto"
> >>>>>>
> >>>>>> makes the build non-reproducible (as in
> >>>>>> https://reproducible-builds.org/).  This patch set fixes some of the
> >>>>>> problems, but is not yet a complete solution.
> >>>>>>
> >>>>>> This patch set addresses the following bugs:
> >>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14481
> >>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14490
> >>>>>>
> >>>>>> openembedded-core patches
> >>>>>> 1/4: lto.inc: disable LTO for grub
> >>>>>> 2/4: gcc: Backport patch to make LTO builds more reproducible
> >>>>>> 3/4: bitbake.conf: add DEBUG_PREFIX_MAP to TARGET_LDFLAGS
> >>>>>>
> >>>>>> openembedded-devel patches
> >>>>>> 4/4: net-snmp: fix QA Issue after LDFLAGS change
> >>>>>>
> >>>>>> Patch 3 adds DEBUG_PREFIX_MAP to TARGET_LDFLAGS (it was already in
> >>>>>> TARGET_CFLAGS).  This could possibly be done differently, such as by
> >>>>>> removing it from TARGET_CFLAGS and adding it to TOOLCHAIN_OPTIONS,
> >>>>>> TARGET_CC_ARCH, or TUNE_CCARGS, but I haven't tested those.  Let me know
> >>>>>> if there is a better/preferred way to do it.
> >>>>>>
> >>>>>> The TARGET_LDFLAGS change broke net-snmp from meta-networking, so I have
> >>>>>> included a patch to fix that.  Other packages might have been broken as
> >>>>>> well, although I haven't found any others.
> >>>>>>
> >>>>>> With all these patches, there are still many shared libraries in /lib
> >>>>>> and /usr/lib that are still not reproducible with LTO enabled because
> >>>>>> x86_64-poky-linux-libtool filters out the -f*-prefix-map flags added by
> >>>>>> patch 3 when linking shared libraries.  /usr/lib/libbz2.so.1.0.6 from
> >>>>>> the libbz2 package from the bzip2 recipe is a good example.  I do not
> >>>>>> have a patch for that problem.  I will leave that to someone else.
> >>>>>>
> >>>>>> So this is what we need to achieve reproducible LTO builds:
> >>>>>> *) This patch set
> >>>>>> *) A fix for libtool
> >>>>>> *) Add LTO to reproducible builds testing if appropriate (e.g.
> >>>>>> meta/lib/oeqa/selftest/cases/reproducible.py, CI on Autobuilder)
> >>>>>> *) Fixes for individual packages, if needed
> >>>>>> *) Once everything works, update the documentation again (I previously
> >>>>>> requested a documentation change for this bug, which will eventually
> >>>>>> show up at
> >>>>>> http://docs.yoctoproject.org/test-manual/reproducible-builds.html).
> >>>>> This is certainly a few steps in the right direction with a good outline
> >>>>> of what's next.
> >>>>>
> >>>>> I suspect that Richard has been busy with other things like overrides
> >>>>> and the update to glibc-2.34.
> >>>>>
> >>>>> Do you have any suggestions or questions about what we should do
> >>>>> with this patch series Richard?
> >>>> Thanks for the reminder, I've been meaning to queue/test this which I will
> >>>> do. The gcc version has been bumped but I ported the patch forward to it
> >>>> and the patches are in master-next for testing. It is great to see this
> >>>> support moving forward.
> >>>>
> >>>> Much as it pains me to admit having any libtool knowledge, I may be able to
> >>>> help with that bit. There is an untested patch below which may address that
> >>>> issue, or at least give a good hint on how to do it if it doesn't work. I
> >>>> don't have an environment to test easily right now.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Richard
> >>> I tested your libtool patch in master-next with the bzip2 recipe.  It
> >>> works as intended; libbz2.so.1.0.6 and all the other output files are
> >>> now reproducible with LTO enabled.  I also verified that libtool is
> >>> passing the -f*-prefix-map flags to the link command in log.do_compile
> >>> as intended.
> >>>
> >>> I also verified that core-image-minimal builds successfully with the
> >>> libtool patch.
> >> Thanks for testing that, sounds like it helps that issue. I'll queue that patch.
> >>
> >> Unfortunately even with the ruby fix I queued, we still see reproduciblity
> >> issues with the LDFLAGS change in curl-dev, gettext-ptest, ruby and ruby-dbg
> >> packages:
> >>
> >> https://autobuilder.yoctoproject.org/typhoon/#/builders/115/builds/515/steps/12/logs/stdio
> >>
> >> We'll need to resolve those before I can merge the flags change.
> >>
> >> Diffoscope output:
> >>
> >> https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210805-8jfd55ki/packages/diff-html/
> >>
> >> Basically LDFLAGS get encoded into a couple of scripts and the ruby binary.
> >>
> >> Cheers,
> >>
> >> Richard
> >>
> > It looks like the ruby problem is being caused by MJIT_DLDFLAGS in
> > build/tmp/work/core2-64-poky-linux/ruby/3.0.2-r0/build/mjit_config.h.
> > There is already a patch
> > poky/meta/recipes-devtools/ruby/ruby/0001-template-Makefile.in-do-not-write-host-cross-cc-item.patch
> > that removes some of these items for the sake of reproducibiltiy.  But I
> > do not know ruby, so I do not know the implications of doing the same
> > thing for MJIT_DLDFLAGS.
> >
> > Tony Battersby
> >
> >
> >
> Richard,
>
> I noticed that you already fixed the problems in curl and gettext in
> master-next.  Here is a patch for ruby if you haven't fixed that already.
>
> Tony Battersby
>
> From 9cfb9777ff6ef95a27ed6a03719c807365d0f116 Mon Sep 17 00:00:00 2001
> From: Tony Battersby <tonyb@cybernetics.com>
> Date: Fri, 6 Aug 2021 16:47:14 -0400
> Subject: [PATCH] ruby: Fix reproducibility issue with LDFLAGS
>
> If we add DEBUG_PREFIX_MAP into LDFLAGS, ruby and ruby-dbg are no
> longer reproducible.  Fix this.
>
> Signed-off-by: Tony Battersby <tonyb@cybernetics.com>
> ---
>  ...-Makefile.in-filter-out-f-prefix-map.patch | 42 +++++++++++++++++++
>  meta/recipes-devtools/ruby/ruby_3.0.2.bb      |  1 +
>  2 files changed, 43 insertions(+)
>  create mode 100644 meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch
>
> diff --git a/meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch b/meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch
> new file mode 100644
> index 0000000000..9387506c26
> --- /dev/null
> +++ b/meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch
> @@ -0,0 +1,42 @@
> +Subject: [PATCH] template/Makefile.in: filter out -f*prefix-map
> +
> +If we add DEBUG_PREFIX_MAP into LDFLAGS, ruby and ruby-dbg are no longer
> +reproducible.  Fix this.
> +
> +Upstream-Status: Inapproppriate [oe-core specific]
> +Signed-off-by: Tony Battersby <tonyb@cybernetics.com>
> +---
> +--- a/tool/mjit_archflag.sh
> ++++ b/tool/mjit_archflag.sh
> +@@ -7,6 +7,20 @@ quote() {
> +     echo
> + }
> +
> ++quote_filtered() {
> ++    printf "#${indent}define $1"
> ++    while shift && [ "$#" -gt 0 ]; do
> ++      case "$1" in
> ++          -ffile-prefix-map=*|-fdebug-prefix-map=*|-fmacro-prefix-map=*)
> ++              ;;
> ++          *)
> ++              printf ' "%s"'$sep "$1"
> ++              ;;
> ++      esac
> ++    done
> ++    echo
> ++}
> ++
> + archs=""
> + arch_flag=""
> +
> +--- a/template/Makefile.in
> ++++ b/template/Makefile.in
> +@@ -666,7 +666,7 @@ mjit_config.h:
> +       quote "MJIT_OPTFLAGS   " $(MJIT_OPTFLAGS); \
> +       quote "MJIT_DEBUGFLAGS " $(MJIT_DEBUGFLAGS); \
> +       quote "MJIT_LDSHARED   " ; \
> +-      quote "MJIT_DLDFLAGS    MJIT_ARCHFLAG" $(MJIT_DLDFLAGS); \
> ++      quote_filtered "MJIT_DLDFLAGS    MJIT_ARCHFLAG" $(MJIT_DLDFLAGS); \
> +       quote "MJIT_LIBS       " $(LIBRUBYARG_SHARED); \
> +       quote 'PRELOADENV       "@PRELOADENV@"'; \
> +       indent=$${archs:+'  '}; \
> diff --git a/meta/recipes-devtools/ruby/ruby_3.0.2.bb b/meta/recipes-devtools/ruby/ruby_3.0.2.bb
> index 0b9c342def..38e594a59e 100644
> --- a/meta/recipes-devtools/ruby/ruby_3.0.2.bb
> +++ b/meta/recipes-devtools/ruby/ruby_3.0.2.bb
> @@ -6,6 +6,7 @@ SRC_URI += " \
>             file://remove_has_include_macros.patch \
>             file://run-ptest \
>             file://0001-template-Makefile.in-do-not-write-host-cross-cc-item.patch \
> +           file://0002-template-Makefile.in-filter-out-f-prefix-map.patch \
>             "
>
>  SRC_URI[sha256sum] = "5085dee0ad9f06996a8acec7ebea4a8735e6fac22f22e2d98c3f2bc3bef7e6f1"
> --
> 2.25.1
>