Re: [PATCH 0/4] oeqa: Run diffoscop on saved output

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Sun, 2020-02-16 at 12:27 -0600, Joshua Watt wrote:
> On Sat, Feb 15, 2020, 5:07 AM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > On Tue, 2020-02-11 at 21:14 -0600, Joshua Watt wrote:
> > > Adds recipes to build the diffoscope tool and run it when the OEQA
> > > reproducible build test saves output to a local path. This should
> > > make
> > > it much easier to debug reproducible build issues from the
> > > autobuilder,
> > > since the published output can be easily viewed on the website.
> > > 
> > > Joshua Watt (4):
> > >   python: Add libarchive-c recipe
> > >   python: Add magic recipe
> > >   recipes-support: Add diffoscope recipe
> > >   oeqa: reproducible: Run diffoscope on saved output
> > 
> > Thanks!
> > 
> > The first production use:
> > 
> > https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200215-t1s21q9r/packages/diff-html/
> > 
> > :)
> 
> That's great!
> 
> > I am a bit puzzled/concerned about how this hasn't been removed
> > from
> > the system yet as it should have been, unless its the hashequiv
> > problem
> > with timestamps continuing to cause problems. Need to fix my
> > patch...
> 
> It looks like a directory ordering issue to me, since the timestamps
> are the same. One way to really flush these out would be to use
> disorderfs (https://salsa.debian.org/reproducible-builds/disorderfs)
> which permutates the order in which entries are listed in a
> directory, but I think we should try to get "wider" coverage (more
> recipes) before we try that.

Its more complicated unfortunately. We should have fixed the directory
order issue with:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=09c6a41b751d080f0c12fc4172a31d1dbe760b0b

However issues which shows signs of that bug still persisted in the
reproducibility test results.

I theorised that since opkg-utils is in ABISAFE, the change wasn't
accounted for. I therefore merged:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=5fd3eb4bda651bddf4e4dc76d456333c70df4ddc

However with current master, we're seeing the above reproducible build
failure. I checked in a build and its using this sstate object for
comparison:

pub/sstate/b0/ca/sstate\:matchbox-config-gtk\:core2-64-poky-linux\:0.2\:r0\:core2-64\:3\:b0ca3b3559d37ba8de8ade8b282baa5082e612ef6590d3ff94fcebe38e9f17b9_package_write_ipk.tgz.siginfo

which is interesting in that its from 2nd Feb (predating both commits
above) and doesn't show the above change to package_ipk.bbclass.

The change to package_ipk means the task's own basehash has to have
changed which means it *has* to have rerun. Somehow hashequiv has then
marked it as equivalent.

I guess the issue may be that in some builds, this did generate output
that was equivalent and things have then raced so this "broken" sstate
object has persisted.

The question then becomes, how to we get this out the cache?! I'm not
entirely sure how we can :(

(obviously I can cleansstate a few things which I'll do now but its not
good this can't recover even through metadata changes)

Cheers,

Richard