Re: [OE-core] [PATCH 0/5] linux-yocto: consolidated pull request

["Bruce Ashfield" <bruce.ashfield@gmail.com>]

On Mon, May 4, 2020 at 9:24 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2020-05-04 at 09:06 -0400, Bruce Ashfield wrote:
> > On Mon, May 4, 2020 at 8:56 AM Bruce Ashfield via
> > lists.openembedded.org
> > <bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
> > > On Mon, May 4, 2020 at 4:31 AM Richard Purdie
> > > <richard.purdie@linuxfoundation.org> wrote:
> > > > Hi Bruce,
> > > >
> > > > On Sun, 2020-05-03 at 11:44 -0400, bruce.ashfield@gmail.com wrote:
> > > > > From: Bruce Ashfield <bruce.ashfield@gmail.com>
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Here are the -stable updates I've collected since m3 of dunfell. I
> > > > > ran things through the autobuilder, and no new kernel issues were
> > > > > picked up.
> > > > >
> > > > > The -dev bump is good for master, while all of the 5.4-stable bumps
> > > > > are good for both master and dunfell.
> > > >
> > > > I pulled these into master-next and saw reproducibility issues. I
> > > > wondered why your test build didn't show it, turns out it also did:
> > > >
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1024
> > > >
> > > > 2020-05-04 05:18:46,863 - oe-selftest - INFO - ======================================================================
> > > > 2020-05-04 05:18:46,863 - oe-selftest - INFO - FAIL: reproducible.ReproducibleTests.test_reproducible_builds (subunit.RemotedTestCase)
> > > > 2020-05-04 05:18:46,863 - oe-selftest - INFO - ----------------------------------------------------------------------
> > > > 2020-05-04 05:18:46,864 - oe-selftest - INFO - testtools.testresult.real._StringException:
> > > >
> > > > AssertionError: The following deb packages are missing or different: /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-36044/reproducibleB/tmp/deploy/deb/./qemux86_64/kernel-module-kheaders-5.4.38-yocto-standard_5.4.38+git0+f405543442_6cb5b11e83-r0_amd64.deb
> > > > The following ipk packages are missing or different: /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-36044/reproducibleB/tmp/deploy/ipk/./qemux86_64/kernel-module-kheaders-5.4.38-yocto-standard_5.4.38+git0+f405543442_6cb5b11e83-r0_qemux86_64.ipk
> > > >
> > >
> > > I don't see any changes in the kernels that would have triggered it,
> > > but they never are obvious.
> >
> > I am working on some kernel reproducibility changes that were
> > discussed about three weeks ago.
> >
> > Is the default for reproducible builds in master-next different from
> > what it was before dunfell was releases ? If so, that might explain
> > why I'm not seeing a change that would have triggered it (i.e. this is
> > unrelated to my -stable changes).
>
> There are no reproducibility changes between master-next and dunfell.

interesting. I'll take a closer look at the changes again and see what
might pop up as a culprit.

>
> > Like I was mentioning before, if there's a document or pastebin of
> > steps I can follow, I can run the tests against my pending changes and
> > finish them for master. They of course would not necessarily be
> > suitable for dunfell.
>
> The good news is the autobuilder has some debugging built in for this
> kind of problem. You can see the diffoscope output here:
>
> https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200504-vhr0o_t9/packages/diff-html/
>
> which tells us its kheaders.ko in .rodata by the looks of it. Some
> changes are cause, some are effect, e.g. the build ID is different as
> the sections are different so that isn't the issue but an effect of it.
> It looks like diffoscope doesn't know how to decode the .rodata which
> could be compressed?
>
> The raw data is also there (the packages that differ), e.g.:
>
> https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200504-vhr0o_t9/packages/reproducibleA/tmp/deploy/deb/qemux86_64/kernel-module-kheaders-5.4.38-yocto-standard_5.4.38%2Bgit0%2Bf405543442_6cb5b11e83-r0_amd64.deb
>
> https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20200504-vhr0o_t9/packages/reproducibleB/tmp/deploy/deb/qemux86_64/kernel-module-kheaders-5.4.38-yocto-standard_5.4.38%2Bgit0%2Bf405543442_6cb5b11e83-r0_amd64.debe
>

cool. That's definitely useful, I'll have a look.

At the same time, I'll try and clean up the documented kernel
reproducible changes and trigger another run to see if they help.

> which would allow you to look further at them by hand.
>
> You can manually run this test locally with:
>
> oe-selftest -r reproducible.ReproducibleTests.test_reproducible_builds
>
> but it will take a while to run as it has to run two builds and compare
> them and only uses sstate for one of them. To make it faster since you
> know where the issue is, edit
> meta/lib/oeqa/selftest/cases/reproducible.py to change images (core-
> image-sato, core-image-minimal and friends to simply "virtual/kernel".

I'll do that cut down of the tests and see if I can trigger it here ..
With those full builds, I'd be at about an 8 hour cycle to see the
results!

Bruce

>
> Cheers,
>
> Richard
>
>
>
>
>
>
>
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II