Hash Equivalency - What this means for developer productivity

Hash Equivalency - What this means for developer productivity

[Richard Purdie]

This has been talked about in a lot of the project meetings but I
wanted to ensure its on the mailing list too.

master-next (and hopefully soon, master) now contain patches which
enable "hash equivalency" support in runqueue in bitbake. That doesn't
sound so interesting until I explain what this means.

Currently, if I do:

$ bitbake core-image-sato

then:

$ bitbake m4-native -c install -f

this would mean that core-image-sato would rebuild, nearly from
scratch. The task stamp for all tasks onward from m4-native would
change.

With the patches in master-next and this configuration in local.conf:

BB_HASHSERVE = "localhost:0"
BB_SIGNATURE_HANDLER = "OEEquivHash"

$ bitbake core-image-sato
$ bitbake m4-native -c install -f
$ bitbake core-image-sato

will result in do_populate_sysroot of m4-native running, it will see
the output matches the previous build and it will then skip to the
rootfs generation pulling all the other pieces from sstate.

Note that for this to work, m4-native has to have previously built with
the hashserv running, otherwise it has nothing to compare its output
to.

I think this should be a "big deal" for many developers, reducing
unneded rebuilds and hence speeding up development.

As with any new feature, there are likely going to be some some
teething issues but I'm very opitmistic about some of the benefits this
should bring!

Cheers,

Richard

Re: Hash Equivalency - What this means for developer productivity

[Richard Purdie]

On Fri, 2019-08-02 at 16:53 +0100, Richard Purdie wrote:
> With the patches in master-next and this configuration in local.conf:
> 
> BB_HASHSERVE = "localhost:0"
> BB_SIGNATURE_HANDLER = "OEEquivHash"
> 
> $ bitbake core-image-sato
> $ bitbake m4-native -c install -f
> $ bitbake core-image-sato
> 
> will result in do_populate_sysroot of m4-native running, it will see
> the output matches the previous build and it will then skip to the
> rootfs generation pulling all the other pieces from sstate.
> 
> Note that for this to work, m4-native has to have previously built
> with the hashserv running, otherwise it has nothing to compare its
> output to.
> 
> I think this should be a "big deal" for many developers, reducing
> unneeded rebuilds and hence speeding up development.

I should have mentioned, this code relies on reproducibile builds as
its comparing the binary output. The more reproducibile builds are, the
more likely sstate reuse will happen.

This is one reason reproducibile builds are important!

We may want to look at more interesting comparison mechanisms in the
future like the APIs exposed by libraries. Having support for matching
"equivalent" binaries in the core is the key first step to these things
though.

Cheers,

Richard

Re: Hash Equivalency - What this means for developer productivity

[Mikko.Rapeli]

Hi,

On Fri, Aug 02, 2019 at 05:17:21PM +0100, Richard Purdie wrote:
> On Fri, 2019-08-02 at 16:53 +0100, Richard Purdie wrote:
> > With the patches in master-next and this configuration in local.conf:
> > 
> > BB_HASHSERVE = "localhost:0"
> > BB_SIGNATURE_HANDLER = "OEEquivHash"
> > 
> > $ bitbake core-image-sato
> > $ bitbake m4-native -c install -f
> > $ bitbake core-image-sato
> > 
> > will result in do_populate_sysroot of m4-native running, it will see
> > the output matches the previous build and it will then skip to the
> > rootfs generation pulling all the other pieces from sstate.
> > 
> > Note that for this to work, m4-native has to have previously built
> > with the hashserv running, otherwise it has nothing to compare its
> > output to.
> > 
> > I think this should be a "big deal" for many developers, reducing
> > unneeded rebuilds and hence speeding up development.

Awesome, thanks for pushing this!

> I should have mentioned, this code relies on reproducibile builds as
> its comparing the binary output. The more reproducibile builds are, the
> more likely sstate reuse will happen.
> 
> This is one reason reproducibile builds are important!

What else do users need to enable to get more reproducible builds, or
are poky defaults enough?

Are there some tools available to debug build reproducibility issues e.g.
when task hashes suddenly changed?

Cheers,

-Mikko

Re: Hash Equivalency - What this means for developer productivity

[Joshua Watt]

On 8/5/19 2:21 AM, Mikko.Rapeli@bmw.de wrote:
> Hi,
>
> On Fri, Aug 02, 2019 at 05:17:21PM +0100, Richard Purdie wrote:
>> On Fri, 2019-08-02 at 16:53 +0100, Richard Purdie wrote:
>>> With the patches in master-next and this configuration in local.conf:
>>>
>>> BB_HASHSERVE = "localhost:0"
>>> BB_SIGNATURE_HANDLER = "OEEquivHash"
>>>
>>> $ bitbake core-image-sato
>>> $ bitbake m4-native -c install -f
>>> $ bitbake core-image-sato
>>>
>>> will result in do_populate_sysroot of m4-native running, it will see
>>> the output matches the previous build and it will then skip to the
>>> rootfs generation pulling all the other pieces from sstate.
>>>
>>> Note that for this to work, m4-native has to have previously built
>>> with the hashserv running, otherwise it has nothing to compare its
>>> output to.
>>>
>>> I think this should be a "big deal" for many developers, reducing
>>> unneeded rebuilds and hence speeding up development.
> Awesome, thanks for pushing this!
>
>> I should have mentioned, this code relies on reproducibile builds as
>> its comparing the binary output. The more reproducibile builds are, the
>> more likely sstate reuse will happen.
>>
>> This is one reason reproducibile builds are important!
> What else do users need to enable to get more reproducible builds, or
> are poky defaults enough?

You can test with what we currently have by adding:

  INHERIT += "reproducible_build"

to local.conf. This is subject to change though as we are still sorting 
out some of the details.

>
> Are there some tools available to debug build reproducibility issues e.g.
> when task hashes suddenly changed?

The current goal of the reproducible build work is to create and OEQA 
test that checks if all recipes that are required for a given image can 
be built in a reproducible manner. This QA test will be run on the 
autobuilder to detect regressions. We are currently working toward 
getting core-image-minimal reproducible, but I think Ross Burton 
investigated core-image-sato and determined it wouldn't be too much 
extra work, so we might try that one also. Either way, the test is 
designed to be be easily extensible so you can write your own tests on 
whatever image you would like.

I think that there is a significant opportunity to improve the reporting 
and tests for reproducible builds, and much of this complements hash 
equivalence. For example, the hash equivalence code is (or can be) 
really good at reporting why a task's output hash changed, which would 
be really useful information for debugging reproducibility.

I also have some patches that should make it easier to debug packages 
that didn't build reproducibly when the QA check fails by allowing you 
to run diffoscope on them.

>
> Cheers,
>
> -Mikko

Re: Hash Equivalency - What this means for developer productivity

[Jacob Kroon]

Hi,

On 8/2/19 5:53 PM, Richard Purdie wrote:
> This has been talked about in a lot of the project meetings but I
> wanted to ensure its on the mailing list too.
>
> master-next (and hopefully soon, master) now contain patches which
> enable "hash equivalency" support in runqueue in bitbake. That doesn't
> sound so interesting until I explain what this means.
>
<snip>
>
> I think this should be a "big deal" for many developers, reducing
> unneded rebuilds and hence speeding up development.
>
> As with any new feature, there are likely going to be some some
> teething issues but I'm very opitmistic about some of the benefits this
> should bring!
>

Very exciting see this feature being worked on! I have high hopes it is
gonna reduce my build times.

I just updated to latest bitbake/oe-core master and started an
incremental build of my distro with hash equivalency enabled, and ran
into one problem:

ERROR: core-image-minimal-1.0-r0 do_rootfs: Unable to install packages.
<snip>
Collected errors:
  * Solver encountered 2 problem(s):
  * Problem 1/2:
  *   - nothing provides eudev needed by <snip>
<snip>
  * Problem 2/2:
  *   - nothing provides udev needed by <snip>
<snip>

Checking the build log I see:

...
NOTE: Task
/home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-core/glib-2.0/glib-2.0_2.60.6.bb:do_populate_sysroot
unihash changed to
b88f1b01421cd7501b290888b783b057b92a7fb7a20f37fd08a56f2eaf8c56ed
NOTE: Setscene task
/home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-core/udev/eudev_3.2.8.bb:do_populate_sysroot
now valid and being rerun
...
NOTE: Setscene tasks completed

NOTE: Task
/home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-core/udev/eudev_3.2.8.bb:do_package_qa
unihash changed to
607d4eb7859bf6c725add7592a2c8bd64464afb259264281cbf2a3ad4e17e8d2
...
NOTE: Task
/home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-core/glib-2.0/glib-2.0_2.60.6.bb:do_package_qa
unihash changed to
89413c6afc25b390471f0cebb6193a226e20a744ac86b473c0b11f7432b5165a
...
NOTE: Setscene tasks completed

...

Checking for built ipk packages I see:

[jkroon@manatarms-greyskull <snip>]$ find
/tmp/ramdisk/<snip>/deploy/ipk/|grep eudev
/tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dbg_3.2.8-r0_i686.ipk
/tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dev_3.2.8-r0_i686.ipk

If I then do:

bitbake -c cleansstate eudev
bitbake eudev

then I get:

[jkroon@manatarms-greyskull <snip>]$ find
/tmp/ramdisk/<snip>/deploy/ipk/|grep eudev
/tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dev_3.2.8-r0_i686.ipk
/tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dbg_3.2.8-r0_i686.ipk
/tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-hwdb_3.2.8-r0_i686.ipk
/tmp/ramdisk/<snip>/deploy/ipk/i686/eudev_3.2.8-r0_i686.ipk
/tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-src_3.2.8-r0_i686.ipk

and then I can complete building the image.

I did not have hash equivalency enabled in the beginning, I've just
recently enabled it, without wiping tmp-build/ or sstate in between.

/Jacob


[mikrodidakt]
Jacob Kroon • +46 46325040

mikrodidakt.se • Skiffervägen 48, SE-224 78 LUND, Sweden

Consultans since 1980 • SW, HW, Embedded Systems, Linux

Re: Hash Equivalency - What this means for developer productivity

[richard.purdie]

On Thu, 2019-08-08 at 10:18 +0000, Jacob Kroon wrote:
> Hi,
> 
> On 8/2/19 5:53 PM, Richard Purdie wrote:
> > This has been talked about in a lot of the project meetings but I
> > wanted to ensure its on the mailing list too.
> > 
> > master-next (and hopefully soon, master) now contain patches which
> > enable "hash equivalency" support in runqueue in bitbake. That
> > doesn't
> > sound so interesting until I explain what this means.
> > 
> <snip>
> > I think this should be a "big deal" for many developers, reducing
> > unneded rebuilds and hence speeding up development.
> > 
> > As with any new feature, there are likely going to be some some
> > teething issues but I'm very opitmistic about some of the benefits
> > this
> > should bring!
> > 
> 
> Very exciting see this feature being worked on! I have high hopes it
> is
> gonna reduce my build times.
> 
> I just updated to latest bitbake/oe-core master and started an
> incremental build of my distro with hash equivalency enabled, and ran
> into one problem:
> 
> ERROR: core-image-minimal-1.0-r0 do_rootfs: Unable to install
> packages.
> <snip>
> Collected errors:
>   * Solver encountered 2 problem(s):
>   * Problem 1/2:
>   *   - nothing provides eudev needed by <snip>
> <snip>
>   * Problem 2/2:
>   *   - nothing provides udev needed by <snip>
> <snip>
> 
> Checking the build log I see:
> 
> ...
> NOTE: Task
> /home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-
> core/glib-2.0/glib-2.0_2.60.6.bb:do_populate_sysroot
> unihash changed to
> b88f1b01421cd7501b290888b783b057b92a7fb7a20f37fd08a56f2eaf8c56ed
> NOTE: Setscene task
> /home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-
> core/udev/eudev_3.2.8.bb:do_populate_sysroot
> now valid and being rerun
> ...
> NOTE: Setscene tasks completed
> 
> NOTE: Task
> /home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-
> core/udev/eudev_3.2.8.bb:do_package_qa
> unihash changed to
> 607d4eb7859bf6c725add7592a2c8bd64464afb259264281cbf2a3ad4e17e8d2
> ...
> NOTE: Task
> /home/jkroon/Projects/<snip>/build/../openembedded-core/meta/recipes-
> core/glib-2.0/glib-2.0_2.60.6.bb:do_package_qa
> unihash changed to
> 89413c6afc25b390471f0cebb6193a226e20a744ac86b473c0b11f7432b5165a
> ...
> NOTE: Setscene tasks completed
> 
> ...
> 
> Checking for built ipk packages I see:
> 
> [jkroon@manatarms-greyskull <snip>]$ find
> /tmp/ramdisk/<snip>/deploy/ipk/|grep eudev
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dbg_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dev_3.2.8-r0_i686.ipk
> 
> If I then do:
> 
> bitbake -c cleansstate eudev
> bitbake eudev
> 
> then I get:
> 
> [jkroon@manatarms-greyskull <snip>]$ find
> /tmp/ramdisk/<snip>/deploy/ipk/|grep eudev
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dev_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dbg_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-hwdb_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-src_3.2.8-r0_i686.ipk
> 
> and then I can complete building the image.
> 
> I did not have hash equivalency enabled in the beginning, I've just
> recently enabled it, without wiping tmp-build/ or sstate in between.

That sounds like some kind of rebuild problem with eudev rather than a
problem with the new runqueue code. It looks like eudev rebuilt but
only produces some subset of the packages it should have done.

If you still had the logs for eudev it would be have been interesting
to see if the install/package logs looked correct.

FWIW, enabling this on an existing build should work fine, it may just
take an initial rebuild of a given recipe for it to have something
compare against so it wouldn't work quite as effectively until after a
few builds where things have rebuilt.

Cheers,

Richard

Re: Hash Equivalency - What this means for developer productivity

[richard.purdie]

On Thu, 2019-08-08 at 10:18 +0000, Jacob Kroon wrote:
> Checking for built ipk packages I see:
> 
> [jkroon@manatarms-greyskull <snip>]$ find
> /tmp/ramdisk/<snip>/deploy/ipk/|grep eudev
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dbg_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dev_3.2.8-r0_i686.ipk
> 
> If I then do:
> 
> bitbake -c cleansstate eudev
> bitbake eudev
> 
> then I get:
> 
> [jkroon@manatarms-greyskull <snip>]$ find
> /tmp/ramdisk/<snip>/deploy/ipk/|grep eudev
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dev_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-dbg_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-hwdb_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev_3.2.8-r0_i686.ipk
> /tmp/ramdisk/<snip>/deploy/ipk/i686/eudev-src_3.2.8-r0_i686.ipk
> 
> and then I can complete building the image.
> 
> I did not have hash equivalency enabled in the beginning, I've just
> recently enabled it, without wiping tmp-build/ or sstate in between.

I've just seen this with dnf on the autobuilders and an rpm build. The
most interesting bit was:

[pokybuild@centos7-ty-1 temp]$ pwd
/home/pokybuild/yocto-worker/genericx86-lsb/build/build/tmp/work/core2-32-poky-linux/dnf/4.2.2-r0/temp
[pokybuild@centos7-ty-1 temp]$ cat log.task_order 
do_populate_lic_setscene (19174): log.do_populate_lic_setscene.19174
do_fetch (89722): log.do_fetch.89722
do_unpack (93503): log.do_unpack.93503
do_patch (97696): log.do_patch.97696
do_generate_toolchain_file (114754): log.do_generate_toolchain_file.114754
do_populate_sysroot_setscene (107361): log.do_populate_sysroot_setscene.107361
do_package (108444): log.do_package.108444
do_packagedata (28709): log.do_packagedata.28709
do_package_qa (21415): log.do_package_qa.21415
do_package_write_rpm (21400): log.do_package_write_rpm.21400
do_package_write_deb (94084): log.do_package_write_deb.94084
do_package_write_ipk (94059): log.do_package_write_ipk.94059

i.e. do_populate_sysroot_setscene let it skip do_configure, do_compile
and do_install and just run do_package. Which is very clearly badly
wrong!

I'll poke into this further tomorrow but I have some idea of how this 
might happen...

Cheers,

Richard

Re: Hash Equivalency - What this means for developer productivity

[Richard Purdie]

On Fri, 2019-08-09 at 00:12 +0100, richard.purdie@linuxfoundation.org
wrote:
> I've just seen this with dnf on the autobuilders and an rpm build.
> The most interesting bit was:
> 
> [pokybuild@centos7-ty-1 temp]$ pwd
> /home/pokybuild/yocto-worker/genericx86-
> lsb/build/build/tmp/work/core2-32-poky-linux/dnf/4.2.2-r0/temp
> [pokybuild@centos7-ty-1 temp]$ cat log.task_order 
> do_populate_lic_setscene (19174): log.do_populate_lic_setscene.19174
> do_fetch (89722): log.do_fetch.89722
> do_unpack (93503): log.do_unpack.93503
> do_patch (97696): log.do_patch.97696
> do_generate_toolchain_file (114754):
> log.do_generate_toolchain_file.114754
> do_populate_sysroot_setscene (107361):
> log.do_populate_sysroot_setscene.107361
> do_package (108444): log.do_package.108444
> do_packagedata (28709): log.do_packagedata.28709
> do_package_qa (21415): log.do_package_qa.21415
> do_package_write_rpm (21400): log.do_package_write_rpm.21400
> do_package_write_deb (94084): log.do_package_write_deb.94084
> do_package_write_ipk (94059): log.do_package_write_ipk.94059
> 
> i.e. do_populate_sysroot_setscene let it skip do_configure,
> do_compile
> and do_install and just run do_package. Which is very clearly badly
> wrong!
> 
> I'll poke into this further tomorrow but I have some idea of how this
> 
> might happen...
> 

This turned out to be worlds of pain and I ended up redoing the way
part of runqueue works, for the better as it turns out as the code is
much simpler. Its given me quite a headache trying to digure out how it
was breaking and reproducing it.

The new code is in master-next and I'm trying it on the autobuilder. it
shouldn't have the above problem any more, what other issues I may have
added remains to be seen!

Cheers,

Richard

Re: Hash Equivalency - What this means for developer productivity

[Jacob Kroon]

On 8/12/19 5:32 PM, Richard Purdie wrote:
> On Fri, 2019-08-09 at 00:12 +0100, richard.purdie@linuxfoundation.org
> wrote:
>> I've just seen this with dnf on the autobuilders and an rpm build.
>> The most interesting bit was:
>>
>> [pokybuild@centos7-ty-1 temp]$ pwd
>> /home/pokybuild/yocto-worker/genericx86-
>> lsb/build/build/tmp/work/core2-32-poky-linux/dnf/4.2.2-r0/temp
>> [pokybuild@centos7-ty-1 temp]$ cat log.task_order
>> do_populate_lic_setscene (19174): log.do_populate_lic_setscene.19174
>> do_fetch (89722): log.do_fetch.89722
>> do_unpack (93503): log.do_unpack.93503
>> do_patch (97696): log.do_patch.97696
>> do_generate_toolchain_file (114754):
>> log.do_generate_toolchain_file.114754
>> do_populate_sysroot_setscene (107361):
>> log.do_populate_sysroot_setscene.107361
>> do_package (108444): log.do_package.108444
>> do_packagedata (28709): log.do_packagedata.28709
>> do_package_qa (21415): log.do_package_qa.21415
>> do_package_write_rpm (21400): log.do_package_write_rpm.21400
>> do_package_write_deb (94084): log.do_package_write_deb.94084
>> do_package_write_ipk (94059): log.do_package_write_ipk.94059
>>
>> i.e. do_populate_sysroot_setscene let it skip do_configure,
>> do_compile
>> and do_install and just run do_package. Which is very clearly badly
>> wrong!
>>
>> I'll poke into this further tomorrow but I have some idea of how this
>>
>> might happen...
>>
>
> This turned out to be worlds of pain and I ended up redoing the way
> part of runqueue works, for the better as it turns out as the code is
> much simpler. Its given me quite a headache trying to digure out how it
> was breaking and reproducing it.
>
> The new code is in master-next and I'm trying it on the autobuilder. it
> shouldn't have the above problem any more, what other issues I may have
> added remains to be seen!
>

Thank you for fixing!
/Jacob


[mikrodidakt]
Jacob Kroon • +46 46325040

mikrodidakt.se • Skiffervägen 48, SE-224 78 LUND, Sweden

Consultans since 1980 • SW, HW, Embedded Systems, Linux