sstate hash equivalence breaks rm_work

sstate hash equivalence breaks rm_work

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

Hey all,

As part of the packagegroup-core-base-utils series I noticed that
with rm_work enabled like I usually have, every build was starting over
with rebuilding linux-libc-headers and going down from there.  I
finished bisecting this now and it comes down to:

commit d889acb4f8f06f09cece80fa12661725e6e5f037
Author: Joshua Watt <jpewhacker@gmail.com>
Date:   Fri Jan 4 10:20:15 2019 -0600

    sstate: Implement hash equivalence sstate

This can be reproduced easily with qemux86, core-image-minimal, oe-core
and this in site.conf:
# Save more space
INHERIT += "rm_work"
# But keep this around for debugging
RM_WORK_EXCLUDE += ""
INHERIT += "buildstats"
INHERIT += "buildstats-summary"

(I doubt the last two matter, but I only noticed that was in my bisect
just now).

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: sstate hash equivalence breaks rm_work

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

On Sun, Mar 3, 2019, 2:47 PM Tom Rini <trini@konsulko.com> wrote:

> Hey all,
>
> As part of the packagegroup-core-base-utils series I noticed that
> with rm_work enabled like I usually have, every build was starting over
> with rebuilding linux-libc-headers and going down from there.  I
> finished bisecting this now and it comes down to:


Just for reference, are you trying to use hash equivalence? It sounds like
not, but I would like to make sure.

Also, it there an AB build that uses rm_work that would have seen this?


> commit d889acb4f8f06f09cece80fa12661725e6e5f037
> Author: Joshua Watt <jpewhacker@gmail.com>
> Date:   Fri Jan 4 10:20:15 2019 -0600
>
>     sstate: Implement hash equivalence sstate
>
> This can be reproduced easily with qemux86, core-image-minimal, oe-core
> and this in site.conf:
> # Save more space
> INHERIT += "rm_work"
> # But keep this around for debugging
> RM_WORK_EXCLUDE += ""
> INHERIT += "buildstats"
> INHERIT += "buildstats-summary"
>
> (I doubt the last two matter, but I only noticed that was in my bisect
> just now).
>
> --
> Tom
>

[-- Attachment #2: Type: text/html, Size: 1876 bytes --]

Re: sstate hash equivalence breaks rm_work

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

On Sun, Mar 03, 2019 at 03:21:12PM -0600, Joshua Watt wrote:
> On Sun, Mar 3, 2019, 2:47 PM Tom Rini <trini@konsulko.com> wrote:
> 
> > Hey all,
> >
> > As part of the packagegroup-core-base-utils series I noticed that
> > with rm_work enabled like I usually have, every build was starting over
> > with rebuilding linux-libc-headers and going down from there.  I
> > finished bisecting this now and it comes down to:
> 
> Just for reference, are you trying to use hash equivalence? It sounds like
> not, but I would like to make sure.

I am not.

> Also, it there an AB build that uses rm_work that would have seen this?

OK, so I see I forgot a step that was clear to me, but, uh, not in the
email.  What breaks is for example:
bitbake core-image-minimal && bitbake core-image-minimal
That second build, rather than see "nothing to do" (or at least, almost
nothing to do) says "Oh, I need to build linux-libc-headers from
scratch" and possibly other stuff too, but that's the first thing I
notice.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: sstate hash equivalence breaks rm_work

[Joshua Watt]

On Sun, Mar 3, 2019 at 3:25 PM Tom Rini <trini@konsulko.com> wrote:
>
> On Sun, Mar 03, 2019 at 03:21:12PM -0600, Joshua Watt wrote:
> > On Sun, Mar 3, 2019, 2:47 PM Tom Rini <trini@konsulko.com> wrote:
> >
> > > Hey all,
> > >
> > > As part of the packagegroup-core-base-utils series I noticed that
> > > with rm_work enabled like I usually have, every build was starting over
> > > with rebuilding linux-libc-headers and going down from there.  I
> > > finished bisecting this now and it comes down to:
> >
> > Just for reference, are you trying to use hash equivalence? It sounds like
> > not, but I would like to make sure.
>
> I am not.
>
> > Also, it there an AB build that uses rm_work that would have seen this?
>
> OK, so I see I forgot a step that was clear to me, but, uh, not in the
> email.  What breaks is for example:
> bitbake core-image-minimal && bitbake core-image-minimal
> That second build, rather than see "nothing to do" (or at least, almost
> nothing to do) says "Oh, I need to build linux-libc-headers from
> scratch" and possibly other stuff too, but that's the first thing I
> notice.

Ok. I've started a build to test it. Would you mind opening a Bugzilla
for it and assigning it to me?

For posterity, the poky commit SHA is adc37721a86c

>
> --
> Tom

Re: sstate hash equivalence breaks rm_work

[Joshua Watt]

On Sun, Mar 3, 2019 at 4:11 PM Joshua Watt <jpewhacker@gmail.com> wrote:
>
> On Sun, Mar 3, 2019 at 3:25 PM Tom Rini <trini@konsulko.com> wrote:
> >
> > On Sun, Mar 03, 2019 at 03:21:12PM -0600, Joshua Watt wrote:
> > > On Sun, Mar 3, 2019, 2:47 PM Tom Rini <trini@konsulko.com> wrote:
> > >
> > > > Hey all,
> > > >
> > > > As part of the packagegroup-core-base-utils series I noticed that
> > > > with rm_work enabled like I usually have, every build was starting over
> > > > with rebuilding linux-libc-headers and going down from there.  I
> > > > finished bisecting this now and it comes down to:
> > >
> > > Just for reference, are you trying to use hash equivalence? It sounds like
> > > not, but I would like to make sure.
> >
> > I am not.
> >
> > > Also, it there an AB build that uses rm_work that would have seen this?
> >
> > OK, so I see I forgot a step that was clear to me, but, uh, not in the
> > email.  What breaks is for example:
> > bitbake core-image-minimal && bitbake core-image-minimal
> > That second build, rather than see "nothing to do" (or at least, almost
> > nothing to do) says "Oh, I need to build linux-libc-headers from
> > scratch" and possibly other stuff too, but that's the first thing I
> > notice.
>
> Ok. I've started a build to test it. Would you mind opening a Bugzilla
> for it and assigning it to me?
>
> For posterity, the poky commit SHA is adc37721a86c

Hmm, FWIW I just tried this on my setup with rm_work enabled and it
worked as expected (e.g. did not rebuild unnecessarily).... would you
be able to share more of your setup (local.conf, layers.conf,
site.conf)?

>
> >
> > --
> > Tom

Re: sstate hash equivalence breaks rm_work

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2316 bytes --]

On Sun, Mar 03, 2019 at 04:20:49PM -0600, Joshua Watt wrote:
> On Sun, Mar 3, 2019 at 4:11 PM Joshua Watt <jpewhacker@gmail.com> wrote:
> >
> > On Sun, Mar 3, 2019 at 3:25 PM Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Sun, Mar 03, 2019 at 03:21:12PM -0600, Joshua Watt wrote:
> > > > On Sun, Mar 3, 2019, 2:47 PM Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > > Hey all,
> > > > >
> > > > > As part of the packagegroup-core-base-utils series I noticed that
> > > > > with rm_work enabled like I usually have, every build was starting over
> > > > > with rebuilding linux-libc-headers and going down from there.  I
> > > > > finished bisecting this now and it comes down to:
> > > >
> > > > Just for reference, are you trying to use hash equivalence? It sounds like
> > > > not, but I would like to make sure.
> > >
> > > I am not.
> > >
> > > > Also, it there an AB build that uses rm_work that would have seen this?
> > >
> > > OK, so I see I forgot a step that was clear to me, but, uh, not in the
> > > email.  What breaks is for example:
> > > bitbake core-image-minimal && bitbake core-image-minimal
> > > That second build, rather than see "nothing to do" (or at least, almost
> > > nothing to do) says "Oh, I need to build linux-libc-headers from
> > > scratch" and possibly other stuff too, but that's the first thing I
> > > notice.
> >
> > Ok. I've started a build to test it. Would you mind opening a Bugzilla
> > for it and assigning it to me?
> >
> > For posterity, the poky commit SHA is adc37721a86c
> 
> Hmm, FWIW I just tried this on my setup with rm_work enabled and it
> worked as expected (e.g. did not rebuild unnecessarily).... would you
> be able to share more of your setup (local.conf, layers.conf,
> site.conf)?

On oe-core/master, local.conf / layers.conf are as setup.  All of site.conf is:
DL_DIR = "${HOME}/work/OE/downloads"
SSTATE_DIR = "${HOME}/work/OE/sstate-cache-${DISTRO_VERSION}${TARGET_VENDOR}"

# Save more space
INHERIT += "rm_work"
# But keep this around for debugging
RM_WORK_EXCLUDE += ""

INHERIT += "buildstats"
INHERIT += "buildstats-summary"

And:
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.6 LTS
Release:	16.04
Codename:	xenial

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: sstate hash equivalence breaks rm_work

[Burton, Ross]

On Sun, 3 Mar 2019 at 22:52, Tom Rini <trini@konsulko.com> wrote:
> SSTATE_DIR = "${HOME}/work/OE/sstate-cache-${DISTRO_VERSION}${TARGET_VENDOR}"

FWIW there isn't much point in isolating sstate like that.  You'll
just rebuild stuff that would have been available.

Ross

Re: sstate hash equivalence breaks rm_work

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

There is, you can prune outdated duplicate sstate archives while still
separately building multiple DISTRO_VERSIONs.

On Mon, Mar 4, 2019 at 11:17 AM Burton, Ross <ross.burton@intel.com> wrote:

> On Sun, 3 Mar 2019 at 22:52, Tom Rini <trini@konsulko.com> wrote:
> > SSTATE_DIR =
> "${HOME}/work/OE/sstate-cache-${DISTRO_VERSION}${TARGET_VENDOR}"
>
> FWIW there isn't much point in isolating sstate like that.  You'll
> just rebuild stuff that would have been available.
>
> Ross
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 1270 bytes --]

Re: sstate hash equivalence breaks rm_work

[Burton, Ross]

On Mon, 4 Mar 2019 at 10:27, Martin Jansa <martin.jansa@gmail.com> wrote:
> There is, you can prune outdated duplicate sstate archives while still separately building multiple DISTRO_VERSIONs.

We don't all have 4TB disks? :)

Ross

Re: sstate hash equivalence breaks rm_work

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]

4TB of sstate is just for few weeks for me :).

On Mon, Mar 4, 2019 at 11:45 AM Burton, Ross <ross.burton@intel.com> wrote:

> On Mon, 4 Mar 2019 at 10:27, Martin Jansa <martin.jansa@gmail.com> wrote:
> > There is, you can prune outdated duplicate sstate archives while still
> separately building multiple DISTRO_VERSIONs.
>
> We don't all have 4TB disks? :)
>
> Ross
>

[-- Attachment #2: Type: text/html, Size: 741 bytes --]

Re: sstate hash equivalence breaks rm_work

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

On Mon, Mar 04, 2019 at 10:17:21AM +0000, Burton, Ross wrote:
> On Sun, 3 Mar 2019 at 22:52, Tom Rini <trini@konsulko.com> wrote:
> > SSTATE_DIR = "${HOME}/work/OE/sstate-cache-${DISTRO_VERSION}${TARGET_VENDOR}"
> 
> FWIW there isn't much point in isolating sstate like that.  You'll
> just rebuild stuff that would have been available.

Martin already chimed in, but this is my "copy this to every project"
site.conf file.  I do modify that base path a little bit for per
project,  But easily removable state is good and important, especially
when chasing "recipe for proprietary thing behaves badly" bugs.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: sstate hash equivalence breaks rm_work

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1751 bytes --]

On Sun, Mar 03, 2019 at 04:11:36PM -0600, Joshua Watt wrote:
> On Sun, Mar 3, 2019 at 3:25 PM Tom Rini <trini@konsulko.com> wrote:
> >
> > On Sun, Mar 03, 2019 at 03:21:12PM -0600, Joshua Watt wrote:
> > > On Sun, Mar 3, 2019, 2:47 PM Tom Rini <trini@konsulko.com> wrote:
> > >
> > > > Hey all,
> > > >
> > > > As part of the packagegroup-core-base-utils series I noticed that
> > > > with rm_work enabled like I usually have, every build was starting over
> > > > with rebuilding linux-libc-headers and going down from there.  I
> > > > finished bisecting this now and it comes down to:
> > >
> > > Just for reference, are you trying to use hash equivalence? It sounds like
> > > not, but I would like to make sure.
> >
> > I am not.
> >
> > > Also, it there an AB build that uses rm_work that would have seen this?
> >
> > OK, so I see I forgot a step that was clear to me, but, uh, not in the
> > email.  What breaks is for example:
> > bitbake core-image-minimal && bitbake core-image-minimal
> > That second build, rather than see "nothing to do" (or at least, almost
> > nothing to do) says "Oh, I need to build linux-libc-headers from
> > scratch" and possibly other stuff too, but that's the first thing I
> > notice.
> 
> Ok. I've started a build to test it. Would you mind opening a Bugzilla
> for it and assigning it to me?
> 
> For posterity, the poky commit SHA is adc37721a86c

Ah, so "poky" got me thinking this morning.  I was just on bitbake v1.40
(since I usually use thud, and that's the version for that).  Moving to
bitbake master/tip and I don't see the problem anymore.

Is it possible to have some logic within OE to check the version of
bitbake and throw an error / warning?

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: sstate hash equivalence breaks rm_work

[Richard Purdie]

On Mon, 2019-03-04 at 10:58 -0500, Tom Rini wrote:
> Ah, so "poky" got me thinking this morning.  I was just on bitbake
> v1.40
> (since I usually use thud, and that's the version for that).  Moving
> to
> bitbake master/tip and I don't see the problem anymore.
> 
> Is it possible to have some logic within OE to check the version of
> bitbake and throw an error / warning?

You mean like the BB_MIN_VERSION variable:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6

?

:)

The changes were intended to be backwards compatible hence it wasn't
bumped. Clearly something isn't playing as intended...

Cheers,

Richard

Re: sstate hash equivalence breaks rm_work

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1430 bytes --]

On Mon, Mar 04, 2019 at 04:06:47PM +0000, Richard Purdie wrote:
> On Mon, 2019-03-04 at 10:58 -0500, Tom Rini wrote:
> > Ah, so "poky" got me thinking this morning.  I was just on bitbake
> > v1.40
> > (since I usually use thud, and that's the version for that).  Moving
> > to
> > bitbake master/tip and I don't see the problem anymore.
> > 
> > Is it possible to have some logic within OE to check the version of
> > bitbake and throw an error / warning?
> 
> You mean like the BB_MIN_VERSION variable:
> 
> http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6
> 
> ?
> 
> :)
>
> The changes were intended to be backwards compatible hence it wasn't
> bumped. Clearly something isn't playing as intended...

Yes, that.  I think we haven't fully used that well historically since I
can only ever recall *kaboom* when using too old of a bitbake rather
than a nice sane error message.  Looking at
https://wiki.yoctoproject.org/wiki/Releases and then some quick git
history, we also must have had that mechanism available when I last ran
into something like that even for the fairly ancient jumping around I've
had to do a time or three.

In fact, can we make BB_MIN_VERSION match the table found there?  It
doesn't look like we have resources / time to test things on both
current and min versions, so we should I think have the min match what
we say should be used.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: sstate hash equivalence breaks rm_work

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Mon, Mar 4, 2019, 10:06 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2019-03-04 at 10:58 -0500, Tom Rini wrote:
> > Ah, so "poky" got me thinking this morning.  I was just on bitbake
> > v1.40
> > (since I usually use thud, and that's the version for that).  Moving
> > to
> > bitbake master/tip and I don't see the problem anymore.
> >
> > Is it possible to have some logic within OE to check the version of
> > bitbake and throw an error / warning?
>
> You mean like the BB_MIN_VERSION variable:
>
> http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6
>
> ?
>
> :)
>
> The changes were intended to be backwards compatible hence it wasn't
> bumped. Clearly something isn't playing as intended...
>

Ah, older bitbake isn't going to set BB_UNIHASH, which I bet is what breaks
it. I bet if we add:

 BB_UNIHASH ?= "${BB_TASKHASH}"

somewhere (bitbake.conf?) it would fix it.


> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 1855 bytes --]

Re: sstate hash equivalence breaks rm_work

[Richard Purdie]

On Mon, 2019-03-04 at 11:14 -0500, Tom Rini wrote:
> On Mon, Mar 04, 2019 at 04:06:47PM +0000, Richard Purdie wrote:
> > You mean like the BB_MIN_VERSION variable:
> > 
> > http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6
> > 
> > ?
> > 
> > :)
> > 
> > The changes were intended to be backwards compatible hence it
> > wasn't
> > bumped. Clearly something isn't playing as intended...
> 
> Yes, that.  I think we haven't fully used that well historically
> since I can only ever recall *kaboom* when using too old of a bitbake
> rather than a nice sane error message.  Looking at
> https://wiki.yoctoproject.org/wiki/Releases and then some quick git
> history, we also must have had that mechanism available when I last
> ran into something like that even for the fairly ancient jumping
> around I've had to do a time or three.

That variable has been around for a long time. It does get incremented
when we do nasty things which we know break things. If something sneaks
in, its more problematic. We do try and avoid API breakage.

> In fact, can we make BB_MIN_VERSION match the table found there?  It
> doesn't look like we have resources / time to test things on both
> current and min versions, so we should I think have the min match
> what we say should be used.

The problem is more "test what?" on the min version. In this case its
rm_work which triggers it, the bugs tend to be unusual corner cases
which our test matrix, vast as it is might not catch. Case in hand, we
don't test rm_work on the autobuilder.

I'm torn on forcing people to update, I've had a lot of complaints
about being being forced when they didn't need to in the past. Its also
true that we're not as backwards compatible as we could be, i.e. using
a modern bitbake on old metadata isn't always guaranteed either due to
the same kinds of problems :(.

By this line of argument, we'd bump the minor bitbake version for every
bug fix and force people to it, just to "protect" them from themselves
:/

Cheers,

Richard

Re: sstate hash equivalence breaks rm_work

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 3175 bytes --]

On Mon, Mar 04, 2019 at 04:21:48PM +0000, Richard Purdie wrote:
> On Mon, 2019-03-04 at 11:14 -0500, Tom Rini wrote:
> > On Mon, Mar 04, 2019 at 04:06:47PM +0000, Richard Purdie wrote:
> > > You mean like the BB_MIN_VERSION variable:
> > > 
> > > http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/sanity.conf#n6
> > > 
> > > ?
> > > 
> > > :)
> > > 
> > > The changes were intended to be backwards compatible hence it
> > > wasn't
> > > bumped. Clearly something isn't playing as intended...
> > 
> > Yes, that.  I think we haven't fully used that well historically
> > since I can only ever recall *kaboom* when using too old of a bitbake
> > rather than a nice sane error message.  Looking at
> > https://wiki.yoctoproject.org/wiki/Releases and then some quick git
> > history, we also must have had that mechanism available when I last
> > ran into something like that even for the fairly ancient jumping
> > around I've had to do a time or three.
> 
> That variable has been around for a long time. It does get incremented
> when we do nasty things which we know break things. If something sneaks
> in, its more problematic. We do try and avoid API breakage.
> 
> > In fact, can we make BB_MIN_VERSION match the table found there?  It
> > doesn't look like we have resources / time to test things on both
> > current and min versions, so we should I think have the min match
> > what we say should be used.
> 
> The problem is more "test what?" on the min version. In this case its
> rm_work which triggers it, the bugs tend to be unusual corner cases
> which our test matrix, vast as it is might not catch. Case in hand, we
> don't test rm_work on the autobuilder.
> 
> I'm torn on forcing people to update, I've had a lot of complaints
> about being being forced when they didn't need to in the past. Its also
> true that we're not as backwards compatible as we could be, i.e. using
> a modern bitbake on old metadata isn't always guaranteed either due to
> the same kinds of problems :(.
> 
> By this line of argument, we'd bump the minor bitbake version for every
> bug fix and force people to it, just to "protect" them from themselves
> :/

So, what's worse, people hit a bug that's fixed in a newer version, or
when people upgrade one set of branches they have to update another
repository too?

Honestly, I could see the "but why do I have to update bitbake too?"
complaint being more reasonable back in the early days after we moved to
proper layers.  Now that it's not just oe-core/poky but also
meta-openembedded and meta-$BSP and maybe 2-5 other layers you also need
to bump from branch to branch, adding bitbake to that mix seems a lot
less of a big deal.

I thought about this and re-drafted something a few times now.  I think
perhaps a happy compromise would be to move the BB_MIN_VERSION check
from a hard-failure to the same level of warning we do on
SANITY_TESTED_DISTROS, and make it track the suggested version for a
release.  This will both help to catch bugs that have been fixed already
but still let people that know what they want (or don't want) to do, do
it.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]