RFC/RFT Checksums as part of stamp files

RFC/RFT Checksums as part of stamp files

[Richard Purdie]

We've been working on checksums and sstate for a while, one of the
pieces we've not put in place so far is using the checksums in the stamp
files.

What this would mean in English is the end of PR bumps since when you
change the recipe, the checksums of the appropriate tasks change and
everything automatically rebuilds that changed.

It also means if you change something in the core packaging, everything
should repackage and so on.

The final key to enabling this is this commit:

http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake-wip&id=b82259b901cae11b0e5715a92a580c9432af52cc

This hasn't been merged to master yet but we should discuss when this
change could/should be merged and under what criteria.

I've only just started testing full builds with the change myself so
keep that in mind. I'd appreciate testing but the change is still very
new and likely has bugs to iron out.

I'm also aware that this idea isn't going to be universally liked,
particularly I can guess that Koen is going to have concerns about this
and Angstrom since the lack of PR bumps means package feeds could start
to suffer.

My proposal for dealing with this will be to start having the packaging
backend take ownership of PE and PR fields. These should be auto
incremented when the build changes. That in itself is easy enough if you
have one build area, the tricky bit is where you have multiple build
sources feeding into a common packages repo like Angstrom.

For this we're going to need some kind of network service that takes a
version and provides the PE/PR values back, incrementing them each time
it sees a new version/checksum.

This would solve one of Koen's major complaints about PR bumps in
general once and for all but there is some effort needed to implement
something that could provide this service.

Also, on the plus side, we could also now make image generation a
"stamp" task with a stamp that will only change when the image inputs
change. That would change a users expectations in some regard so again
needs discussion.

Cheers,

Richard

Re: RFC/RFT Checksums as part of stamp files

[Gary Thomas]

On 01/06/2011 03:56 PM, Richard Purdie wrote:
> We've been working on checksums and sstate for a while, one of the
> pieces we've not put in place so far is using the checksums in the stamp
> files.
    ... snip
> My proposal for dealing with this will be to start having the packaging
> backend take ownership of PE and PR fields. These should be auto
> incremented when the build changes. That in itself is easy enough if you
> have one build area, the tricky bit is where you have multiple build
> sources feeding into a common packages repo like Angstrom.
>
> For this we're going to need some kind of network service that takes a
> version and provides the PE/PR values back, incrementing them each time
> it sees a new version/checksum.

How would this play with non-core packages/recipes, e.g. the ones that
are only in my local trees?

Also, how could this work in a no-network-available environment?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RFC/RFT Checksums as part of stamp files

[Richard Purdie]

On Thu, 2011-01-06 at 16:04 -0700, Gary Thomas wrote:
> On 01/06/2011 03:56 PM, Richard Purdie wrote:
> > We've been working on checksums and sstate for a while, one of the
> > pieces we've not put in place so far is using the checksums in the stamp
> > files.
>     ... snip
> > My proposal for dealing with this will be to start having the packaging
> > backend take ownership of PE and PR fields. These should be auto
> > incremented when the build changes. That in itself is easy enough if you
> > have one build area, the tricky bit is where you have multiple build
> > sources feeding into a common packages repo like Angstrom.
> >
> > For this we're going to need some kind of network service that takes a
> > version and provides the PE/PR values back, incrementing them each time
> > it sees a new version/checksum.
> 
> How would this play with non-core packages/recipes, e.g. the ones that
> are only in my local trees?
>
> Also, how could this work in a no-network-available environment?

Keep in mind this server solution is only needed where you are feeding
the output of multiple build machines into one package feed and need to
synchronise them with respect to versions. This is something you'd have
to explicitly setup (just like the package feed itself).

If you are feeding to a package feed its assumed you have some kind of
connectivity to it and that it can presumably host the synchronisation
solution.

Local recipes would work just the same as core ones, they all behave the
same way.

Yocto itself has no plans to host actual package feeds, we are likely to
end up with some kind of public sstate cache but that doesn't need any
synchronisation.

Cheers,

Richard

Re: RFC/RFT Checksums as part of stamp files

[Tian, Kevin]

>From: Richard Purdie
>Sent: Friday, January 07, 2011 6:56 AM
>
>We've been working on checksums and sstate for a while, one of the
>pieces we've not put in place so far is using the checksums in the stamp
>files.
>
>What this would mean in English is the end of PR bumps since when you
>change the recipe, the checksums of the appropriate tasks change and
>everything automatically rebuilds that changed.
>
>It also means if you change something in the core packaging, everything
>should repackage and so on.
>
>The final key to enabling this is this commit:
>
>http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake-wip&id=b82259
>b901cae11b0e5715a92a580c9432af52cc
>
>This hasn't been merged to master yet but we should discuss when this
>change could/should be merged and under what criteria.
>
>I've only just started testing full builds with the change myself so
>keep that in mind. I'd appreciate testing but the change is still very
>new and likely has bugs to iron out.

I can help test it out too.

>
>I'm also aware that this idea isn't going to be universally liked,
>particularly I can guess that Koen is going to have concerns about this
>and Angstrom since the lack of PR bumps means package feeds could start
>to suffer.

worthy of a config option? We may need see from real use whether this will
trigger many unnecessary rebuilds and how much impact to people's daily
work. Basically the incremental builds will be much more heavier than before,
though this in concept is more correct...

>
>My proposal for dealing with this will be to start having the packaging
>backend take ownership of PE and PR fields. These should be auto
>incremented when the build changes. That in itself is easy enough if you
>have one build area, the tricky bit is where you have multiple build
>sources feeding into a common packages repo like Angstrom.
>
>For this we're going to need some kind of network service that takes a
>version and provides the PE/PR values back, incrementing them each time
>it sees a new version/checksum.
>
>This would solve one of Koen's major complaints about PR bumps in
>general once and for all but there is some effort needed to implement
>something that could provide this service.
>
>Also, on the plus side, we could also now make image generation a
>"stamp" task with a stamp that will only change when the image inputs
>change. That would change a users expectations in some regard so again
>needs discussion.
>


Thanks
Kevin

Re: RFC/RFT Checksums as part of stamp files

[Koen Kooi]

Op 6 jan 2011, om 23:56 heeft Richard Purdie het volgende geschreven:

> We've been working on checksums and sstate for a while, one of the
> pieces we've not put in place so far is using the checksums in the stamp
> files.
> 
> What this would mean in English is the end of PR bumps since when you
> change the recipe, the checksums of the appropriate tasks change and
> everything automatically rebuilds that changed.
> 
> It also means if you change something in the core packaging, everything
> should repackage and so on.
> 
> The final key to enabling this is this commit:
> 
> http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake-wip&id=b82259b901cae11b0e5715a92a580c9432af52cc
> 
> This hasn't been merged to master yet but we should discuss when this
> change could/should be merged and under what criteria.
> 
> I've only just started testing full builds with the change myself so
> keep that in mind. I'd appreciate testing but the change is still very
> new and likely has bugs to iron out.
> 
> I'm also aware that this idea isn't going to be universally liked,
> particularly I can guess that Koen is going to have concerns about this
> and Angstrom since the lack of PR bumps means package feeds could start
> to suffer.

You saved me from writing a lengthy email about that :) 

> My proposal for dealing with this will be to start having the packaging
> backend take ownership of PE and PR fields. These should be auto
> incremented when the build changes. That in itself is easy enough if you
> have one build area, the tricky bit is where you have multiple build
> sources feeding into a common packages repo like Angstrom.
> 
> For this we're going to need some kind of network service that takes a
> version and provides the PE/PR values back, incrementing them each time
> it sees a new version/checksum.

As a simpler short term solution, can the build "master" write the info to disk (e.g. sqlite) so we can put it in a layer and build "slaves" would just read that back? That we we only need a variable to either read (slaves) or write (master) to that file. It also involves as little as possible network access, since it sits in the metadata git.

regards,

Koen

Re: RFC/RFT Checksums as part of stamp files

[Richard Purdie]

On Fri, 2011-01-07 at 12:49 +0800, Tian, Kevin wrote:
> >I'm also aware that this idea isn't going to be universally liked,
> >particularly I can guess that Koen is going to have concerns about this
> >and Angstrom since the lack of PR bumps means package feeds could start
> >to suffer.
> 
> worthy of a config option? We may need see from real use whether this will
> trigger many unnecessary rebuilds and how much impact to people's daily
> work. Basically the incremental builds will be much more heavier than before,
> though this in concept is more correct...

They are potentially heavier, yes.

With this and other factors in mind, I've merged a new signature
generator class into poky with this as the only added feature. Now with
poky's master branch you can add:

BB_SIGNATURE_HANDLER = "basichash"

to local.conf to enable this functionality (the default is "basic").

At some point we are likely to decide to change the default in
bitbake.conf but this makes it easier for people to test.

For reference, my test builds completed successfully.

Cheers,

Richard

Re: RFC/RFT Checksums as part of stamp files

[Tian, Kevin]

>From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
>Sent: Friday, January 07, 2011 7:09 PM
>
>On Fri, 2011-01-07 at 12:49 +0800, Tian, Kevin wrote:
>> >I'm also aware that this idea isn't going to be universally liked,
>> >particularly I can guess that Koen is going to have concerns about this
>> >and Angstrom since the lack of PR bumps means package feeds could start
>> >to suffer.
>>
>> worthy of a config option? We may need see from real use whether this will
>> trigger many unnecessary rebuilds and how much impact to people's daily
>> work. Basically the incremental builds will be much more heavier than before,
>> though this in concept is more correct...
>
>They are potentially heavier, yes.
>
>With this and other factors in mind, I've merged a new signature
>generator class into poky with this as the only added feature. Now with
>poky's master branch you can add:
>
>BB_SIGNATURE_HANDLER = "basichash"
>
>to local.conf to enable this functionality (the default is "basic").
>
>At some point we are likely to decide to change the default in
>bitbake.conf but this makes it easier for people to test.
>
>For reference, my test builds completed successfully.
>

my build finishes successfully too. I will manually bump PR to test its chain
rebuild effect for some gut-feeling. :-)

Thanks
Kevin

Re: RFC/RFT Checksums as part of stamp files

[Tian, Kevin]

>From: Tian, Kevin
>Sent: Friday, January 07, 2011 7:23 PM
>
>>From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
>>Sent: Friday, January 07, 2011 7:09 PM
>>
>>On Fri, 2011-01-07 at 12:49 +0800, Tian, Kevin wrote:
>>> >I'm also aware that this idea isn't going to be universally liked,
>>> >particularly I can guess that Koen is going to have concerns about this
>>> >and Angstrom since the lack of PR bumps means package feeds could start
>>> >to suffer.
>>>
>>> worthy of a config option? We may need see from real use whether this will
>>> trigger many unnecessary rebuilds and how much impact to people's daily
>>> work. Basically the incremental builds will be much more heavier than before,
>>> though this in concept is more correct...
>>
>>They are potentially heavier, yes.
>>
>>With this and other factors in mind, I've merged a new signature
>>generator class into poky with this as the only added feature. Now with
>>poky's master branch you can add:
>>
>>BB_SIGNATURE_HANDLER = "basichash"
>>
>>to local.conf to enable this functionality (the default is "basic").
>>
>>At some point we are likely to decide to change the default in
>>bitbake.conf but this makes it easier for people to test.
>>
>>For reference, my test builds completed successfully.
>>
>
>my build finishes successfully too. I will manually bump PR to test its chain
>rebuild effect for some gut-feeling. :-)
>

I tried to bump PR for m4-native, and it's effectively rebuilt from task 743 of
1625 with lots of native recipes rebuilt accordingly, as expected.

Thanks
Kevin