SRCPV migration

SRCPV migration

[Martin Jansa]

As I sent earlier as reply to [PATCH] bitbake.conf: SRCPV always upgradeable 
paths for git recipes.

First few changes are already in martin_jansa/srcpv branch. All changes to svn.bb 
recipes seems safe, so I would like to push them really soon. For git.bb changes
I need your advise/ack so please reply.

Original e-mail:

1) svn recipes with +svnr{SRCREV} could be just replaced with
+svnr{SRCPV} as the output should be the same

2) svn recipes with -r{SRCREV}, -svn{SRCREV}, -svnr{SRCREV}, +r{SRCREV},
+svn{SRCREV} unified to +svnr{SRCPV}? I'm not sure if '+' will be sorted
higher version wise, so do we need to bump PE?

3) git recipes with -git{SRCREV}, -gitr{SRCREV}, +git{SRCREV} unified to
+gitr{SRCPV} with PE bump?

Is there better way to create upgradeable path then bumping PE? Package
names/.ipk filenames are a bit ugly with PE imho.

Should I prepare one BIG patch for all recipes or commit them 1 by 1?

XorA said that migrating all recipes at once in his xora/angstrom-srcpv
branch was too much work to maintain and push. So I would like to push
it directly to oe.dev or with really short lived tmp branch, if all
agree that we should migrate all packages globally.

Some statistics:
1032 lines - grep -R "PV *=" recipes

571 "\(svn.bb\)\|\(git.bb\)\|\(SRCREV\)\|\(SRCPV\)"
342 "svn.bb"
258 "svn.bb.*SRCREV"
13  "svn.bb" | grep -v SRCREV | grep -v SRCDATE

without SRCDATEs
0   "svn.bb.*-r"
14  "svn.bb.*+svn\\$"
218 "svn.bb.*+svnr\\$" (this should be safe)

187 "git.bb"
42  "git.bb.*" | grep -v SRCREV (few with SRCDATE for git?)
0   "git.bb.*-git\\$"
32  "git.bb.*-gitr\\$"
18  "git.bb.*+git\\$"
100 "git.bb.*+gitr\\$"


Which should be resolved by package maintained maybe?
13  "svn.bb" | grep -v SRCREV | grep -v SRCDATE
      (probably should have SRCPV in PV, maybe its in PR)
./fltk/fltk2_svn.bb:PV = "1.9.9+svnr${SVNREL}"
./openscada/openscada_svn.bb:PV = "0.6.4"
./mythtv/mythtv_svn.bb:REALPV = "0.22"
./freesmartphone/fsoraw_svn.bb:PV = "0.0.1"
this should be easy:
./openmoko-panel-plugins/openmoko-panel-mainmenu_svn.bb:PV = "0.1.0+svn${SVNREV}"
./openmoko-panel-plugins/openmoko-panel-battery_svn.bb:PV = "0.1.1+svn${SVNREV}"
./openmoko-panel-plugins/openmoko-panel-bt_svn.bb:PV = "0.1.0+svn${SVNREV}"
./openmoko-panel-plugins/openmoko-panel-gps_svn.bb:PV = "0.1.0+svn${SVNREV}"
./openmoko-panel-plugins/openmoko-panel-memory_svn.bb:PV = "0.0.0+svn${SVNREV}"
./openmoko-panel-plugins/openmoko-panel-wifi_svn.bb:PV = "0.0.0+svn${SVNREV}"
./openmoko-panel-plugins/openmoko-panel-gsm_svn.bb:PV = "0.1.0+svn${SVNREV}"
./openmoko-panel-plugins/openmoko-panel-usb_svn.bb:PV = "0.1.0+svn${SVNREV}"
./openmoko-panel-plugins/openmoko-panel-clock_svn.bb:PV = "0.1.0+svn${SVNREV}"

42  "git.bb.*" | grep -v SRCREV (few with SRCDATE for git?)
./gnome/gnome-bluetooth_git.bb:PV = "2.28.1"
./libdlo/libdlo_git.bb:PV = "0.1.0"
./networkmanager/networkmanager_git.bb:PV = "0.7.1+git"
./networkmanager/cnetworkmanager_git.bb:PV = "0.8+git"
./networkmanager/netm-cli_git.bb:PV = "0.4+git"
./networkmanager/network-manager-applet_git.bb:PV = "0.7.1+git"
./cairo/cairo_git.bb:PV = "1.9.3"
./xorg-xserver/xserver-kdrive_git.bb:PV = "1.4+git${SRCDATE}"
./linux/linux-hackndev-2.6_git.bb:PV = "${K_MAJOR}.${K_MINOR}.${K_MICRO}-${HHV}"
./linux/linux-efika_2.6.21+git.bb:PV = "2.6.21+git${SRCDATE}"
./linux/linux-omap_git.bb:PV = "2.6.31"
./linux/linux-powerpc-fsl_git.bb:PV = "2.6.30"
./linux/linux-omap2_git.bb:PV = "2.6.26"
./kexecboot/kexecboot_git.bb:PV = "0.5"
./ekiga/ekiga_git.bb:PV = "3.2.6+git"
./moblin/nbtk_git.bb:PV = "0.8.0"
./moblin/hornsey_git.bb:PV = "0.0"
./moblin/bognor-regis_git.bb:PV = "0.4.1"
./moblin/bickley_git.bb:PV = "0.0"
./gtk-theme-torturer/gtk-theme-torturer_git.bb:PV = "0.0.0+git${SRCDATE}"
./matchbox-keyboard/mboxkbd-layouts-gui_git.bb:PV = "0.0+git5b42aeff36d930dc3a9b75eedc74dacfec45f43f"
./connman/connman-gnome_git.bb:PV = "0.5+git"
./connman/connman_git.bb:PV = "0.42+git"
./python/python-phoneutils_git.bb:PV = "0.0.2+gitr${SRCPV}"
./quake/quake3-pandora-gles_git.bb:PV = "0.0"
./gstreamer/gst-plugin-gles_git.bb:PV = "0.10"
./bluez/bluez-gnome_git.bb:PV = "0.10+git${SRCDATE}"
./mamona/mamona-input-methods_git.bb:PV = "0.1+git"
./mamona/mamonaim-e-applet_git.bb:PV = "0.1+git"
./xorg-lib/pixman_git.bb:PV = "0.17.1"
./packagekit/packagekit_git.bb:PV = "0.4.6+git"
./hal/hal_git.bb:PV = "0.5.9.1+git${SRCDATE}"
./hal/hal-info_git.bb:PV = "${SRCDATE}+git"
./xcb/xcb-demo_git.bb:PV = "0.1+git"
./xcb/libxcb_git.bb:PV = "1.0+git"
./xcb/xcb-util_git.bb:PV = "0.2+git"
./xcb/xcb-proto_git.bb:PV = "1.0+git"
./gphoto2/ptp-gadget_git.bb:PV = "1.1"
./clutter/clutter-gst-0.9_git.bb:PV = "0.9.0"
./clutter/clutter_0.8+git.bb:PV = "0.8.8"
./clutter/clutter-0.9_git.bb:PV = "1.1.0"
./geoclue/geoclue_git.bb:PV = "0.11.1"

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration

[Martin Jansa]

On Sun, Nov 15, 2009 at 5:36 PM, Martin Jansa <martin.jansa@gmail.com>wrote:

> As I sent earlier as reply to [PATCH] bitbake.conf: SRCPV always
> upgradeable
> paths for git recipes.
>
> First few changes are already in martin_jansa/srcpv branch. All changes to
> svn.bb
> recipes seems safe, so I would like to push them really soon. For git.bbchanges
> I need your advise/ack so please reply.
>
> Is there better way to create upgradeable path then bumping PE? Package
> names/.ipk filenames are a bit ugly with PE imho.
>
> Should I prepare one BIG patch for all recipes or commit them 1 by 1?
>
> XorA said that migrating all recipes at once in his xora/angstrom-srcpv
> branch was too much work to maintain and push. So I would like to push
> it directly to oe.dev or with really short lived tmp branch, if all
> agree that we should migrate all packages globally.
>

 Almost all changes for SRCPV (no SRCREV in my tree now, just few recipes
named *git.bb doesn't have SRCPV in PV yet..)
are in martin_jansa/srcpv branch please review them or even cherry-pick some
if you're maintainter of some changed recipes.

Thanks a lot

Regards

Re: SRCPV migration

[Koen Kooi]

On 15-11-09 17:36, Martin Jansa wrote:
> As I sent earlier as reply to [PATCH] bitbake.conf: SRCPV always upgradeable
> paths for git recipes.
>
> First few changes are already in martin_jansa/srcpv branch. All changes to svn.bb
> recipes seems safe, so I would like to push them really soon. For git.bb changes
> I need your advise/ack so please reply.
>
> Original e-mail:
>
> 1) svn recipes with +svnr{SRCREV} could be just replaced with
> +svnr{SRCPV} as the output should be the same
>
> 2) svn recipes with -r{SRCREV}, -svn{SRCREV}, -svnr{SRCREV}, +r{SRCREV},
> +svn{SRCREV} unified to +svnr{SRCPV}? I'm not sure if '+' will be sorted
> higher version wise, so do we need to bump PE?
>
> 3) git recipes with -git{SRCREV}, -gitr{SRCREV}, +git{SRCREV} unified to
> +gitr{SRCPV} with PE bump?

If something needs a PE bump, make sure you do it properly and bump PE 
on the non scm fetched entries as well. For the kernel I wouldn't bump 
PE, since too many machines can use different kernel recipes.

And can someone tell me how SRCPV works when you have multiple 
buildmachines across the world, how does one keep SRCPV in sync between 
all those machines?

How does SRCPV work when you have recipes for 1.2+gitrfoo, 1.3+gitrbar, etc?

IOW: will it create more work for me (like the proposed checksums 
changes) instead of making things easier?

regards,

Koen

Re: SRCPV migration

[Richard Purdie]

On Mon, 2009-11-16 at 09:38 +0100, Koen Kooi wrote:
> If something needs a PE bump, make sure you do it properly and bump PE 
> on the non scm fetched entries as well. For the kernel I wouldn't bump 
> PE, since too many machines can use different kernel recipes.
> 
> And can someone tell me how SRCPV works when you have multiple 
> buildmachines across the world, how does one keep SRCPV in sync between 
> all those machines?
> 
> How does SRCPV work when you have recipes for 1.2+gitrfoo, 1.3+gitrbar, etc?

They'd become 1.2+LOCALBUILDREV+gitrfoo and 1.3+LOCALBUILDREV+gitrfoo.

> IOW: will it create more work for me (like the proposed checksums 
> changes) instead of making things easier?

SRCPV solves one particular set of bugs where local build revisions were
only injected into revisions part of the time. This change makes things
more consistent and predicable in that its always injected.

Nobody has ever written code to share the bitbake persistent data
between machines. It shouldn't be that hard with some kind of database
server but nobody has done it.

It doesn't really change the position for Angstrom as I understand it
though as you don't (can't) use these packages now due to the multiple
build servers and that position hasn't changed :/.

Cheers,

Richard

Re: SRCPV migration

[Koen Kooi]

On 16-11-09 10:39, Richard Purdie wrote:
> On Mon, 2009-11-16 at 09:38 +0100, Koen Kooi wrote:
>> If something needs a PE bump, make sure you do it properly and bump PE
>> on the non scm fetched entries as well. For the kernel I wouldn't bump
>> PE, since too many machines can use different kernel recipes.
>>
>> And can someone tell me how SRCPV works when you have multiple
>> buildmachines across the world, how does one keep SRCPV in sync between
>> all those machines?
>>
>> How does SRCPV work when you have recipes for 1.2+gitrfoo, 1.3+gitrbar, etc?
>
> They'd become 1.2+LOCALBUILDREV+gitrfoo and 1.3+LOCALBUILDREV+gitrfoo.
>
>> IOW: will it create more work for me (like the proposed checksums
>> changes) instead of making things easier?
>
> SRCPV solves one particular set of bugs where local build revisions were
> only injected into revisions part of the time. This change makes things
> more consistent and predicable in that its always injected.
>
> Nobody has ever written code to share the bitbake persistent data
> between machines. It shouldn't be that hard with some kind of database
> server but nobody has done it.
>
> It doesn't really change the position for Angstrom as I understand it
> though as you don't (can't) use these packages now due to the multiple
> build servers and that position hasn't changed :/.

So basically every recipe that is using SRCPV in PV or PR is unsuitable 
to be put in online feeds. That should be enough to stop the SRCPV merge 
into .dev.

regards,

Koen

Re: SRCPV migration

[Holger Hans Peter Freyther]

On Monday 16 November 2009 10:39:16 Richard Purdie wrote:

> SRCPV solves one particular set of bugs where local build revisions were
> only injected into revisions part of the time. This change makes things
> more consistent and predicable in that its always injected.
> 
> Nobody has ever written code to share the bitbake persistent data
> between machines. It shouldn't be that hard with some kind of database
> server but nobody has done it.

Partially true, with the "git rev-list | wc -l" trick used at Openmoko we had 
a "global" number. The problem with the counting is if people use (rebase and 
such on the system...)

z.

Re: SRCPV migration

[Richard Purdie]

On Mon, 2009-11-16 at 11:37 +0100, Koen Kooi wrote:
> So basically every recipe that is using SRCPV in PV or PR is unsuitable 
> to be put in online feeds. That should be enough to stop the SRCPV merge 
> into .dev.

How is this different to the current situation though?

If SRCREV is locked down you will get 1 as the local build revision back
and it will not change and will be the same for everyone.

If its not locked down, all bets are off but they always have been - no
change.

Cheers,

Richard

Re: SRCPV migration

[Koen Kooi]

On 16-11-09 11:49, Richard Purdie wrote:
> On Mon, 2009-11-16 at 11:37 +0100, Koen Kooi wrote:
>> So basically every recipe that is using SRCPV in PV or PR is unsuitable
>> to be put in online feeds. That should be enough to stop the SRCPV merge
>> into .dev.
>
> How is this different to the current situation though?

In the current situation (actually, last weeks situation) SRCPV was 
never used.

> If SRCREV is locked down you will get 1 as the local build revision back
> and it will not change and will be the same for everyone.

> If its not locked down, all bets are off but they always have been - no
> change.

Ah, that was the bit of info I was missing. But if SRCREV is locked down 
(as it should!) what's the point of putting SRCPV in PV or PR? It seems 
to make things worse if you update a locked SRCREV, since SRCPV will 
remain '1'.

regards,

Koen

Re: SRCPV migration

[Richard Purdie]

On Mon, 2009-11-16 at 11:59 +0100, Koen Kooi wrote:
> On 16-11-09 11:49, Richard Purdie wrote:
> > On Mon, 2009-11-16 at 11:37 +0100, Koen Kooi wrote:
> >> So basically every recipe that is using SRCPV in PV or PR is unsuitable
> >> to be put in online feeds. That should be enough to stop the SRCPV merge
> >> into .dev.
> >
> > How is this different to the current situation though?
> 
> In the current situation (actually, last weeks situation) SRCPV was 
> never used.
> 
> > If SRCREV is locked down you will get 1 as the local build revision back
> > and it will not change and will be the same for everyone.
> 
> > If its not locked down, all bets are off but they always have been - no
> > change.
> 
> Ah, that was the bit of info I was missing. But if SRCREV is locked down 
> (as it should!) what's the point of putting SRCPV in PV or PR? It seems 
> to make things worse if you update a locked SRCREV, since SRCPV will 
> remain '1'.

How does Angstrom currently solve this problem? You bump the SRCREV and
override PV manually?

Angstrom would probably need a policy of wiping out the persistent data
DB so it didn't auto increase and then the situation doesn't change (for
Angstrom).

The benefit of this change is that local builds start automatically
working for people with fixed or floating SRCREV (and allows easily
switching between them) and people generating feeds from a single build
machine also benefit.

What we really need is a way to force the "local" build revisions for
the likes of Angstrom, I'll keep that in mind for the next bitbake
release...

Cheers,

Richard

Re: SRCPV migration

[Martin Jansa]

On Mon, Nov 16, 2009 at 11:59:10AM +0100, Koen Kooi wrote:
> On 16-11-09 11:49, Richard Purdie wrote:
> >On Mon, 2009-11-16 at 11:37 +0100, Koen Kooi wrote:
> >>So basically every recipe that is using SRCPV in PV or PR is unsuitable
> >>to be put in online feeds. That should be enough to stop the SRCPV merge
> >>into .dev.
> >
> >How is this different to the current situation though?
> 
> In the current situation (actually, last weeks situation) SRCPV was
> never used.
> 
> >If SRCREV is locked down you will get 1 as the local build revision back
> >and it will not change and will be the same for everyone.
> 
> >If its not locked down, all bets are off but they always have been - no
> >change.
> 
> Ah, that was the bit of info I was missing. But if SRCREV is locked
> down (as it should!) what's the point of putting SRCPV in PV or PR?
> It seems to make things worse if you update a locked SRCREV, since
> SRCPV will remain '1'.
> 
> regards,
> 
> Koen

Hi,

I'm not sure if this is 100% right but it behaves like this here:

with AUTOREV used as SRCREV you get ${SRCREV} in format
gitrNNNN+hash
where NNNN is localcount incremented on buildhost every time you build
that recipe AND hash is changed

if you have fixed SRCREV then without SRCPV you get just gitrHash which 
can be < gitrNNNN+hash or > gitrNNNN+hash depending on actual hash value.
That's why you cannot switch between SRCREV and SRCPV without PE bump.

If there is SRCPV in PV and you're using fixed SRCREV than you don't
need to bump PR in cases like this 
./xorg-proto/calibrateproto_git.bb:PV = "0.0+${PR}+gitr${SRCPV}"
because +gitr${SRCPV} will create non-decreasing sequence as localcount
will be incremented every time you bump fixed SRCREV for that recipe AND
build it.

You're right that there can be inconsistency between builders if one ie
you're changing fixed srcrev every day and rebuild it on that buildhost
and othere buildhost is rebuilding that package once a month.

If all your builders are building all your targets every day and you're
not changing fixed SRCREV more then once a day, you need to sync
bb_persist_data.sqlite3 only for new builder or after some failures
(which I know is quite error-prone :().

This inconsistency can be fixed imho only by sharing 
tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite3 between hosts
(better using some online database access for getting localcounts)

sqlite3 ../tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite3 
"select * from BB_URI_LOCALCOUNT where key like '%kernel.git-linux-openmoko-shr-drm-devel_count';"
git:git.openmoko.org.git.kernel.git-linux-openmoko-shr-drm-devel_count|5

regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration

[Koen Kooi]

On 16-11-09 12:39, Richard Purdie wrote:
> On Mon, 2009-11-16 at 11:59 +0100, Koen Kooi wrote:
>> On 16-11-09 11:49, Richard Purdie wrote:
>>> On Mon, 2009-11-16 at 11:37 +0100, Koen Kooi wrote:
>>>> So basically every recipe that is using SRCPV in PV or PR is unsuitable
>>>> to be put in online feeds. That should be enough to stop the SRCPV merge
>>>> into .dev.
>>>
>>> How is this different to the current situation though?
>>
>> In the current situation (actually, last weeks situation) SRCPV was
>> never used.
>>
>>> If SRCREV is locked down you will get 1 as the local build revision back
>>> and it will not change and will be the same for everyone.
>>
>>> If its not locked down, all bets are off but they always have been - no
>>> change.
>>
>> Ah, that was the bit of info I was missing. But if SRCREV is locked down
>> (as it should!) what's the point of putting SRCPV in PV or PR? It seems
>> to make things worse if you update a locked SRCREV, since SRCPV will
>> remain '1'.
>
> How does Angstrom currently solve this problem? You bump the SRCREV and
> override PV manually?

With 'angstrom' you mean 'oe', right? If a SRCREV gets updated the 
person doing that checks if PV or PR need to get changed to make it sort 
higher (or lower, depending on the change). I don't see how SRCPV will 
make life or less error-prone. By the looks of it it only makes things 
worse.

regards,

Koen

>
> Angstrom would probably need a policy of wiping out the persistent data
> DB so it didn't auto increase and then the situation doesn't change (for
> Angstrom).
>
> The benefit of this change is that local builds start automatically
> working for people with fixed or floating SRCREV (and allows easily
> switching between them) and people generating feeds from a single build
> machine also benefit.
>
> What we really need is a way to force the "local" build revisions for
> the likes of Angstrom, I'll keep that in mind for the next bitbake
> release...
>
> Cheers,
>
> Richard

Re: SRCPV migration

[Koen Kooi]

On 16-11-09 12:51, Martin Jansa wrote:
> On Mon, Nov 16, 2009 at 11:59:10AM +0100, Koen Kooi wrote:
>> On 16-11-09 11:49, Richard Purdie wrote:
>>> On Mon, 2009-11-16 at 11:37 +0100, Koen Kooi wrote:
>>>> So basically every recipe that is using SRCPV in PV or PR is unsuitable
>>>> to be put in online feeds. That should be enough to stop the SRCPV merge
>>>> into .dev.
>>>
>>> How is this different to the current situation though?
>>
>> In the current situation (actually, last weeks situation) SRCPV was
>> never used.
>>
>>> If SRCREV is locked down you will get 1 as the local build revision back
>>> and it will not change and will be the same for everyone.
>>
>>> If its not locked down, all bets are off but they always have been - no
>>> change.
>>
>> Ah, that was the bit of info I was missing. But if SRCREV is locked
>> down (as it should!) what's the point of putting SRCPV in PV or PR?
>> It seems to make things worse if you update a locked SRCREV, since
>> SRCPV will remain '1'.
>>
>> regards,
>>
>> Koen
>
> Hi,
>
> I'm not sure if this is 100% right but it behaves like this here:
>
> with AUTOREV used

Which angstrom tries to avoid at great lenghts and any OE recipe fetched 
from an scm will by default and policy have a locked down rev available.
A user (or distro) has to explicitly enable AUTOREV for recipes.

> as SRCREV you get ${SRCREV} in format
> gitrNNNN+hash
> where NNNN is localcount incremented on buildhost every time you build
> that recipe AND hash is changed
> if you have fixed SRCREV then without SRCPV you get just gitrHash which
> can be<  gitrNNNN+hash or>  gitrNNNN+hash depending on actual hash value.
> That's why you cannot switch between SRCREV and SRCPV without PE bump.

I'm extremely allergic to PE bumps, because it means we screwed up big time.

> If there is SRCPV in PV and you're using fixed SRCREV than you don't
> need to bump PR in cases like this
> ./xorg-proto/calibrateproto_git.bb:PV = "0.0+${PR}+gitr${SRCPV}"
> because +gitr${SRCPV} will create non-decreasing sequence as localcount
> will be incremented every time you bump fixed SRCREV for that recipe AND
> build it.

So, how do I say

PREFERRED_VERSION_calibrateproto = 0.0+r1+gitr483843874931943189393 # 
(4838... is what's in sane-srcrevs.inc)

with SRCPV? It seems I would loose the ability to do that 'globally' 
since SRCPV is tied to a buildhost.

> You're right that there can be inconsistency between builders if one ie
> you're changing fixed srcrev every day and rebuild it on that buildhost
> and othere buildhost is rebuilding that package once a month.
>
> If all your builders are building all your targets every day and you're
> not changing fixed SRCREV more then once a day, you need to sync
> bb_persist_data.sqlite3 only for new builder or after some failures
> (which I know is quite error-prone :().

So SRCPV makes things worse than the current situation.

> This inconsistency can be fixed imho only by sharing
> tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite3 between hosts
> (better using some online database access for getting localcounts)
>
> sqlite3 ../tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite3
> "select * from BB_URI_LOCALCOUNT where key like '%kernel.git-linux-openmoko-shr-drm-devel_count';"
> git:git.openmoko.org.git.kernel.git-linux-openmoko-shr-drm-devel_count|5

So it seems that using SRCPV in PV and PR shouldn't be allowed till 
there is *good* way to share the 'localcount' to all builders.

regards,

Koen

Re: SRCPV migration

[Richard Purdie]

On Mon, 2009-11-16 at 13:10 +0100, Koen Kooi wrote:
> > How does Angstrom currently solve this problem? You bump the SRCREV
> > and override PV manually? 
> 
> With 'angstrom' you mean 'oe', right? If a SRCREV gets updated the 
> person doing that checks if PV or PR need to get changed to make it sort 
> higher (or lower, depending on the change). I don't see how SRCPV will 
> make life or less error-prone. By the looks of it it only makes things 
> worse.

I meant Angstrom which looks like it relies on anyone doing the updating
to OE in general to get this right :/. As I'm sure you're aware this is
less than ideal.

You have a point in that we can't lock down the local build revisions
and this causes a problem with the distributed nature of Angstrom's
builds.

I think we will have to hold off some of the SRCPV migration until
bitbake has some kind of lock down functionality for the local build
numbers. Any volunteers to write a patch?

Note that converting svn recipes can go ahead as they don't need local
build revisions, its mainly the distributed SCMS that have a problem
(e.g. git/hg).

Cheers,

Richard

Re: SRCPV migration

[Martin Jansa]

On Mon, Nov 16, 2009 at 01:19:15PM +0100, Koen Kooi wrote:
> On 16-11-09 12:51, Martin Jansa wrote:
> >On Mon, Nov 16, 2009 at 11:59:10AM +0100, Koen Kooi wrote:
> >>On 16-11-09 11:49, Richard Purdie wrote:
> >>>On Mon, 2009-11-16 at 11:37 +0100, Koen Kooi wrote:
> >>>>So basically every recipe that is using SRCPV in PV or PR is unsuitable
> >>>>to be put in online feeds. That should be enough to stop the SRCPV merge
> >>>>into .dev.
> >>>
> >>>How is this different to the current situation though?
> >>
> >>In the current situation (actually, last weeks situation) SRCPV was
> >>never used.
> >>
> >>>If SRCREV is locked down you will get 1 as the local build revision back
> >>>and it will not change and will be the same for everyone.
> >>
> >>>If its not locked down, all bets are off but they always have been - no
> >>>change.
> >>
> >>Ah, that was the bit of info I was missing. But if SRCREV is locked
> >>down (as it should!) what's the point of putting SRCPV in PV or PR?
> >>It seems to make things worse if you update a locked SRCREV, since
> >>SRCPV will remain '1'.
> >>
> >>regards,
> >>
> >>Koen
> >
> >Hi,
> >
> >I'm not sure if this is 100% right but it behaves like this here:
> >
> >with AUTOREV used
> 
> Which angstrom tries to avoid at great lenghts and any OE recipe
> fetched from an scm will by default and policy have a locked down
> rev available.
> A user (or distro) has to explicitly enable AUTOREV for recipes.

Sure, but AUTOREV is still usefull for recipes which are ie distro
specific (ie shr telephony stack) which is still developing pretty fast
and no one would like to see SRCREV bumps twice a day just because
developers are still fixing bugs in that package and users want bleeding
edge version for their distribution.

But once development slows down, would be nice to easily switch to fixed
SRCREV with upgradeable PV.

> >as SRCREV you get ${SRCREV} in format
> >gitrNNNN+hash
> >where NNNN is localcount incremented on buildhost every time you build
> >that recipe AND hash is changed
> >if you have fixed SRCREV then without SRCPV you get just gitrHash which
> >can be<  gitrNNNN+hash or>  gitrNNNN+hash depending on actual hash value.
> >That's why you cannot switch between SRCREV and SRCPV without PE bump.
> 
> I'm extremely allergic to PE bumps, because it means we screwed up big time.
> 
> >If there is SRCPV in PV and you're using fixed SRCREV than you don't
> >need to bump PR in cases like this
> >./xorg-proto/calibrateproto_git.bb:PV = "0.0+${PR}+gitr${SRCPV}"
> >because +gitr${SRCPV} will create non-decreasing sequence as localcount
> >will be incremented every time you bump fixed SRCREV for that recipe AND
> >build it.
> 
> So, how do I say
> 
> PREFERRED_VERSION_calibrateproto = 0.0+r1+gitr483843874931943189393
> # (4838... is what's in sane-srcrevs.inc)

This is imho wrong by design as you need to update SRCREV in sane-srcrevs 
or recipe then bump PR and then check if there is
PREFERRED_VERSION_calibrateproto for any distribution even outside OE
repo, because otherwise preferred version won't exist anymore and maybe 
even lower version will be picked (ie if there is DEFAULT_PREFERENCE = "-1" 
in that git recipe)?

That's why I proposed wildcard character to PREFERRED_VERSION before in 
http://patchwork.openembedded.org/patch/1057/

BTW: what about using something like this
PV = "0.0+git"
PR = "${INC_PR}.1+gitr${SRCPV}"

would be easy to set PREFERRED_VERSION to constant string "0.0+git"

> with SRCPV? It seems I would loose the ability to do that 'globally'
> since SRCPV is tied to a buildhost.
> 
> >You're right that there can be inconsistency between builders if one ie
> >you're changing fixed srcrev every day and rebuild it on that buildhost
> >and othere buildhost is rebuilding that package once a month.
> >
> >If all your builders are building all your targets every day and you're
> >not changing fixed SRCREV more then once a day, you need to sync
> >bb_persist_data.sqlite3 only for new builder or after some failures
> >(which I know is quite error-prone :().
> 
> So SRCPV makes things worse than the current situation.

switching AUTOREV <-> fixed SRCREV is not possible at all now, you even
need PE bump when switching, which is IMHO worse.

> >This inconsistency can be fixed imho only by sharing
> >tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite3 between hosts
> >(better using some online database access for getting localcounts)
> >
> >sqlite3 ../tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite3
> >"select * from BB_URI_LOCALCOUNT where key like '%kernel.git-linux-openmoko-shr-drm-devel_count';"
> >git:git.openmoko.org.git.kernel.git-linux-openmoko-shr-drm-devel_count|5
> 
> So it seems that using SRCPV in PV and PR shouldn't be allowed till
> there is *good* way to share the 'localcount' to all builders.

I like RP's idea about dropping localcount for distros never using
AUTOREV. Only change for you then would be gitr0+Hash instead gitrHash
which isn't that bad, is it?

> regards,

regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration

[Koen Kooi]

On 16-11-09 13:37, Richard Purdie wrote:
> On Mon, 2009-11-16 at 13:10 +0100, Koen Kooi wrote:
>>> How does Angstrom currently solve this problem? You bump the SRCREV
>>> and override PV manually?
>>
>> With 'angstrom' you mean 'oe', right? If a SRCREV gets updated the
>> person doing that checks if PV or PR need to get changed to make it sort
>> higher (or lower, depending on the change). I don't see how SRCPV will
>> make life or less error-prone. By the looks of it it only makes things
>> worse.
>
> I meant Angstrom which looks like it relies on anyone doing the updating
> to OE in general to get this right :/. As I'm sure you're aware this is
> less than ideal.

I used to spend a non-trivial amount of time cleaning up such things, 
but since I stopped caring about various recipes it isn't such a problem 
anymore.

> You have a point in that we can't lock down the local build revisions
> and this causes a problem with the distributed nature of Angstrom's
> builds.

Note that 'distributed' can mean something as simple as laptop + 
desktop. Or ubuntu-vm +fedora-vm. Or even "I wiped TMPDIR this morning".

> I think we will have to hold off some of the SRCPV migration until
> bitbake has some kind of lock down functionality for the local build
> numbers.

It would at least need a lockdown feature and a way to share 
changes/updates that is firewall friendly (basically git or http in my 
case).
I don't think forcing every distro to setup and manage their own 
database server just to make life easier for AUTOREV git users is worth 
it. I'm not saying we should make life harder for them, but screwing 
over everyone for what I would call a niche usecase (autorev users 
caring about upgradepaths) goes too far.
The current SRCREV situation sucks, but SRCPV only helps with making the 
packagemanager see that your change sorts higher and will get installed 
when upgrading. It does *NOT* solve any of the other issues, like 
needing to change PV if the new revision increased the upstream version 
(e.g. going from 2.6.31 to 2.6.32 for kernels).

> Any volunteers to write a patch?

I would guess the AUTOREV people would.

regards,

Koen

Re: SRCPV migration

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Mon, Nov 16, 2009 at 12:37:02PM +0000, Richard Purdie wrote:
> On Mon, 2009-11-16 at 13:10 +0100, Koen Kooi wrote:
> I think we will have to hold off some of the SRCPV migration until
> bitbake has some kind of lock down functionality for the local build
> numbers. Any volunteers to write a patch?

This could be enough?

Just putting
BB_GIT_LOCALCOUNT_FOR_SRCREV = "0"
somewhere (local.conf/distro.conf) and _count will always stay on 0
(gitr0+abc1234def)

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

[-- Attachment #2: 0001-Lock-down-LOCALCOUNT-if-you-need-consistent-SRCPV-be.patch --]
[-- Type: text/plain, Size: 1807 bytes --]

From 9c9cc27acde3d7fe059c8bae79b958180f8247d8 Mon Sep 17 00:00:00 2001
From: Martin Jansa <Martin.Jansa@gmail.com>
Date: Mon, 16 Nov 2009 14:36:34 +0100
Subject: [PATCH] Lock down LOCALCOUNT if you need consistent SRCPV between multiple buildhosts

---
 lib/bb/fetch/__init__.py |    3 ++-
 lib/bb/fetch/git.py      |    3 +++
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 8c0d7ea..0b9a052 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -537,6 +537,7 @@ class Fetch(object):
         
         """
         has_want_sortable = hasattr(self, "_want_sortable_revision")
+        has_want_increasing_localcount = hasattr(self, "_want_increasing_localcount")
         has_sortable = hasattr(self, "_sortable_revision")
 
         if not has_want_sortable and has_sortable:
@@ -555,7 +556,7 @@ class Fetch(object):
         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
 
-        if count is None:
+        if not has_want_increasing_localcount or count is None:
             count = "0"
         else:
             count = str(int(count) + 1)
diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py
index 5cdf656..e4d0156 100644
--- a/lib/bb/fetch/git.py
+++ b/lib/bb/fetch/git.py
@@ -149,6 +149,9 @@ class Git(Fetch):
     def _want_sortable_revision(self, url, ud, d):
         return bb.data.getVar("BB_GIT_CLONE_FOR_SRCREV", d, True) or False
 
+    def _want_increasing_localcount(self, url, ud, d):
+        return bb.data.getVar("BB_GIT_LOCALCOUNT_FOR_SRCREV", d, False) or True
+
     def _sortable_revision(self, url, ud, d):
         """
         This is only called when _want_sortable_revision called true
-- 
1.6.5.2

Re: SRCPV migration

[Richard Purdie]

On Mon, 2009-11-16 at 14:43 +0100, Martin Jansa wrote:
> On Mon, Nov 16, 2009 at 12:37:02PM +0000, Richard Purdie wrote:
> > On Mon, 2009-11-16 at 13:10 +0100, Koen Kooi wrote:
> > I think we will have to hold off some of the SRCPV migration until
> > bitbake has some kind of lock down functionality for the local build
> > numbers. Any volunteers to write a patch?
> 
> This could be enough?
> 
> Just putting
> BB_GIT_LOCALCOUNT_FOR_SRCREV = "0"
> somewhere (local.conf/distro.conf) and _count will always stay on 0
> (gitr0+abc1234def)

No, we need to be able to control the version and it shouldn't be git
specific...

Cheers,

Richard

Re: SRCPV migration

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1899 bytes --]

On Mon, Nov 16, 2009 at 01:55:48PM +0000, Richard Purdie wrote:
> On Mon, 2009-11-16 at 14:43 +0100, Martin Jansa wrote:
> > On Mon, Nov 16, 2009 at 12:37:02PM +0000, Richard Purdie wrote:
> > > On Mon, 2009-11-16 at 13:10 +0100, Koen Kooi wrote:
> > > I think we will have to hold off some of the SRCPV migration until
> > > bitbake has some kind of lock down functionality for the local build
> > > numbers. Any volunteers to write a patch?
> > 
> > This could be enough?
> > 
> > Just putting
> > BB_GIT_LOCALCOUNT_FOR_SRCREV = "0"
> > somewhere (local.conf/distro.conf) and _count will always stay on 0
> > (gitr0+abc1234def)
> 
> No, we need to be able to control the version and it shouldn't be git
> specific...
> 
> Cheers,
> 
> Richard

OK better version

with this you can add

LOCALCOUNT_pn-bar ?= "4"
to ie sane-srcrevs.inc

or

LOCALCOUNT ?= "4" to 
recipes/foo/bar_git.bb

(btw seems like LOCALCOUNT_pn-bar has higher preferrence even if I use
"LOCALCOUNT = 4" in recipe, why?)

And it will be ignored for all distros where BB_LOCALCOUNT_OVERRIDE is
not set. 

With BB_LOCALCOUNT_OVERRIDE enabled, you can use LOCALCOUNT instead of
PR bump if you just change SRCREV, for others will be LOCALCOUNT in
SRCPV incremented by default.

If BB_GIT_CLONE_FOR_SRCREV is set than LOCALCOUNT is ALWAYS set to 
"git list-rev | wc -l" which could be considered also as consistent PV
scheme for multiple buildhosts.

BTW: SRCPV seems to be expanded in PV a bit sooner than SRCREV was,
which with combination with BB_GIT_CLONE_FOR_SRCREV means that all git
repositories are checked during recipe parsing (Klaus Kurzmann has a
list of git repositories used in OE recipes which are not accessible)

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

[-- Attachment #2: 0001-Optional-LOCALCOUNT-for-recipe.patch --]
[-- Type: text/plain, Size: 2680 bytes --]

From dcb8732640c031ab86150deb4933f1a87501c78b Mon Sep 17 00:00:00 2001
From: Martin Jansa <Martin.Jansa@gmail.com>
Date: Tue, 17 Nov 2009 08:24:52 +0100
Subject: [PATCH 1/2] Optional LOCALCOUNT for recipe

* Instead of autoincrement from persistent cache when srcrev is changed.
* Should be used by distributions with multiple builders, where consistent
  PV is needed.
* Can be used instead of PR bump in PVs like this "0.0+${PR}+gitr${SRCPV}"
---
 lib/bb/fetch/__init__.py |   36 +++++++++++++++++++++++++++++++-----
 1 files changed, 31 insertions(+), 5 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 8c0d7ea..3b527e2 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -448,6 +448,27 @@ class Fetch(object):
 
     srcrev_internal_helper = staticmethod(srcrev_internal_helper)
 
+    def localcount_internal_helper(ud, d):
+        """
+        Return:
+            a) a locked localcount if specified
+	    b) None otherwise
+        """
+
+        localcount= None
+        if 'name' in ud.parm:
+            pn = data.getVar("PN", d, 1)
+            localcount = data.getVar("LOCALCOUNT_pn-" + pn + "_" + ud.parm['name'], d, 1)
+        if not localcount:
+            localcount = data.getVar("LOCALCOUNT", d, 1)
+        if localcount == "INVALID":
+            raise InvalidSRCREV("Please set LOCALCOUNT to a valid value")
+        if not localcount:
+            return None
+        return localcount
+
+    localcount_internal_helper = staticmethod(localcount_internal_helper)
+
     def try_mirror(d, tarfn):
         """
         Try to use a mirrored version of the sources. We do this
@@ -550,15 +571,20 @@ class Fetch(object):
 
         latest_rev = self._build_revision(url, ud, d)
         last_rev = pd.getValue("BB_URI_LOCALCOUNT", key + "_rev")
-        count = pd.getValue("BB_URI_LOCALCOUNT", key + "_count")
+        localcount = Fetch.localcount_internal_helper(ud, d)
+        if localcount is None:
+            count = pd.getValue("BB_URI_LOCALCOUNT", key + "_count")
+        else:
+            count = localcount
 
         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
 
-        if count is None:
-            count = "0"
-        else:
-            count = str(int(count) + 1)
+        if localcount is None:
+            if count is None:
+                count = "0"
+            else:
+                count = str(int(count) + 1)
 
         pd.setValue("BB_URI_LOCALCOUNT", key + "_rev", latest_rev)
         pd.setValue("BB_URI_LOCALCOUNT", key + "_count", count)
-- 
1.6.5.2


[-- Attachment #3: 0002-BB_LOCALCOUNT_OVERRIDE-to-enable-setting-LOCALCOUNT-.patch --]
[-- Type: text/plain, Size: 1631 bytes --]

From 1b010656ae8a77a6a5ce12d8ef7ec2a8ef628704 Mon Sep 17 00:00:00 2001
From: Martin Jansa <Martin.Jansa@gmail.com>
Date: Tue, 17 Nov 2009 09:11:57 +0100
Subject: [PATCH 2/2] BB_LOCALCOUNT_OVERRIDE to enable setting LOCALCOUNT for recipe

* This way LOCALCOUNTs can be specified directly in recipes instead of
  separated distro config (as not all want to use them). And will be
  used only when BB_LOCALCOUNT_OVERRIDE set in distro config.
---
 lib/bb/fetch/__init__.py |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 3b527e2..9e50de6 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -183,6 +183,9 @@ def checkstatus(d):
         if not ret:
             bb.msg.fatal(bb.msg.domain.Fetcher, "URL %s doesn't work" % u)
 
+def want_localcount_override(d):
+    return bb.data.getVar("BB_LOCALCOUNT_OVERRIDE", d, True) or False
+
 def localpaths(d):
     """
     Return a list of the local filenames, assuming successful fetch
@@ -571,7 +574,10 @@ class Fetch(object):
 
         latest_rev = self._build_revision(url, ud, d)
         last_rev = pd.getValue("BB_URI_LOCALCOUNT", key + "_rev")
-        localcount = Fetch.localcount_internal_helper(ud, d)
+        localcount = None
+        if want_localcount_override(d):
+            bb.msg.error(1, 'LOCALCOUNT override enabled')
+            localcount = Fetch.localcount_internal_helper(ud, d)
         if localcount is None:
             count = pd.getValue("BB_URI_LOCALCOUNT", key + "_count")
         else:
-- 
1.6.5.2

Re: SRCPV migration

[Phil Blundell]

On Tue, 2009-11-17 at 09:55 +0100, Martin Jansa wrote:
> (btw seems like LOCALCOUNT_pn-bar has higher preferrence even if I use
> "LOCALCOUNT = 4" in recipe, why?)

This is the way overrides always work.  See the bitbake manual:

http://bitbake.berlios.de/manual/ch02.html#id869082

p.

Re: SRCPV migration

[Martin Jansa]

On Mon, Nov 16, 2009 at 12:37:02PM +0000, Richard Purdie wrote:
> Note that converting svn recipes can go ahead as they don't need local
> build revisions, its mainly the distributed SCMS that have a problem
> (e.g. git/hg).

OK pushed first 4 patches, svn recipes should be converted now..

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration

[Richard Purdie]

On Tue, 2009-11-17 at 09:55 +0100, Martin Jansa wrote: 
> OK better version
> 
> with this you can add
> 
> LOCALCOUNT_pn-bar ?= "4"
> to ie sane-srcrevs.inc
> 
> or
> 
> LOCALCOUNT ?= "4" to 
> recipes/foo/bar_git.bb
> 
> (btw seems like LOCALCOUNT_pn-bar has higher preferrence even if I use
> "LOCALCOUNT = 4" in recipe, why?)

As Phil mentioned, OVERRIDES

> And it will be ignored for all distros where BB_LOCALCOUNT_OVERRIDE is
> not set. 
> 
> With BB_LOCALCOUNT_OVERRIDE enabled, you can use LOCALCOUNT instead of
> PR bump if you just change SRCREV, for others will be LOCALCOUNT in
> SRCPV incremented by default.
> 
> If BB_GIT_CLONE_FOR_SRCREV is set than LOCALCOUNT is ALWAYS set to 
> "git list-rev | wc -l" which could be considered also as consistent PV
> scheme for multiple buildhosts.

Thats fine.

> BTW: SRCPV seems to be expanded in PV a bit sooner than SRCREV was,
> which with combination with BB_GIT_CLONE_FOR_SRCREV means that all git
> repositories are checked during recipe parsing (Klaus Kurzmann has a
> list of git repositories used in OE recipes which are not accessible)

SRCPV and SRCREV would expand at the same time. BB_GIT_CLONE_FOR_SRCREV
will hit the network a lot more though a I don't think it gets cached as
nicely as SRCREV/PV does.

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 8c0d7ea..3b527e2 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -448,6 +448,27 @@ class Fetch(object):
 
     srcrev_internal_helper = staticmethod(srcrev_internal_helper)
 
+    def localcount_internal_helper(ud, d):
+        """
+        Return:
+            a) a locked localcount if specified
+           b) None otherwise
+        """
+
+        localcount= None
+        if 'name' in ud.parm: 
+            pn = data.getVar("PN", d, 1)
+            localcount = data.getVar("LOCALCOUNT_pn-" + pn + "_" + ud.parm['name'], d, 1)

Just check getVar("LOCALCOUNT_" + ud.parm['name'], d, 1)

and require the user to set LOCALCOUNT_name_pn-foo = "bar".

+        if not localcount:
+            localcount = data.getVar("LOCALCOUNT", d, 1)
+        if localcount == "INVALID":
+            raise InvalidSRCREV("Please set LOCALCOUNT to a valid
value")

No point in checking for "INVALID", thats a special catch for the SRCREV
code you copied :)

+        if not localcount:
+            return None

You can just remove the above two lines.

+        return localcount
+
+    localcount_internal_helper =
staticmethod(localcount_internal_helper)
+
     def try_mirror(d, tarfn):
         """
         Try to use a mirrored version of the sources. We do this
@@ -550,15 +571,20 @@ class Fetch(object):
 
         latest_rev = self._build_revision(url, ud, d)
         last_rev = pd.getValue("BB_URI_LOCALCOUNT", key + "_rev")
-        count = pd.getValue("BB_URI_LOCALCOUNT", key + "_count")
+        localcount = Fetch.localcount_internal_helper(ud, d)
+        if localcount is None:
+            count = pd.getValue("BB_URI_LOCALCOUNT", key + "_count")
+        else:
+            count = localcount

How about just doing:

uselocalcount = bb.data.getVar("BB_LOCALCOUNT_OVERRIDE", d, True) or False
count = None
if uselocalcount
         count = Fetch.localcount_internal_helper(ud, d)
if count is None:
         count = pd.getValue("BB_URI_LOCALCOUNT", key + "_count")


which combines you second patch too?

         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
 
-        if count is None:
-            count = "0"
-        else:
-            count = str(int(count) + 1)
+        if localcount is None:
+            if count is None:
+                count = "0"
+            else:
+                count = str(int(count) + 1)

There should be no need to do this?

Cheers,

Richard

Re: SRCPV migration

[mok]

On Tue, 17 Nov 2009, Martin Jansa wrote:

> BTW: SRCPV seems to be expanded in PV a bit sooner than SRCREV was,
> which with combination with BB_GIT_CLONE_FOR_SRCREV means that all git
> repositories are checked during recipe parsing (Klaus Kurzmann has a
> list of git repositories used in OE recipes which are not accessible)

this is the list of recipes, which are failing when enabling
BB_GIT_CLONE_FOR_SRCREV and using SRCPV... either because the SRC_URI is
not correct, or there is a srcrev set somewhere that does not exist
anymore...

recipes/avahi/mango-lassi_git.bb
recipes/clutter/clutter-gtk-0.6_git.bb
recipes/clutter/clutter-gtk_git.bb
recipes/clutter/clutter-gtk-0.8_git.bb
recipes/clutter/moblin-proto_git.bb
recipes/disko/disko_git.bb
recipes/gnuradio/gnuradio_git.bb
recipes/gstreamer/gst-omapfb_1.0.bb
recipes/gtk-theme-torturer/gtk-theme-torturer_git.bb
recipes/gtk-webcore/midori_git.bb
recipes/libgdbus/libgdbus_git.bb
recipes/linux/linux-davinci_git.bb
recipes/linux/linux-davinci_2.6.25.bb
recipes/linux/linux-davinci_2.6.27.bb
recipes/linux/linux-davinci_2.6.28.bb
recipes/linux/linux-davinci_2.6.30.bb
recipes/linux/linux-kirkwood_2.6.29.5.bb
recipes/linux/linux-kirkwood_2.6.30.5.bb
recipes/linux/linux-kirkwood_2.6.31.bb
recipes/linux/linux-neuros_git.bb
recipes/linux/linux-omap-pm_2.6.28.bb
recipes/linux/linux-omap-pm_2.6.29.bb
recipes/linux/linux-omap-pm_git.bb
recipes/linux/linux-omapzoom_git.bb
recipes/linux/linux-openmoko-2.6.24_git.bb
recipes/linux/linux-openmoko-2.6.28_git.bb
recipes/linux/linux-replicant_git.bb
recipes/linux/linux-xo_git.bb
recipes/linux/openezx-kernel_git.bb
recipes/madbufferfly/madbutterfly_git.bb
recipes/moblin/json-glib_git.bb
recipes/moblin/libccss_git.bb
recipes/moblin/librest_git.bb
recipes/moblin/moblin-menus_git.bb
recipes/moblin/mojito_git.bb
recipes/moblin/twitter-glib_git.bb
recipes/mux/mux_git.bb
recipes/neuros-public/neuros-app-photoalbum_git.bb
recipes/neuros-public/neuros-app-vplayer_git.bb
recipes/neuros-public/neuros-lib-gui_git.bb
recipes/neuros-public/neuros-lib-widgets_git.bb
recipes/neuros-public/neuros-mainmenu_git.bb
recipes/neuros-public/neuros-nwm_git.bb
recipes/neuros-public/neuros-qt-plugins_git.bb
recipes/openbsc/openbsc_git.bb
recipes/openmoko-projects/om-locations_git.bb
recipes/openmoko-projects/tichy_git.bb
recipes/toscoterm/toscoterm_git.bb
recipes/u-boot/u-boot-omap3_git.bb
recipes/vlc/vlc-davinci_0.8.6h.bb
recipes/x-load/x-load_git.bb
recipes/xorg-driver/xf86-video-omapfb_git.bb

-- 
Klaus 'mrmoku' Kurzmann

Re: SRCPV migration

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 179 bytes --]

Updated patch, thanks RP!

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

[-- Attachment #2: 0001-Optional-LOCALCOUNT-for-recipe.patch --]
[-- Type: text/plain, Size: 2609 bytes --]

From 55c8f06a1382dfb11214b6e12ac47b06fce412a4 Mon Sep 17 00:00:00 2001
From: Martin Jansa <Martin.Jansa@gmail.com>
Date: Tue, 17 Nov 2009 08:24:52 +0100
Subject: [PATCH] Optional LOCALCOUNT for recipe

* Instead of autoincrement from persistent cache when srcrev is changed.
* Should be used by distributions with multiple builders, where consistent
  PV is needed.
* Can be used instead of PR bump in PVs like this "0.0+${PR}+gitr${SRCPV}"
* BB_LOCALCOUNT_OVERRIDE to enable setting LOCALCOUNT for recipe
* This way LOCALCOUNTs can be specified directly in recipes instead of
  separated distro config (as not all want to use them). And will be
  used only when BB_LOCALCOUNT_OVERRIDE set in distro config.
---
 lib/bb/fetch/__init__.py |   26 +++++++++++++++++++++++++-
 1 files changed, 25 insertions(+), 1 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 8c0d7ea..9508908 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -448,6 +448,23 @@ class Fetch(object):
 
     srcrev_internal_helper = staticmethod(srcrev_internal_helper)
 
+    def localcount_internal_helper(ud, d):
+        """
+        Return:
+            a) a locked localcount if specified
+            b) None otherwise
+        """
+
+        localcount= None
+        if 'name' in ud.parm:
+            pn = data.getVar("PN", d, 1)
+            localcount = data.getVar("LOCALCOUNT_" + ud.parm['name'], d, 1)
+        if not localcount:
+            localcount = data.getVar("LOCALCOUNT", d, 1)
+        return localcount
+
+    localcount_internal_helper = staticmethod(localcount_internal_helper)
+
     def try_mirror(d, tarfn):
         """
         Try to use a mirrored version of the sources. We do this
@@ -550,13 +567,20 @@ class Fetch(object):
 
         latest_rev = self._build_revision(url, ud, d)
         last_rev = pd.getValue("BB_URI_LOCALCOUNT", key + "_rev")
-        count = pd.getValue("BB_URI_LOCALCOUNT", key + "_count")
+        uselocalcount = bb.data.getVar("BB_LOCALCOUNT_OVERRIDE", d, True) or False
+        count = None
+        if uselocalcount:
+            count = Fetch.localcount_internal_helper(ud, d)
+        if count is None:
+            count = pd.getValue("BB_URI_LOCALCOUNT", key + "_count")
 
         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
 
         if count is None:
             count = "0"
+        elif uselocalcount:
+            count = str(count)
         else:
             count = str(int(count) + 1)
 
-- 
1.6.5.2

Re: SRCPV migration

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Tue, Nov 17, 2009 at 11:18:02AM +0100, mok@mnet-online.de wrote:
> On Tue, 17 Nov 2009, Martin Jansa wrote:
> 
> > BTW: SRCPV seems to be expanded in PV a bit sooner than SRCREV was,
> > which with combination with BB_GIT_CLONE_FOR_SRCREV means that all git
> > repositories are checked during recipe parsing (Klaus Kurzmann has a
> > list of git repositories used in OE recipes which are not accessible)

Hopefully resolved with this bitbake patch

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

[-- Attachment #2: 0002-BB_GIT_CLONE_FOR_SRCREV-using-only-_sortable_buildnu.patch --]
[-- Type: text/plain, Size: 3974 bytes --]

From 6956d60446de01af655eb992cbe31d02c8bb2a21 Mon Sep 17 00:00:00 2001
From: Martin Jansa <Martin.Jansa@gmail.com>
Date: Tue, 17 Nov 2009 15:44:43 +0100
Subject: [PATCH 2/2] BB_GIT_CLONE_FOR_SRCREV using only _sortable_buildnumber() for known revision

---
 lib/bb/fetch/__init__.py |   15 +++++++--------
 lib/bb/fetch/git.py      |   32 ++++++--------------------------
 2 files changed, 13 insertions(+), 34 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 9508908..db3eb40 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -553,14 +553,8 @@ class Fetch(object):
         """
         
         """
-        has_want_sortable = hasattr(self, "_want_sortable_revision")
-        has_sortable = hasattr(self, "_sortable_revision")
-
-        if not has_want_sortable and has_sortable:
-            return self._sortable_revision(url, ud, d)
-        elif has_want_sortable and self._want_sortable_revision(url, ud, d) and has_sortable:
-            return self._sortable_revision(url, ud, d)
-        
+        has_want_sortable = hasattr(self, "_want_sortable_buildnumber")
+        has_sortable = hasattr(self, "_sortable_buildnumber")
 
         pd = persist_data.PersistData(d)
         key = self.generate_revision_key(url, ud, d)
@@ -577,6 +571,11 @@ class Fetch(object):
         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
 
+        if not has_want_sortable and has_sortable:
+            count = self._sortable_buildnumber(url, ud, d, latest_rev)
+        elif has_want_sortable and self._want_sortable_buildnumber(url, ud, d) and has_sortable:
+            count = self._sortable_buildnumber(url, ud, d, latest_rev)
+
         if count is None:
             count = "0"
         elif uselocalcount:
diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py
index 5cdf656..b5c4c0b 100644
--- a/lib/bb/fetch/git.py
+++ b/lib/bb/fetch/git.py
@@ -146,44 +146,24 @@ class Git(Fetch):
     def _build_revision(self, url, ud, d):
         return ud.tag
 
-    def _want_sortable_revision(self, url, ud, d):
+    def _want_sortable_buildnumber(self, url, ud, d):
         return bb.data.getVar("BB_GIT_CLONE_FOR_SRCREV", d, True) or False
 
-    def _sortable_revision(self, url, ud, d):
+    def _sortable_buildnumber(self, url, ud, d, rev):
         """
         This is only called when _want_sortable_revision called true
 
-        We will have to get the updated revision.
+        Latest revision is already known, we need only to count revisions in git repo.
         """
         gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
         repodir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
 
-        key = "GIT_CACHED_REVISION-%s-%s"  % (gitsrcname, ud.tag)
-        if bb.data.getVar(key, d):
-            return bb.data.getVar(key, d)
-
-
-        # Runtime warning on wrongly configured sources
-        if ud.tag == "1":
-            bb.msg.error(1, bb.msg.domain.Fetcher, "SRCREV is '1'. This indicates a configuration error of %s" % url)
-            return "0+1"
-
         cwd = os.getcwd()
 
-        # Check if we have the rev already
-        if not os.path.exists(repodir):
-            print "no repo"
-            self.go(None, ud, d)
-
         os.chdir(repodir)
-        if not self._contains_ref(ud.tag, d):
-            self.go(None, ud, d)
 
-        output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % ud.tag, d, quiet=True)
+        output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % rev, d, quiet=True)
         os.chdir(cwd)
 
-        sortable_revision = "%s+%s" % (output.split()[0], ud.tag)
-        bb.data.setVar(key, sortable_revision, d)
-        return sortable_revision
-        
-
+        sortable_buildnumber= "%s" % (output.split()[0])
+        return sortable_buildnumber
-- 
1.6.5.2

Re: SRCPV migration

[Henning Heinold]

On Tue, Nov 17, 2009 at 11:18:02AM +0100, mok@mnet-online.de wrote:
> On Tue, 17 Nov 2009, Martin Jansa wrote:
> 
> > BTW: SRCPV seems to be expanded in PV a bit sooner than SRCREV was,
> > which with combination with BB_GIT_CLONE_FOR_SRCREV means that all git
> > repositories are checked during recipe parsing (Klaus Kurzmann has a
> > list of git repositories used in OE recipes which are not accessible)
> 
> this is the list of recipes, which are failing when enabling
> BB_GIT_CLONE_FOR_SRCREV and using SRCPV... either because the SRC_URI is
> not correct, or there is a srcrev set somewhere that does not exist
> anymore...
> 
> recipes/avahi/mango-lassi_git.bb
> recipes/clutter/clutter-gtk-0.6_git.bb
> recipes/clutter/clutter-gtk_git.bb
> recipes/clutter/clutter-gtk-0.8_git.bb
> recipes/clutter/moblin-proto_git.bb
> recipes/disko/disko_git.bb
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Ah thanks I will take care about disko.

Bye Henning

Re: SRCPV migration

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

On Tue, Nov 17, 2009 at 04:12:12PM +0100, Martin Jansa wrote:
> On Tue, Nov 17, 2009 at 11:18:02AM +0100, mok@mnet-online.de wrote:
> > On Tue, 17 Nov 2009, Martin Jansa wrote:
> > 
> > > BTW: SRCPV seems to be expanded in PV a bit sooner than SRCREV was,
> > > which with combination with BB_GIT_CLONE_FOR_SRCREV means that all git
> > > repositories are checked during recipe parsing (Klaus Kurzmann has a
> > > list of git repositories used in OE recipes which are not accessible)
> 
> Hopefully resolved with this bitbake patch

Sorry previous patch used localcount even for svn recipes, that was
wrong.

This version should be ok.

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

[-- Attachment #2: 0002-BB_GIT_CLONE_FOR_SRCREV-using-only-_sortable_buildnu.patch --]
[-- Type: text/plain, Size: 4086 bytes --]

From 572f9bd1f6f0cf43f5d0bd9c419d6024b8f378f4 Mon Sep 17 00:00:00 2001
From: Martin Jansa <Martin.Jansa@gmail.com>
Date: Tue, 17 Nov 2009 15:44:43 +0100
Subject: [PATCH 2/2] BB_GIT_CLONE_FOR_SRCREV using only _sortable_buildnumber() for known revision

---
 lib/bb/fetch/__init__.py |   15 ++++++++-------
 lib/bb/fetch/git.py      |   34 +++++++---------------------------
 2 files changed, 15 insertions(+), 34 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 9508908..cc095e3 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -553,15 +553,13 @@ class Fetch(object):
         """
         
         """
-        has_want_sortable = hasattr(self, "_want_sortable_revision")
-        has_sortable = hasattr(self, "_sortable_revision")
+        has_want_sortable_buildnumber = hasattr(self, "_want_sortable_buildnumber")
+        has_sortable_buildnumber = hasattr(self, "_sortable_buildnumber")
+        has_sortable_revision = hasattr(self, "_sortable_revision")
 
-        if not has_want_sortable and has_sortable:
+        if has_sortable_revision:
             return self._sortable_revision(url, ud, d)
-        elif has_want_sortable and self._want_sortable_revision(url, ud, d) and has_sortable:
-            return self._sortable_revision(url, ud, d)
-        
-
+    
         pd = persist_data.PersistData(d)
         key = self.generate_revision_key(url, ud, d)
 
@@ -577,6 +575,9 @@ class Fetch(object):
         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
 
+        if has_sortable_buildnumber and has_want_sortable_buildnumber and self._want_sortable_buildnumber(url, ud, d):
+            count = self._sortable_buildnumber(url, ud, d, latest_rev)
+
         if count is None:
             count = "0"
         elif uselocalcount:
diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py
index 5cdf656..b98ac7e 100644
--- a/lib/bb/fetch/git.py
+++ b/lib/bb/fetch/git.py
@@ -146,44 +146,24 @@ class Git(Fetch):
     def _build_revision(self, url, ud, d):
         return ud.tag
 
-    def _want_sortable_revision(self, url, ud, d):
+    def _want_sortable_buildnumber(self, url, ud, d):
         return bb.data.getVar("BB_GIT_CLONE_FOR_SRCREV", d, True) or False
 
-    def _sortable_revision(self, url, ud, d):
+    def _sortable_buildnumber(self, url, ud, d, rev):
         """
-        This is only called when _want_sortable_revision called true
+        This is only called when _want_sortable_buildnumber called true
 
-        We will have to get the updated revision.
+        Latest revision is already known, we need only to count revisions in git repo.
         """
         gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
         repodir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
 
-        key = "GIT_CACHED_REVISION-%s-%s"  % (gitsrcname, ud.tag)
-        if bb.data.getVar(key, d):
-            return bb.data.getVar(key, d)
-
-
-        # Runtime warning on wrongly configured sources
-        if ud.tag == "1":
-            bb.msg.error(1, bb.msg.domain.Fetcher, "SRCREV is '1'. This indicates a configuration error of %s" % url)
-            return "0+1"
-
         cwd = os.getcwd()
 
-        # Check if we have the rev already
-        if not os.path.exists(repodir):
-            print "no repo"
-            self.go(None, ud, d)
-
         os.chdir(repodir)
-        if not self._contains_ref(ud.tag, d):
-            self.go(None, ud, d)
 
-        output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % ud.tag, d, quiet=True)
+        output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % rev, d, quiet=True)
         os.chdir(cwd)
 
-        sortable_revision = "%s+%s" % (output.split()[0], ud.tag)
-        bb.data.setVar(key, sortable_revision, d)
-        return sortable_revision
-        
-
+        sortable_buildnumber= "%s" % (output.split()[0])
+        return sortable_buildnumber
-- 
1.6.5.2

Re: SRCPV migration

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]

On Tue, Nov 17, 2009 at 05:23:24PM +0100, Martin Jansa wrote:
> On Tue, Nov 17, 2009 at 04:12:12PM +0100, Martin Jansa wrote:
> > On Tue, Nov 17, 2009 at 11:18:02AM +0100, mok@mnet-online.de wrote:
> > > On Tue, 17 Nov 2009, Martin Jansa wrote:
> > > 
> > > > BTW: SRCPV seems to be expanded in PV a bit sooner than SRCREV was,
> > > > which with combination with BB_GIT_CLONE_FOR_SRCREV means that all git
> > > > repositories are checked during recipe parsing (Klaus Kurzmann has a
> > > > list of git repositories used in OE recipes which are not accessible)
> > 
> > Hopefully resolved with this bitbake patch
> 
> Sorry previous patch used localcount even for svn recipes, that was
> wrong.

Hopefully last time? :)

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

[-- Attachment #2: 0002-BB_GIT_CLONE_FOR_SRCREV-using-only-_sortable_buildnu.patch --]
[-- Type: text/plain, Size: 4257 bytes --]

From 77bdb1a1175e21af65e8c50f708b930d011a2e01 Mon Sep 17 00:00:00 2001
From: Martin Jansa <Martin.Jansa@gmail.com>
Date: Tue, 17 Nov 2009 15:44:43 +0100
Subject: [PATCH 2/2] BB_GIT_CLONE_FOR_SRCREV using only _sortable_buildnumber() for known revision

---
 lib/bb/fetch/__init__.py |   15 ++++++++-------
 lib/bb/fetch/git.py      |   33 +++++++++------------------------
 2 files changed, 17 insertions(+), 31 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 9508908..cc095e3 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -553,15 +553,13 @@ class Fetch(object):
         """
         
         """
-        has_want_sortable = hasattr(self, "_want_sortable_revision")
-        has_sortable = hasattr(self, "_sortable_revision")
+        has_want_sortable_buildnumber = hasattr(self, "_want_sortable_buildnumber")
+        has_sortable_buildnumber = hasattr(self, "_sortable_buildnumber")
+        has_sortable_revision = hasattr(self, "_sortable_revision")
 
-        if not has_want_sortable and has_sortable:
+        if has_sortable_revision:
             return self._sortable_revision(url, ud, d)
-        elif has_want_sortable and self._want_sortable_revision(url, ud, d) and has_sortable:
-            return self._sortable_revision(url, ud, d)
-        
-
+    
         pd = persist_data.PersistData(d)
         key = self.generate_revision_key(url, ud, d)
 
@@ -577,6 +575,9 @@ class Fetch(object):
         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
 
+        if has_sortable_buildnumber and has_want_sortable_buildnumber and self._want_sortable_buildnumber(url, ud, d):
+            count = self._sortable_buildnumber(url, ud, d, latest_rev)
+
         if count is None:
             count = "0"
         elif uselocalcount:
diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py
index 5cdf656..4dd58b0 100644
--- a/lib/bb/fetch/git.py
+++ b/lib/bb/fetch/git.py
@@ -146,44 +146,29 @@ class Git(Fetch):
     def _build_revision(self, url, ud, d):
         return ud.tag
 
-    def _want_sortable_revision(self, url, ud, d):
+    def _want_sortable_buildnumber(self, url, ud, d):
         return bb.data.getVar("BB_GIT_CLONE_FOR_SRCREV", d, True) or False
 
-    def _sortable_revision(self, url, ud, d):
+    def _sortable_buildnumber(self, url, ud, d, rev):
         """
-        This is only called when _want_sortable_revision called true
+        This is only called when _want_sortable_buildnumber called true
 
-        We will have to get the updated revision.
+        Latest revision is already known, we need only to count revisions in git repo.
         """
         gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
         repodir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
 
-        key = "GIT_CACHED_REVISION-%s-%s"  % (gitsrcname, ud.tag)
-        if bb.data.getVar(key, d):
-            return bb.data.getVar(key, d)
-
-
-        # Runtime warning on wrongly configured sources
-        if ud.tag == "1":
-            bb.msg.error(1, bb.msg.domain.Fetcher, "SRCREV is '1'. This indicates a configuration error of %s" % url)
-            return "0+1"
-
         cwd = os.getcwd()
 
         # Check if we have the rev already
         if not os.path.exists(repodir):
-            print "no repo"
-            self.go(None, ud, d)
+             bb.msg.error(bb.msg.domain.Fetcher, "GIT repository for %s doesn't exist in %s, cannot get sortable buildnumber" % (url, repodir))
+             return "0"
 
         os.chdir(repodir)
-        if not self._contains_ref(ud.tag, d):
-            self.go(None, ud, d)
 
-        output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % ud.tag, d, quiet=True)
+        output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % rev, d, quiet=True)
         os.chdir(cwd)
 
-        sortable_revision = "%s+%s" % (output.split()[0], ud.tag)
-        bb.data.setVar(key, sortable_revision, d)
-        return sortable_revision
-        
-
+        sortable_buildnumber= "%s" % (output.split()[0])
+        return sortable_buildnumber
-- 
1.6.5.2

Re: SRCPV migration

[Koen Kooi]

On 16-11-09 13:37, Richard Purdie wrote:

> I think we will have to hold off some of the SRCPV migration until
> bitbake has some kind of lock down functionality for the local build
> numbers.

So at least 2 core developers (Richard, me) have said we shouldn't use 
SRCPV for git stuff, and what do I spot today:

http://cgit.openembedded.org/cgit.cgi/openembedded/commit/?id=4f1f60e0f46324c6a968aff9bffeb727d98d5283

+PV = "0.0+1.0rc2+gitr${SRCPV}"

FFS!

Re: SRCPV migration

[Martin Jansa]

On Thu, Nov 19, 2009 at 05:02:05PM +0100, Koen Kooi wrote:
> On 16-11-09 13:37, Richard Purdie wrote:
> 
> >I think we will have to hold off some of the SRCPV migration until
> >bitbake has some kind of lock down functionality for the local build
> >numbers.
> 
> So at least 2 core developers (Richard, me) have said we shouldn't
> use SRCPV for git stuff, and what do I spot today:
> 
> http://cgit.openembedded.org/cgit.cgi/openembedded/commit/?id=4f1f60e0f46324c6a968aff9bffeb727d98d5283
> 
> +PV = "0.0+1.0rc2+gitr${SRCPV}"
> 
> FFS!

Sorry for that :/

I'm merging stuff from shr/merge branch where its already used and it
sliped through.

Should I commit SRCPV->SRCREV now, or is it ok for now as there is also
DEFAULT_PREFERRENCE = -1? which I also should set to -2 to ensure that
won't be pulled instead of mplayer_svn :/.

Sorry again..

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration

[Martin Jansa]

On Thu, Nov 19, 2009 at 05:02:05PM +0100, Koen Kooi wrote:
> On 16-11-09 13:37, Richard Purdie wrote:
> 
> >I think we will have to hold off some of the SRCPV migration until
> >bitbake has some kind of lock down functionality for the local build
> >numbers.
> 
> So at least 2 core developers (Richard, me) have said we shouldn't
> use SRCPV for git stuff, and what do I spot today:
> 
> http://cgit.openembedded.org/cgit.cgi/openembedded/commit/?id=4f1f60e0f46324c6a968aff9bffeb727d98d5283
> 
> +PV = "0.0+1.0rc2+gitr${SRCPV}"
> 
> FFS!

Hi again.. 

there is more SRCPVs in stuff I merged.. (which weren't probably built before and hopefully
aren't built now)

Do I have to kill all SRCPV there?

.. bitbake patches are prepared, waiting for review.. as soon as there is newer bitbake version
forced by sanity check, then all builders with 
BB_LOCALCOUNT_OVERRIDE = "1"
LOCALCOUNT = "0"
in local.conf, should get consistent localcount numbers == 0

If its ok to stay with SRCPV for those new recipes then we can save PE bump there..

Thanks..

Regards,

./calc/calc_git.bb:PV = "0.0.1+gitr${SRCPV}"
./xorg-util/glamo-dri-tests/glamo-dri-tests_git.bb:PV = "1.0.0+gitr${SRCPV}"
./efl1/illume-keyboard-default-alt_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-dutch_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-german_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-russian-terminal_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-numeric-alt_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-dvorak_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-arabic_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-hebrew_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-russian_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-browse_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-french_git.bb:PV = "0.0+gitr${SRCPV}"
./efl1/illume-keyboard-danish_git.bb:PV = "0.0+gitr${SRCPV}"
./linux/linux-openmoko-2.6.31_git.bb:PV = "${KERNEL_RELEASE}-${OEV}+gitr${SRCPV}"
./linux/linux-openmoko-shr-devel_git.bb:PV = "${KERNEL_RELEASE}-${OMV}+gitr${SRCPV}"
./linux/linux-openmoko-shr-drm-devel_git.bb:PV = "${KERNEL_RELEASE}-drm-${OMV}+gitr${SRCPV}"
./openmoko-3rdparty/advancedcaching_git.bb:PV = "0.1.2+gitr${SRCPV}"
./blipomoko/blipomoko_git.bb:PV = "0.0+gitr${SRCPV}"
./openmoocow/openmoocow_git.bb:PV = "0.0.3+gitr${SRCPV}"
./shr/shr-dialer_git.bb:PV = "0.0.2+gitr${SRCPV}"
./shr/shr-theme_git.bb:PV = "0.0.2+gitr${SRCPV}"
./shr/shr-splash-theme-simple_git.bb:PV = "1.2+gitr${SRCPV}"
./shr/gtk-theme-neo_git.bb:PV = "0.2-${EFL_SRCREV}+gitr${SRCPV}"
./shr/shr-splash-theme-niebiee_git.bb:PV = "1.2+gitr${SRCPV}"
./shr/e-wm-theme-illume-sixteen_git.bb:PV = "0.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/shr-splash_git.bb:PV = "1.2+gitr${SRCPV}"
./shr/libphone-ui-shr_git.bb:PV = "0.0.0+gitr${SRCPV}"
./shr/phoneuid_git.bb:PV = "0.0.0+gitr${SRCPV}"
./shr/shr-config_git.bb:PV = "0.0.2+gitr${SRCPV}"
./shr/shr-settings_git.bb:PV = "0.1.1+gitr${SRCPV}"
./shr/shr-specs_git.bb:PV = "0.0.0+gitr${SRCPV}"
./shr/etk-theme-shr_git.bb:PV = "1.1.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/icon-theme-neo_git.bb:PV = "0.2-${EFL_SRCREV}+gitr${SRCPV}"
./shr/libframeworkd-phonegui_git.bb:PV = "0.0.2+gitr${SRCPV}"
./shr/etk-theme-neo_git.bb:PV = "0.2-${EFL_SRCREV}+gitr${SRCPV}"
./shr/e-wm-theme-illume-shr_git.bb:PV = "1.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/elementary-theme-neo_git.bb:PV = "0.2.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/shr-splash-theme-dontpanic_git.bb:PV = "1.2+gitr${SRCPV}"
./shr/frameworkd-config-shr_git.bb:PV = "0.9.5.9+gitr${SRCPV}"
./shr/e-wm-theme-illume-gry_git.bb:PV = "0.3-${EFL_SRCREV}+gitr${SRCPV}"
./shr/elementary-theme-gry_git.bb:PV = "0.8-${EFL_SRCREV}+gitr${SRCPV}"
./shr/libframeworkd-phonegui-efl_git.bb:PV = "0.0.3+gitr${SRCPV}"
./shr/phoneui-apps_git.bb:PV = "0.0.0+gitr${SRCPV}"
./shr/shr-splash-theme-handy_git.bb:PV = "1.2+gitr${SRCPV}"
./shr/phonefsod_git.bb:PV = "0.0.0+gitr${SRCPV}"
./shr/libframeworkd-phonegui-efl-theme-neo_git.bb:PV = "0.2-${EFL_SRCREV}+gitr${SRCPV}"
./shr/e-wm-menu-shr_git.bb:PV = "1.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/e-wm-theme-illume-neo_git.bb:PV = "0.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/libframeworkd-phonegui-efl2_git.bb:PV = "0.0.1+gitr${SRCPV}"
./shr/elementary-theme-niebiee_git.bb:PV = "0.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/e-wm-config-illume-shr_git.bb:PV = "1.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/shr-messages_git.bb:PV = "0.0.2+gitr${SRCPV}"
./shr/shr-splash-theme-tux_git.bb:PV = "0.1+gitr${SRCPV}"
./shr/alsa-scenarii-shr_git.bb:PV = "1.0+gitr${SRCPV}"
./shr/shr-contacts_git.bb:PV = "0.0.2+gitr${SRCPV}"
./shr/elementary-theme-sixteen_git.bb:PV = "0.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/libmodulo_git.bb:PV = "0.0.1+gitr${SRCPV}"
./shr/shr-installer_git.bb:PV = "0.0.1+gitr${SRCPV}"
./shr/shr-theme-gtk-e17lookalike_git.bb:PV = "0.1.1+gitr${SRCPV}"
./shr/libphone-utils_git.bb:PV = "0.0.2+gitr${SRCPV}"
./shr/shr-today_git.bb:PV = "0.0.1+gitr${SRCPV}"
./shr/e-wm-theme-illume-niebiee_git.bb:PV = "0.1-${EFL_SRCREV}+gitr${SRCPV}"
./shr/ologicd_git.bb:PV = "0.0.1+gitr${SRCPV}"
./shr/shr-splash-theme-logo_git.bb:PV = "0.1+gitr${SRCPV}"
./shr/libphone-ui_git.bb:PV = "0.0.0+gitr${SRCPV}"
./shr/e-wm-sysactions-shr_git.bb:PV = "1.1-${EFL_SRCREV}+gitr${SRCPV}"
./pyphonelog/pyphonelog_git.bb:PV = "0.17.0+gitr${SRCPV}"
./bt-configure/bt-configure_git.bb:PV = "1.0.0+gitr${SRCPV}"
./pidgin/msn-pecan_git.bb:PV="0.0.1+gitr${SRCPV}"
./openmoko-projects/paroli_git.bb:PV = "0.2.1+gitr${SRCPV}"
./python/python-phoneutils_git.bb:PV = "0.0.2+gitr${SRCPV}"

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration

[Koen Kooi]

On 19-11-09 17:34, Martin Jansa wrote:
> On Thu, Nov 19, 2009 at 05:02:05PM +0100, Koen Kooi wrote:
>> On 16-11-09 13:37, Richard Purdie wrote:
>>
>>> I think we will have to hold off some of the SRCPV migration until
>>> bitbake has some kind of lock down functionality for the local build
>>> numbers.
>>
>> So at least 2 core developers (Richard, me) have said we shouldn't
>> use SRCPV for git stuff, and what do I spot today:
>>
>> http://cgit.openembedded.org/cgit.cgi/openembedded/commit/?id=4f1f60e0f46324c6a968aff9bffeb727d98d5283
>>
>> +PV = "0.0+1.0rc2+gitr${SRCPV}"
>>
>> FFS!
>
> Hi again..
>
> there is more SRCPVs in stuff I merged.. (which weren't probably built before and hopefully
> aren't built now)
>
> Do I have to kill all SRCPV there?

Yes. Your patch isn't in bitbake yet, so it's not in any bitbake release 
and hence OE can't use it.

Re: SRCPV migration

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 220 bytes --]

Even better fix for builders with BB_GIT_CLONE_FOR_SRCREV enabled.

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

[-- Attachment #2: 0002-BB_GIT_CLONE_FOR_SRCREV-using-only-_sortable_buildnu.patch --]
[-- Type: text/plain, Size: 5676 bytes --]

From b2a572666d6cf31c2c39b3123b97e972f01bbb36 Mon Sep 17 00:00:00 2001
From: Martin Jansa <Martin.Jansa@gmail.com>
Date: Tue, 17 Nov 2009 15:44:43 +0100
Subject: [PATCH 2/2] BB_GIT_CLONE_FOR_SRCREV using only _sortable_buildnumber() for known revision

* Sortable_buildnumber: return old count value if new isn't available
* Protects before putting wrong number (ie 0) to persistent cache when
  git repo is not available, or something happen to local git checkout
* Buildnumber shouldn't go backwards ever
* If this error happen always for some recipe SRCPV cannot ensure
  upgradeable versions
---
 lib/bb/fetch/__init__.py |   18 ++++++++++--------
 lib/bb/fetch/git.py      |   36 +++++++++++++-----------------------
 2 files changed, 23 insertions(+), 31 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 9508908..3e04f2e 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -553,21 +553,20 @@ class Fetch(object):
         """
         
         """
-        has_want_sortable = hasattr(self, "_want_sortable_revision")
-        has_sortable = hasattr(self, "_sortable_revision")
+        has_want_sortable_buildnumber = hasattr(self, "_want_sortable_buildnumber")
+        has_sortable_buildnumber = hasattr(self, "_sortable_buildnumber")
+        has_sortable_revision = hasattr(self, "_sortable_revision")
 
-        if not has_want_sortable and has_sortable:
+        if has_sortable_revision:
             return self._sortable_revision(url, ud, d)
-        elif has_want_sortable and self._want_sortable_revision(url, ud, d) and has_sortable:
-            return self._sortable_revision(url, ud, d)
-        
-
+    
         pd = persist_data.PersistData(d)
         key = self.generate_revision_key(url, ud, d)
 
         latest_rev = self._build_revision(url, ud, d)
         last_rev = pd.getValue("BB_URI_LOCALCOUNT", key + "_rev")
         uselocalcount = bb.data.getVar("BB_LOCALCOUNT_OVERRIDE", d, True) or False
+        usesortable_buildnumber = has_sortable_buildnumber and has_want_sortable_buildnumber and self._want_sortable_buildnumber(url, ud, d)
         count = None
         if uselocalcount:
             count = Fetch.localcount_internal_helper(ud, d)
@@ -577,9 +576,12 @@ class Fetch(object):
         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
 
+        if usesortable_buildnumber:
+            count = self._sortable_buildnumber(url, ud, d, latest_rev, count)
+
         if count is None:
             count = "0"
-        elif uselocalcount:
+        elif uselocalcount or usesortable_buildnumber:
             count = str(count)
         else:
             count = str(int(count) + 1)
diff --git a/lib/bb/fetch/git.py b/lib/bb/fetch/git.py
index 5cdf656..23fc377 100644
--- a/lib/bb/fetch/git.py
+++ b/lib/bb/fetch/git.py
@@ -146,44 +146,34 @@ class Git(Fetch):
     def _build_revision(self, url, ud, d):
         return ud.tag
 
-    def _want_sortable_revision(self, url, ud, d):
+    def _want_sortable_buildnumber(self, url, ud, d):
         return bb.data.getVar("BB_GIT_CLONE_FOR_SRCREV", d, True) or False
 
-    def _sortable_revision(self, url, ud, d):
+    def _sortable_buildnumber(self, url, ud, d, rev, count):
         """
-        This is only called when _want_sortable_revision called true
+        This is only called when _want_sortable_buildnumber called true
 
-        We will have to get the updated revision.
+        Latest revision is already known, we need only to count revisions in git repo.
         """
         gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
         repodir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
 
-        key = "GIT_CACHED_REVISION-%s-%s"  % (gitsrcname, ud.tag)
-        if bb.data.getVar(key, d):
-            return bb.data.getVar(key, d)
-
-
-        # Runtime warning on wrongly configured sources
-        if ud.tag == "1":
-            bb.msg.error(1, bb.msg.domain.Fetcher, "SRCREV is '1'. This indicates a configuration error of %s" % url)
-            return "0+1"
-
         cwd = os.getcwd()
 
         # Check if we have the rev already
         if not os.path.exists(repodir):
-            print "no repo"
-            self.go(None, ud, d)
+            bb.msg.error(bb.msg.domain.Fetcher, "GIT repository for %s doesn't exist in %s, cannot get sortable buildnumber, using old value" % (url, repodir))
+            return count
 
         os.chdir(repodir)
-        if not self._contains_ref(ud.tag, d):
-            self.go(None, ud, d)
 
-        output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % ud.tag, d, quiet=True)
+        output = runfetchcmd("git rev-list %s -- 2> /dev/null | wc -l" % rev, d, quiet=True)
         os.chdir(cwd)
 
-        sortable_revision = "%s+%s" % (output.split()[0], ud.tag)
-        bb.data.setVar(key, sortable_revision, d)
-        return sortable_revision
-        
+        sortable_buildnumber= "%s" % (output.split()[0])
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "GIT repository for %s in %s is returning %s revisions in rev-list before %s, old valued is %s" % (url, repodir, sortable_buildnumber, rev, count))
+        if sortable_buildnumber < count:
+            bb.msg.error(bb.msg.domain.Fetcher, "GIT repository for %s in %s is returning %s revisions in rev-list before %s which is lower than previous count %s, using old value" % (url, repodir, sortable_buildnumber, rev, count))
+            return count
 
+        return sortable_buildnumber
-- 
1.6.5.3

Re: SRCPV migration - How SRCPV works!

[Martin Jansa]

After longer IRC discussion with mwester I was persuaded to write some
summary of what SRCPV do and what's bad on SRCPV+BB_GIT_CLONE_FOR_SRCREV
combination.

SRCPV usage in svn recipes is safe, because subversion has sortable
revisions, so there is no problem with that and I won't write about it.

====

Why do we need AUTOREV for git recipes?

Every git recipe in OE tree should have some sane hash stored in
conf/distro/include/sane-srcrevs.inc
at least to make sure, that patches applied from that recipe will apply
to this known hash.

If some distribution is following upstream very close (ie freesmartphone
stack is developing pretty fast and to have latest features with same
stability you want to bump srcrev pretty often). This would create lots
of commits just changing SRCREV in sane-srcrev or some distro config in
OE tree and too much noise. That's why there is AUTOREV feature to
update to newest revision everytime you have that git recipe in dependency
tree.

AUTOREV can be dangerous as you cannot ensure that the recipe can be
even built (due to some upstream bug just commited or that patches in bb file
doesn't apply anymore).

That is why AUTOREV should be used only in for well justified recipes
end never enabled by default for any recipe.

Distributions or local builders can choose which packages will be using
AUTOREV in their distro config or local.conf. Insane builders can
include conf/distro/include/insane-srcrevs.inc where is lots of recipes
using AUTOREV.

====

Why do we need SRCPV for git recipes?

1)

Without AUTOREV the PV string looks like this
1.0+gitrDEADBEEFHASH
where PV in bb recipe looks like
1.0+gitr${SRCREV}
and in sane-srcrevs.inc is
SRCREV_pn-foo ?= "DEADBEEFHASH"

Even without AUTOREV we bump SRCREV from time to time to get new stuff
and when we change DEADBEEFHASH to BEEFDEADHASH in sane-srcrevs.inc or
somewhere else we want PV witch is higher than it was with DEADBEEFHASH,
which is not the case in this example.

That's why you can sometimes see PR used in PV string like this
PV="1.0+${PR}+gitr${SRCREV}"
and then every SRCREV bump goes with PR bump to ensure PV.old < PV.new.

2) 

With AUTOREV the PV string looks like this
1.0+gitrNNNN+DEADBEEFHASH
where NNNN is some localy incremented number which is incremented every
time you're building that recipe with different hash then last time.
Thats because we need sortable PVs for upgradable paths and from git
hash opkg/bitbake cannot see if DEADBEEFHASH > BEEFDEADHASH or
vice-versa.

3)

If we change PV in every git recipe in tree, we will always get PV
string in format
1.0+gitrNNNN+DEADBEEFHASH
and those NNNN will always ensure that PV.old < PV.new witch is great
and we don't need those ${PR} in PV right?

BUT!! as Koen promptly pointed pretty soon, before SRCREV->SRCPV migration 
could break OE tree. Really thanks for that! I really appreciate that, 
even thought it created more work for me and I need to write this rather 
long e-mail now :).

====

Problem with SRCPV

If your devices are using multiple feeds, built on multiple buildhost
across world (as angstrom does). Then locally increased number cannot
ensure non-decreasing PV between buildhosts. One builder can build some
recipe with changed revision every day and other will build that recipe
once a month. Then PV from first builder will be ie 1.0+gitr30+olderHash
and from second builder which just built newest Hash in sane-srcrevs
will get only 1.0+gitr2+newestHash.

The problem is not only in PV for that recipe and opk filename, but also
DEPENDS in .ipk file in recipes which depends on ie
1.0+gitr29+evenOlderHash which cannot be satisfied in feed from 2nd
builder even when there is package with newer hash!

In this case AUTOREV has same problem as SRCPV and is not used there.

How to ensure consistent PV across multiple builders?

1)

Never use AUTOREV + Never allow SRCPV merge :)

(I don't like this solution :))

2)

Never use AUTOREV + Use new LOCALCOUNT feature, which is now in bitbake
master and will be in bitbake before SRCPV merge.

http://cgit.openembedded.net/cgit.cgi/bitbake/commit/?id=2f32735463159e9e42e03819d6b505dba49c7f17

Instead of using ${PR} in PV you can now let SRCPV to hangle upgradable
paths for you. All you need to do is anable LOCALCOUNT feature for your
distro or whatever with
BB_LOCALCOUNT_OVERRIDE = "1"
and set default value for LOCALCOUNT
LOCALCOUNT = "0"

After this you will get almost the same PVs as before
1.0+r1+gitr0+DEADBEEFHASH
instead of
1.0+r1+gitrDEADBEEFHASH
and that 0 will be constant for you

To eliminate need for PR bump with every SRCREV change you need to put
LOCALCOUNT override for that package (I think it could be in
sane-srcrevs.inc with SRCREVS)

And with every SRCREV change, you'll bump LOCALCOUNT, ${PR} could/should 
be removed from PV, if you're providing LOCALCOUNT bumps.

Distributions which don't need to share feeds from multiple builders in
the same image, don't need to do a thing with LOCALCOUNT and upgradable
patch will be always provided for them.

But be carefull with persistent cache file
something like this:
tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite3

If you remove this file (ie for build from scratch), your counters will start
from 0 and you won't get upgradeable paths from images built before to
new feeds. You just need to backup this file and restore it before
building from scratch.

3)

You can get globally consistent revision number if you do
git clone git://repository
git rev-list $revision_you're_interested_in | wc -l

thats what bitbake do if you enable it with BB_GIT_CLONE_FOR_SRCREV

Problem with this is that it downloads whole git repository (huge load
on git repository servers) just to count how many revisions are in some
branch before your revision.

That's what it do now (without SRCPV merge) for every recipe you set to
AUTOREV. And this is done during recipe parsing phase (because we need
to know PV string for every recipe).

We all agreed that upstream server maintainers won't like us this way.

Also there is problem what revision number should be in PV if you're 
parsing OE tree when upstream git repository is not accessible or someone 
rebased branch there or just SRCREV doesn't exist anymore or is invalid.

This problem (accessing upstream servers during parsing) is elevated by
SRCPV, because SRCPV needs to know "revision number" everytime not only
for recipes set to AUTOREV.

Because of this I propose to BAN BB_GIT_CLONE_FOR_SRCREV unless someone
creates support in git for revision counting remotely without git clone
as git ls-remote does for checking if there is newer revision=hash available.

With patches I sent to this thread and RP prepared in bitbake/master
http://cgit.openembedded.net/cgit.cgi/bitbake/commit/?id=08e8144be8b564f2dea35c1680a4653f043ec2e2
bitbake will cache build numbers even with BB_GIT_CLONE_FOR_SRCREV
enabled and will count revisions only ONCE per SRCREV hash change.


====

What needs to be done?

SRCREV->SRCPV for subversion recipes was just formal and is merged
already.

SRCREV->SRCPV for git recipes needs those patches in new bitbake 
release forced by sanity check and SRCPV cannot be used before that.

All git recipes were converted to SRCPV in my short lived :) wip branch
martin_jansa/srcpv.

We need to bump PE with this change because
1.0+gitr01234DEADBEEFHASH > 1.0+gitr0+01234DEADBEEFHASH

I did it with SRCPV change for all recipes witch weren't in oe.dev when
I created that branch, new recipes added later to oe.dev from shr are
without PE bump, becase in that time I was sure that nobody built them
without SRCPV, now I cannot be sure, because they are already merged to
oe.dev with SRCREV now.

Also be carefull when bumping PE in git recipe that all versions of the
same recipe needs PE bump too. Without it git recipes have highest
versions and can be preferred instead of "newer" release versions". In
this cases I moved PE to .inc file if there was one and bumped it there
globally.

Kernel recipes are even more tricky, I bumped PE with SRCPV there too, 
but Koen warned me later (thanks again) that there needs to be consistent PE
even between different recipes, because multiple machines can share same
kernel recipe and machine can pick between multiple recipes with highest
PV? Hmm now I don't understand what you mean with kernel recipes in this:

<snip>
If something needs a PE bump, make sure you do it properly and bump PE
on the non scm fetched entries as well. For the kernel I wouldn't bump
PE, since too many machines can use different kernel recipes.
</snip>

====

Do we need something more?

Someone should summarize this too long mail to something shorter
more understandable :).

After bitbake with patches in sanity check we need ACK from all builder
which wants BB_LOCALCOUNT_OVERRIDE that they put it in their config with
default value for LOCALCOUNT, maybe warning with sanity check for
bitbake upgrade if its possible.

Something else I forgot to mention or even didn't think about?

Regards
-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration - How SRCPV works!

[Koen Kooi]

On 22-11-09 20:05, Martin Jansa wrote:

> Every git recipe in OE tree should have some sane hash stored in
> conf/distro/include/sane-srcrevs.inc

The cabal decided that checksums are "part of the metadata" and belongs 
in the recipes. I don't understand why SRCREVs are so different. For 
SRCREVs my life would be a lot easier if all SRCREV where put in their 
respective recipes. Distros can always do

SRCREV_pn-foo = "bar"
PV_pn-foo = "1.2.3+gitr$SRCPV"

in their distro.conf if needed. Due to scoping we do need some include 
for EFL_SRCREV, since those recipes are tightly tied together.

 > Kernel recipes are even more tricky, I bumped PE with SRCPV there too,
 > but Koen warned me later (thanks again) that there needs to be 
consistent PE
 > even between different recipes, because multiple machines can share same
 > kernel recipe and machine can pick between multiple recipes with highest
 > PV? Hmm now I don't understand what you mean with kernel recipes in this:
 >
 > <snip>
 > If something needs a PE bump, make sure you do it properly and bump PE
 > on the non scm fetched entries as well. For the kernel I wouldn't bump
 > PE, since too many machines can use different kernel recipes.
 > </snip>

Basically: if you bump PE for one kernel recipe, you need to bump PE for 
all of them, so I guess it should be done in e.g. kernel.bbclass or 
linux.inc.

Imagine I use a git kernel for 2.6.32rc6 and then add a recipe for 
2.6.32, I'm fairly sure I'm going to forget to add PE in 2.6.32 the 
first time.

 > But be carefull with persistent cache file
 > something like this:
 > tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite

So if I build pixman_git.bb for om-gta02 weekly, but monthly for 
beagleboard or om-gta01 I'll also get different numbers, right?
I think the count should only be in a machine specific database if the 
SRC_URI/SRCREV is machine specific.

regards,

Koen

Re: SRCPV migration - How SRCPV works!

[Martin Jansa]

On Mon, Nov 23, 2009 at 09:07:26AM +0100, Koen Kooi wrote:
> On 22-11-09 20:05, Martin Jansa wrote:
> 
> >Every git recipe in OE tree should have some sane hash stored in
> >conf/distro/include/sane-srcrevs.inc
> 
> The cabal decided that checksums are "part of the metadata" and
> belongs in the recipes. I don't understand why SRCREVs are so
> different. For SRCREVs my life would be a lot easier if all SRCREV
> where put in their respective recipes. Distros can always do
> 
> SRCREV_pn-foo = "bar"
> PV_pn-foo = "1.2.3+gitr$SRCPV"
> 
> in their distro.conf if needed. Due to scoping we do need some
> include for EFL_SRCREV, since those recipes are tightly tied
> together.

Sorry, I didn't intend to make all srcrevs in sane-srcrevs.inc.

I should write it a bit longer:
"should have some hash stored somewhere ie in sane-srcrevs.inc (which is
included probably in all sane distros) or in recipe itself.
If the SRCREV is defined ONLY in file like shr-autorev.inc (as we had some),
then all other distributions won't get defined SRCREV for that recipe
and fail to parse it."

> > Kernel recipes are even more tricky, I bumped PE with SRCPV there too,
> > but Koen warned me later (thanks again) that there needs to be
> consistent PE
> > even between different recipes, because multiple machines can share same
> > kernel recipe and machine can pick between multiple recipes with highest
> > PV? Hmm now I don't understand what you mean with kernel recipes in this:
> >
> > <snip>
> > If something needs a PE bump, make sure you do it properly and bump PE
> > on the non scm fetched entries as well. For the kernel I wouldn't bump
> > PE, since too many machines can use different kernel recipes.
> > </snip>
> 
> Basically: if you bump PE for one kernel recipe, you need to bump PE
> for all of them, so I guess it should be done in e.g. kernel.bbclass
> or linux.inc.
> 
> Imagine I use a git kernel for 2.6.32rc6 and then add a recipe for
> 2.6.32, I'm fairly sure I'm going to forget to add PE in 2.6.32 the
> first time.

Ah ok, now I see, thanks.

> > But be carefull with persistent cache file
> > something like this:
> > tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite
> 
> So if I build pixman_git.bb for om-gta02 weekly, but monthly for
> beagleboard or om-gta01 I'll also get different numbers, right?
> I think the count should only be in a machine specific database if
> the SRC_URI/SRCREV is machine specific.

If you want have consistent buildnumbers across machines, ie to see that
0.1+gitr12+hashA on om-gta01 is really newer than 0.1+gitr4+hashB on
beaglebord, I think you can point PERSISTENT_DIR variable to directory shared 
by all your builds.

But I'm not sure if there is some problem with it.
Only data I have there now is
BB_URI_HEADREVS    - latest revisions for autorev packages
BB_URI_LOCALCOUNT  - last built revision + last buildnumber

But there is also just url like this
git:gnuradio.org.git.balister.git-gnuradio_rev|bf7ad4d17514aba9fc5209bc916ce37482f77eaa
git:gnuradio.org.git.balister.git-gnuradio_count|0

I guess we should put at least also branch to key column (not sure how 
to easily migrate those revs already stored without parsing recipes which 
stored original value, but we can still store new revisions with branch
and look for last ones first with branch and if not found with old key)

Hmm sorry now I see that shared PERSISTENT_DIR can be much worse,
because if you have
SRCREV-pn_BLAH_om-gta01 = "hashA"
SRCREV-pn_BLAH_om-gta02 = "hashB"
then every rebuild will increment _count every time you build still the
same revisions hashA and hashB but once on gta01 and once on gta02. But
I think this is not addressed by bumping ${PR} too, because I will have
BLAH-0.1-r1-hashA on gta01 and BLAH-0.1-r1-hashB on gta02 right?

It can be still solved by schema
url, rev, count to get _count for every particular _rev ever built for
you

and
last used rev in another table with url_branch_machine key (I guess easy
to implement) or in machine specific another PERSISTENT_DIR (messy to
have 2 PERSISTENT_DIRs, harder to implement)

Do we need this now, or can be LOCALCOUNT override used for this too?

> regards,

Regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration - How SRCPV works!

[Koen Kooi]

On 23-11-09 09:52, Martin Jansa wrote:
> On Mon, Nov 23, 2009 at 09:07:26AM +0100, Koen Kooi wrote:
>> On 22-11-09 20:05, Martin Jansa wrote:
>>
>>> Every git recipe in OE tree should have some sane hash stored in
>>> conf/distro/include/sane-srcrevs.inc
>>
>> The cabal decided that checksums are "part of the metadata" and
>> belongs in the recipes. I don't understand why SRCREVs are so
>> different. For SRCREVs my life would be a lot easier if all SRCREV
>> where put in their respective recipes. Distros can always do
>>
>> SRCREV_pn-foo = "bar"
>> PV_pn-foo = "1.2.3+gitr$SRCPV"
>>
>> in their distro.conf if needed. Due to scoping we do need some
>> include for EFL_SRCREV, since those recipes are tightly tied
>> together.
>
> Sorry, I didn't intend to make all srcrevs in sane-srcrevs.inc.
>
> I should write it a bit longer:
> "should have some hash stored somewhere ie in sane-srcrevs.inc (which is
> included probably in all sane distros) or in recipe itself.
> If the SRCREV is defined ONLY in file like shr-autorev.inc (as we had some),
> then all other distributions won't get defined SRCREV for that recipe
> and fail to parse it."
>
>>> Kernel recipes are even more tricky, I bumped PE with SRCPV there too,
>>> but Koen warned me later (thanks again) that there needs to be
>> consistent PE
>>> even between different recipes, because multiple machines can share same
>>> kernel recipe and machine can pick between multiple recipes with highest
>>> PV? Hmm now I don't understand what you mean with kernel recipes in this:
>>>
>>> <snip>
>>> If something needs a PE bump, make sure you do it properly and bump PE
>>> on the non scm fetched entries as well. For the kernel I wouldn't bump
>>> PE, since too many machines can use different kernel recipes.
>>> </snip>
>>
>> Basically: if you bump PE for one kernel recipe, you need to bump PE
>> for all of them, so I guess it should be done in e.g. kernel.bbclass
>> or linux.inc.
>>
>> Imagine I use a git kernel for 2.6.32rc6 and then add a recipe for
>> 2.6.32, I'm fairly sure I'm going to forget to add PE in 2.6.32 the
>> first time.
>
> Ah ok, now I see, thanks.
>
>>> But be carefull with persistent cache file
>>> something like this:
>>> tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite
>>
>> So if I build pixman_git.bb for om-gta02 weekly, but monthly for
>> beagleboard or om-gta01 I'll also get different numbers, right?
>> I think the count should only be in a machine specific database if
>> the SRC_URI/SRCREV is machine specific.
>
> If you want have consistent buildnumbers across machines, ie to see that
> 0.1+gitr12+hashA on om-gta01 is really newer than 0.1+gitr4+hashB on
> beaglebord, I think you can point PERSISTENT_DIR variable to directory shared
> by all your builds.
>
> But I'm not sure if there is some problem with it.
> Only data I have there now is
> BB_URI_HEADREVS    - latest revisions for autorev packages
> BB_URI_LOCALCOUNT  - last built revision + last buildnumber
>
> But there is also just url like this
> git:gnuradio.org.git.balister.git-gnuradio_rev|bf7ad4d17514aba9fc5209bc916ce37482f77eaa
> git:gnuradio.org.git.balister.git-gnuradio_count|0
>
> I guess we should put at least also branch to key column (not sure how
> to easily migrate those revs already stored without parsing recipes which
> stored original value, but we can still store new revisions with branch
> and look for last ones first with branch and if not found with old key)
>
> Hmm sorry now I see that shared PERSISTENT_DIR can be much worse,
> because if you have
> SRCREV-pn_BLAH_om-gta01 = "hashA"
> SRCREV-pn_BLAH_om-gta02 = "hashB"
> then every rebuild will increment _count every time you build still the
> same revisions hashA and hashB but once on gta01 and once on gta02. But
> I think this is not addressed by bumping ${PR} too, because I will have
> BLAH-0.1-r1-hashA on gta01 and BLAH-0.1-r1-hashB on gta02 right?
>
> It can be still solved by schema
> url, rev, count to get _count for every particular _rev ever built for
> you
>
> and
> last used rev in another table with url_branch_machine key (I guess easy
> to implement) or in machine specific another PERSISTENT_DIR (messy to
> have 2 PERSISTENT_DIRs, harder to implement)
>
> Do we need this now, or can be LOCALCOUNT override used for this too?

I'd like to have SRCPV working well before importing it into .dev.

regards,

Koen

Re: SRCPV migration - How SRCPV works!

[Martin Jansa]

On Mon, Nov 23, 2009 at 12:12:03PM +0100, Koen Kooi wrote:
> On 23-11-09 09:52, Martin Jansa wrote:
> >On Mon, Nov 23, 2009 at 09:07:26AM +0100, Koen Kooi wrote:
> >>On 22-11-09 20:05, Martin Jansa wrote:
> >>
> >>>Every git recipe in OE tree should have some sane hash stored in
> >>>conf/distro/include/sane-srcrevs.inc
> >>
> >>The cabal decided that checksums are "part of the metadata" and
> >>belongs in the recipes. I don't understand why SRCREVs are so
> >>different. For SRCREVs my life would be a lot easier if all SRCREV
> >>where put in their respective recipes. Distros can always do
> >>
> >>SRCREV_pn-foo = "bar"
> >>PV_pn-foo = "1.2.3+gitr$SRCPV"
> >>
> >>in their distro.conf if needed. Due to scoping we do need some
> >>include for EFL_SRCREV, since those recipes are tightly tied
> >>together.
> >
> >Sorry, I didn't intend to make all srcrevs in sane-srcrevs.inc.
> >
> >I should write it a bit longer:
> >"should have some hash stored somewhere ie in sane-srcrevs.inc (which is
> >included probably in all sane distros) or in recipe itself.
> >If the SRCREV is defined ONLY in file like shr-autorev.inc (as we had some),
> >then all other distributions won't get defined SRCREV for that recipe
> >and fail to parse it."
> >
> >>>Kernel recipes are even more tricky, I bumped PE with SRCPV there too,
> >>>but Koen warned me later (thanks again) that there needs to be
> >>consistent PE
> >>>even between different recipes, because multiple machines can share same
> >>>kernel recipe and machine can pick between multiple recipes with highest
> >>>PV? Hmm now I don't understand what you mean with kernel recipes in this:
> >>>
> >>><snip>
> >>>If something needs a PE bump, make sure you do it properly and bump PE
> >>>on the non scm fetched entries as well. For the kernel I wouldn't bump
> >>>PE, since too many machines can use different kernel recipes.
> >>></snip>
> >>
> >>Basically: if you bump PE for one kernel recipe, you need to bump PE
> >>for all of them, so I guess it should be done in e.g. kernel.bbclass
> >>or linux.inc.
> >>
> >>Imagine I use a git kernel for 2.6.32rc6 and then add a recipe for
> >>2.6.32, I'm fairly sure I'm going to forget to add PE in 2.6.32 the
> >>first time.
> >
> >Ah ok, now I see, thanks.
> >
> >>>But be carefull with persistent cache file
> >>>something like this:
> >>>tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite
> >>
> >>So if I build pixman_git.bb for om-gta02 weekly, but monthly for
> >>beagleboard or om-gta01 I'll also get different numbers, right?
> >>I think the count should only be in a machine specific database if
> >>the SRC_URI/SRCREV is machine specific.
> >
> >If you want have consistent buildnumbers across machines, ie to see that
> >0.1+gitr12+hashA on om-gta01 is really newer than 0.1+gitr4+hashB on
> >beaglebord, I think you can point PERSISTENT_DIR variable to directory shared
> >by all your builds.
> >
> >But I'm not sure if there is some problem with it.
> >Only data I have there now is
> >BB_URI_HEADREVS    - latest revisions for autorev packages
> >BB_URI_LOCALCOUNT  - last built revision + last buildnumber
> >
> >But there is also just url like this
> >git:gnuradio.org.git.balister.git-gnuradio_rev|bf7ad4d17514aba9fc5209bc916ce37482f77eaa
> >git:gnuradio.org.git.balister.git-gnuradio_count|0
> >
> >I guess we should put at least also branch to key column (not sure how
> >to easily migrate those revs already stored without parsing recipes which
> >stored original value, but we can still store new revisions with branch
> >and look for last ones first with branch and if not found with old key)
> >
> >Hmm sorry now I see that shared PERSISTENT_DIR can be much worse,
> >because if you have
> >SRCREV-pn_BLAH_om-gta01 = "hashA"
> >SRCREV-pn_BLAH_om-gta02 = "hashB"
> >then every rebuild will increment _count every time you build still the
> >same revisions hashA and hashB but once on gta01 and once on gta02. But
> >I think this is not addressed by bumping ${PR} too, because I will have
> >BLAH-0.1-r1-hashA on gta01 and BLAH-0.1-r1-hashB on gta02 right?
> >
> >It can be still solved by schema
> >url, rev, count to get _count for every particular _rev ever built for
> >you
> >
> >and
> >last used rev in another table with url_branch_machine key (I guess easy
> >to implement) or in machine specific another PERSISTENT_DIR (messy to
> >have 2 PERSISTENT_DIRs, harder to implement)
> >
> >Do we need this now, or can be LOCALCOUNT override used for this too?
> 
> I'd like to have SRCPV working well before importing it into .dev.

RP or someone else:

What do you think about proposed implementation?

Another table with localcount for every revision built, not only for last
one.

And table with last revisions with branch and machine in key?

Then script for persistent cache upgrade or upgrade it while building if
new table with localname will have have new name and if not found there,
bitbake will try in oldtable and safe result to new one.

regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration - How SRCPV works!

[Richard Purdie]

On Mon, 2009-11-23 at 12:42 +0100, Martin Jansa wrote:
> RP or someone else:
> 
> What do you think about proposed implementation?
> 
> Another table with localcount for every revision built, not only for last
> one.
> 
> And table with last revisions with branch and machine in key?

Keeping history is actually quite pointless. The only thing you can use
it for is to make PV go backwards which is exactly what we're trying to
avoid.

The machine specific part is a massive nightmare and my advise is simple
- don't use different revisions for different machines, this is simply
not designed to work. Worst case you create a locked down version for
one of the machines.

> Then script for persistent cache upgrade or upgrade it while building if
> new table with localname will have have new name and if not found there,
> bitbake will try in oldtable and safe result to new one.

This is getting way too complicated for a set of hypothetical situations
nobody is likely to use. Lets try and keep this simple.

The core problem is that nobody has created a way to share the persist
data between build machines. The agreed workaround was to add a way for
the revisions to be locked down though configuration and we're on our
way to do this.

Cheers,

Richard

Re: SRCPV migration - How SRCPV works!

[Richard Purdie]

On Mon, 2009-11-23 at 09:07 +0100, Koen Kooi wrote:
> On 22-11-09 20:05, Martin Jansa wrote:
> 
> > Every git recipe in OE tree should have some sane hash stored in
> > conf/distro/include/sane-srcrevs.inc
> 
> The cabal decided that checksums are "part of the metadata" and belongs 
> in the recipes. I don't understand why SRCREVs are so different. For 
> SRCREVs my life would be a lot easier if all SRCREV where put in their 
> respective recipes. Distros can always do
> 
> SRCREV_pn-foo = "bar"
> PV_pn-foo = "1.2.3+gitr$SRCPV"
> 
> in their distro.conf if needed. Due to scoping we do need some include 
> for EFL_SRCREV, since those recipes are tightly tied together.

FWIW I dislike it that we have SRCREVs elsewhere. There is a problem
(bug?) with bitbake that makes this necessary though and that bug is
hard to fix :(.

>  > But be carefull with persistent cache file
>  > something like this:
>  > tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite
> 
> So if I build pixman_git.bb for om-gta02 weekly, but monthly for 
> beagleboard or om-gta01 I'll also get different numbers, right?
> I think the count should only be in a machine specific database if the 
> SRC_URI/SRCREV is machine specific.

As I understand it you'll lock locking down the local build revisions
with Angstrom anyway?

If we try and solve every issue all at once with this, I doubt we're
going to get anywhere fast. If we take the issues one at a time and chip
away at them we may end up somewhere better.

The whole SRCREV thing is a can of worms. Its no secret I dislike the
thing and the fact we've never had a totally clean way to implement it.
I also recognise its useful though, I use it myself :/. SRCPV is a step
to fixing a number of problems, we just need to be careful we don't
create many others.

Cheers,

Richard

Re: SRCPV migration - How SRCPV works!

[Philip Balister]

On 11/23/2009 07:15 AM, Richard Purdie wrote:
> On Mon, 2009-11-23 at 09:07 +0100, Koen Kooi wrote:
>> On 22-11-09 20:05, Martin Jansa wrote:
>>
>>> Every git recipe in OE tree should have some sane hash stored in
>>> conf/distro/include/sane-srcrevs.inc
>>
>> The cabal decided that checksums are "part of the metadata" and belongs
>> in the recipes. I don't understand why SRCREVs are so different. For
>> SRCREVs my life would be a lot easier if all SRCREV where put in their
>> respective recipes. Distros can always do
>>
>> SRCREV_pn-foo = "bar"
>> PV_pn-foo = "1.2.3+gitr$SRCPV"
>>
>> in their distro.conf if needed. Due to scoping we do need some include
>> for EFL_SRCREV, since those recipes are tightly tied together.
>
> FWIW I dislike it that we have SRCREVs elsewhere. There is a problem
> (bug?) with bitbake that makes this necessary though and that bug is
> hard to fix :(.

The difference between checksums and SRCREV's is that there is one 
checksum per file, but different distro's may use different versions of 
the same software.

As Koen noted a little later, before SRCPV goes into dev, it should be 
well worked out. I'm finding the email threads hard to follow :(

Philip


>
>>   >  But be carefull with persistent cache file
>>   >  something like this:
>>   >  tmpdir-dev-shr/cache/om-gta02/bb_persist_data.sqlite
>>
>> So if I build pixman_git.bb for om-gta02 weekly, but monthly for
>> beagleboard or om-gta01 I'll also get different numbers, right?
>> I think the count should only be in a machine specific database if the
>> SRC_URI/SRCREV is machine specific.
>
> As I understand it you'll lock locking down the local build revisions
> with Angstrom anyway?
>
> If we try and solve every issue all at once with this, I doubt we're
> going to get anywhere fast. If we take the issues one at a time and chip
> away at them we may end up somewhere better.
>
> The whole SRCREV thing is a can of worms. Its no secret I dislike the
> thing and the fact we've never had a totally clean way to implement it.
> I also recognise its useful though, I use it myself :/. SRCPV is a step
> to fixing a number of problems, we just need to be careful we don't
> create many others.
>
> Cheers,
>
> Richard
>
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: SRCPV migration - How SRCPV works!

[Koen Kooi]

On 23-11-09 13:29, Philip Balister wrote:
> On 11/23/2009 07:15 AM, Richard Purdie wrote:
>> On Mon, 2009-11-23 at 09:07 +0100, Koen Kooi wrote:
>>> On 22-11-09 20:05, Martin Jansa wrote:
>>>
>>>> Every git recipe in OE tree should have some sane hash stored in
>>>> conf/distro/include/sane-srcrevs.inc
>>>
>>> The cabal decided that checksums are "part of the metadata" and belongs
>>> in the recipes. I don't understand why SRCREVs are so different. For
>>> SRCREVs my life would be a lot easier if all SRCREV where put in their
>>> respective recipes. Distros can always do
>>>
>>> SRCREV_pn-foo = "bar"
>>> PV_pn-foo = "1.2.3+gitr$SRCPV"
>>>
>>> in their distro.conf if needed. Due to scoping we do need some include
>>> for EFL_SRCREV, since those recipes are tightly tied together.
>>
>> FWIW I dislike it that we have SRCREVs elsewhere. There is a problem
>> (bug?) with bitbake that makes this necessary though and that bug is
>> hard to fix :(.
>
> The difference between checksums and SRCREV's is that there is one
> checksum per file, but different distro's may use different versions of
> the same software.

And that makes it not part of the metadata?

regards,

Koen

Re: SRCPV migration - How SRCPV works!

[Koen Kooi]

On 23-11-09 13:15, Richard Purdie wrote:

> As I understand it you'll lock locking down the local build revisions
> with Angstrom anyway?

Dunno about that, ideally the SRCPV merge should have no impact at all 
on existing distros, but it looks like everyone will be forced to lock 
revisions/counts down.
If there is a way to convert the database to a .inc file then we'd be a 
step closer to coordinating counts between buildhosts (or rebuilds from 
scratch).
Currently the SRCPV looks like a major step backwards to the current 
situation unless you are on a single buildhost *and* never delete TMPDIR 
*and* use AUTOREV *and* care about upgrade paths.

It would be a lot better if bitbake could just do the revlog | wc -l 
trick after do_fetch has run. Or at least use that as localcount if a 
snapshot exists in TMPDIR during parsing.

regards,

Koen

Re: SRCPV migration - How SRCPV works!

[Otavio Salvador]

Hello,

On Mon, Nov 23, 2009 at 11:31 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
> On 23-11-09 13:15, Richard Purdie wrote:
>
>> As I understand it you'll lock locking down the local build revisions
>> with Angstrom anyway?
>
> Dunno about that, ideally the SRCPV merge should have no impact at all on
> existing distros, but it looks like everyone will be forced to lock
> revisions/counts down.
> If there is a way to convert the database to a .inc file then we'd be a step
> closer to coordinating counts between buildhosts (or rebuilds from scratch).
> Currently the SRCPV looks like a major step backwards to the current
> situation unless you are on a single buildhost *and* never delete TMPDIR
> *and* use AUTOREV *and* care about upgrade paths.
>
> It would be a lot better if bitbake could just do the revlog | wc -l trick
> after do_fetch has run. Or at least use that as localcount if a snapshot
> exists in TMPDIR during parsing.

After looking at what SRCPV means for distro POV I fully agree with Koen.

It is going to add more problems then it solves. If a distro has more
then one buildhost it will be a nightmare to manage it and very error
prone :-(

-- 
Otavio Salvador                  O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854         http://projetos.ossystems.com.br

Re: SRCPV migration - How SRCPV works!

[Richard Purdie]

On Mon, 2009-11-23 at 14:31 +0100, Koen Kooi wrote:
> On 23-11-09 13:15, Richard Purdie wrote:
> 
> > As I understand it you'll lock locking down the local build revisions
> > with Angstrom anyway?
> 
> Dunno about that, ideally the SRCPV merge should have no impact at all 
> on existing distros, but it looks like everyone will be forced to lock 
> revisions/counts down.

How is locking the counts down using LOCALCOUNT any different to the
current situation?

> If there is a way to convert the database to a .inc file then we'd be a 
> step closer to coordinating counts between buildhosts (or rebuilds from 
> scratch).

Any method using .inc files is going to race. The only solution that is
likely to work is a single server allocating numbers in some form.

> Currently the SRCPV looks like a major step backwards to the current 
> situation unless you are on a single buildhost *and* never delete TMPDIR 
> *and* use AUTOREV *and* care about upgrade paths.

Well this clearly isn't the case. 

Its intended to be a neural step (apart from some PE issues) for
everyone except for the users of AUTOREV who it helps. Their use case is
limited to a single autobuilder model where they need to keep one file
in TMPDIR but can otherwise delete it.

> It would be a lot better if bitbake could just do the revlog | wc -l 
> trick after do_fetch has run. Or at least use that as localcount if a 
> snapshot exists in TMPDIR during parsing.

As you're more than well aware, two different behaviours depending on
whether "a snapshot exists in TMPDIR during parsing" is maintenance and
reproducibility nightmare.

The first suggestion would be nice, patches welcome.

Cheers,

Richard

Re: SRCPV migration - How SRCPV works!

[Koen Kooi]

On 23-11-09 14:52, Otavio Salvador wrote:
> Hello,
>
> On Mon, Nov 23, 2009 at 11:31 AM, Koen Kooi<k.kooi@student.utwente.nl>  wrote:
>> On 23-11-09 13:15, Richard Purdie wrote:
>>
>>> As I understand it you'll lock locking down the local build revisions
>>> with Angstrom anyway?
>>
>> Dunno about that, ideally the SRCPV merge should have no impact at all on
>> existing distros, but it looks like everyone will be forced to lock
>> revisions/counts down.
>> If there is a way to convert the database to a .inc file then we'd be a step
>> closer to coordinating counts between buildhosts (or rebuilds from scratch).
>> Currently the SRCPV looks like a major step backwards to the current
>> situation unless you are on a single buildhost *and* never delete TMPDIR
>> *and* use AUTOREV *and* care about upgrade paths.
>>
>> It would be a lot better if bitbake could just do the revlog | wc -l trick
>> after do_fetch has run. Or at least use that as localcount if a snapshot
>> exists in TMPDIR during parsing.
>
> After looking at what SRCPV means for distro POV I fully agree with Koen.
>
> It is going to add more problems then it solves. If a distro has more
> then one buildhost it will be a nightmare to manage it and very error
> prone :-(

Even in the single buildhost world things start falling down if you rm 
TMPDIR.

regards,

Koen

Re: SRCPV migration - How SRCPV works!

[Koen Kooi]

On 23-11-09 15:29, Richard Purdie wrote:
> On Mon, 2009-11-23 at 14:31 +0100, Koen Kooi wrote:
>> On 23-11-09 13:15, Richard Purdie wrote:
>>
>>> As I understand it you'll lock locking down the local build revisions
>>> with Angstrom anyway?
>>
>> Dunno about that, ideally the SRCPV merge should have no impact at all
>> on existing distros, but it looks like everyone will be forced to lock
>> revisions/counts down.
>
> How is locking the counts down using LOCALCOUNT any different to the
> current situation?

In the current situation you lock down SRCREV and change PV,PR 
accordingly. In the new world you lock down SRCREV, increase localcount 
(per recipe or globally) and change PV,PR accordingly.

I just went from a 2 step error prone situation to a 3 step error prone 
situation. And I need to track an *extra* variable as well now.

regards,

Koen

Re: SRCPV migration - How SRCPV works!

[Philip Balister]

On 11/23/2009 09:29 AM, Richard Purdie wrote:
> On Mon, 2009-11-23 at 14:31 +0100, Koen Kooi wrote:
>> On 23-11-09 13:15, Richard Purdie wrote:
>>
>>> As I understand it you'll lock locking down the local build revisions
>>> with Angstrom anyway?
>>
>> Dunno about that, ideally the SRCPV merge should have no impact at all
>> on existing distros, but it looks like everyone will be forced to lock
>> revisions/counts down.
>
> How is locking the counts down using LOCALCOUNT any different to the
> current situation?
>
>> If there is a way to convert the database to a .inc file then we'd be a
>> step closer to coordinating counts between buildhosts (or rebuilds from
>> scratch).
>
> Any method using .inc files is going to race. The only solution that is
> likely to work is a single server allocating numbers in some form.
>
>> Currently the SRCPV looks like a major step backwards to the current
>> situation unless you are on a single buildhost *and* never delete TMPDIR
>> *and* use AUTOREV *and* care about upgrade paths.
>
> Well this clearly isn't the case.
>
> Its intended to be a neural step (apart from some PE issues) for
> everyone except for the users of AUTOREV who it helps. Their use case is
> limited to a single autobuilder model where they need to keep one file
> in TMPDIR but can otherwise delete it.

Given the only concrete benefit I can see is that it makes it easier for 
people using AUTOREV and git, can we examine the use case for this and 
see if there are alternatives? If this is needed for people doing 
development work, wouldn't it make more sense to focus in sdk issues?

Philip



>
>> It would be a lot better if bitbake could just do the revlog | wc -l
>> trick after do_fetch has run. Or at least use that as localcount if a
>> snapshot exists in TMPDIR during parsing.
>
> As you're more than well aware, two different behaviours depending on
> whether "a snapshot exists in TMPDIR during parsing" is maintenance and
> reproducibility nightmare.
>
> The first suggestion would be nice, patches welcome.
>
> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: SRCPV migration - How SRCPV works!

[Martin Jansa]

On Mon, Nov 23, 2009 at 03:58:00PM +0100, Koen Kooi wrote:
> On 23-11-09 14:52, Otavio Salvador wrote:
> >Hello,
> >
> >On Mon, Nov 23, 2009 at 11:31 AM, Koen Kooi<k.kooi@student.utwente.nl>  wrote:
> >>On 23-11-09 13:15, Richard Purdie wrote:
> >>
> >>>As I understand it you'll lock locking down the local build revisions
> >>>with Angstrom anyway?
> >>
> >>Dunno about that, ideally the SRCPV merge should have no impact at all on
> >>existing distros, but it looks like everyone will be forced to lock
> >>revisions/counts down.
> >>If there is a way to convert the database to a .inc file then we'd be a step
> >>closer to coordinating counts between buildhosts (or rebuilds from scratch).
> >>Currently the SRCPV looks like a major step backwards to the current
> >>situation unless you are on a single buildhost *and* never delete TMPDIR
> >>*and* use AUTOREV *and* care about upgrade paths.
> >>
> >>It would be a lot better if bitbake could just do the revlog | wc -l trick
> >>after do_fetch has run. Or at least use that as localcount if a snapshot
> >>exists in TMPDIR during parsing.
> >
> >After looking at what SRCPV means for distro POV I fully agree with Koen.
> >
> >It is going to add more problems then it solves. If a distro has more
> >then one buildhost it will be a nightmare to manage it and very error
> >prone :-(
> 
> Even in the single buildhost world things start falling down if you
> rm TMPDIR.
> 
> regards,

No if you're using LOCALCOUNT_OVERRIDE

Yes if you're using AUTOREV without SRCPV.

What about enabling LOCALCOUNT_OVERRIDE by default.
Nothing change for everybody (only harmless constant '0+' in PV).
If they want to stay with bumping PR, its their choice. I think bumping
LOCALCOUNT with SRCREV change is nicer, because it changes only number 
before hash and not both PV and PR, but package maintainer can choose
which he likes better.

And people using AUTOREV will get upgradable versions even when they
switch to sane-srcrev for some package from time to time. And nothings
is worse then before for them, because they cannot rm TMPDIR now as well
as with SRCPV.

regards,
-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration - How SRCPV works!

[Martin Jansa]

On Mon, Nov 23, 2009 at 04:00:26PM +0100, Koen Kooi wrote:
> On 23-11-09 15:29, Richard Purdie wrote:
> >On Mon, 2009-11-23 at 14:31 +0100, Koen Kooi wrote:
> >>On 23-11-09 13:15, Richard Purdie wrote:
> >>
> >>>As I understand it you'll lock locking down the local build revisions
> >>>with Angstrom anyway?
> >>
> >>Dunno about that, ideally the SRCPV merge should have no impact at all
> >>on existing distros, but it looks like everyone will be forced to lock
> >>revisions/counts down.
> >
> >How is locking the counts down using LOCALCOUNT any different to the
> >current situation?
> 
> In the current situation you lock down SRCREV and change PV,PR
> accordingly. In the new world you lock down SRCREV, increase
> localcount (per recipe or globally) and change PV,PR accordingly.

Why do you need to change PV,PR if you bump LOCALCOUNT?

> I just went from a 2 step error prone situation to a 3 step error
> prone situation. And I need to track an *extra* variable as well
> now.

I see only 1 step in one place (SRCREV+LOCALCOUNT in sane-srcrevs or
recipe). And as long as its in the same file, it can be a bit less
error-prone then change of SRCREV in sane-srcrevs and PV/PR bump in
recipe.

> regards,

regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa                         

Re: SRCPV migration - How SRCPV works!

[Chris Conroy]

On Sun, 2009-11-22 at 20:05 +0100, Martin Jansa wrote:
> Problem with this is that it downloads whole git repository (huge load
> on git repository servers) just to count how many revisions are in
> some
> branch before your revision.
> 
> That's what it do now (without SRCPV merge) for every recipe you set
> to
> AUTOREV. And this is done during recipe parsing phase (because we need
> to know PV string for every recipe).
> 
> We all agreed that upstream server maintainers won't like us this way.

Forgive me since before this thread I have not been following the SRCPV
stuff terribly closely, but this seems to be like a lot of work to solve
a different sort of use case.

If you are a distro that is tracking untagged, unreleased git revisions
from an upstream tree, then do you really care if you have to clone the
tree once per build machine? You're probably already doing this anyways
if you are using a package in this fashion. Doing a full clone on every
single build would obviously be a terrible thing, but as long as this
gets cached it doesn't seem like an awful cost to me.

And, if you have many build users in your distro, then you could easily
fix this by hosting your own mirrored git server.

I'm probably ignorant of the real use case, but just in case I'm not,
there's my two cents.

--Chris

Re: SRCPV migration - How SRCPV works!

[Koen Kooi]

On 23-11-09 16:12, Martin Jansa wrote:
> On Mon, Nov 23, 2009 at 04:00:26PM +0100, Koen Kooi wrote:
>> On 23-11-09 15:29, Richard Purdie wrote:
>>> On Mon, 2009-11-23 at 14:31 +0100, Koen Kooi wrote:
>>>> On 23-11-09 13:15, Richard Purdie wrote:
>>>>
>>>>> As I understand it you'll lock locking down the local build revisions
>>>>> with Angstrom anyway?
>>>>
>>>> Dunno about that, ideally the SRCPV merge should have no impact at all
>>>> on existing distros, but it looks like everyone will be forced to lock
>>>> revisions/counts down.
>>>
>>> How is locking the counts down using LOCALCOUNT any different to the
>>> current situation?
>>
>> In the current situation you lock down SRCREV and change PV,PR
>> accordingly. In the new world you lock down SRCREV, increase
>> localcount (per recipe or globally) and change PV,PR accordingly.
>
> Why do you need to change PV,PR if you bump LOCALCOUNT?

Because the version might have gone from 1.4 to 1.6?

regards,

Koen

>
>> I just went from a 2 step error prone situation to a 3 step error
>> prone situation. And I need to track an *extra* variable as well
>> now.
>
> I see only 1 step in one place (SRCREV+LOCALCOUNT in sane-srcrevs or
> recipe). And as long as its in the same file, it can be a bit less
> error-prone then change of SRCREV in sane-srcrevs and PV/PR bump in
> recipe.
>
>> regards,
>
> regards,
>

Re: SRCPV migration - How SRCPV works!

[Martin Jansa]

On Mon, Nov 23, 2009 at 04:52:03PM +0100, Koen Kooi wrote:
> Because the version might have gone from 1.4 to 1.6?

But this isn't any worse with SRCPV or is it?

Now you need to bump SRCREV *and* PR with every SRCREV change unless PV changed.
And SRCREV *and* PV if PV changed (maybe you also reset PR).

With LOCALCOUNT you bump only LOCALCOUNT with every SRCREV change.

And if PV changed you change it too as before and you don't need to
update LOCALCOUNT.

Its still the same, you need to change at least 2 variables SRCREV +
PR/LOCALCOUNT.

And if PV changed you change it in both cases.

Only advantage of LOCALCOUNT is that it would be easier to maintain it
in the same file as SRCREV (less error-prone to change them together)

> regards,

regards,

-- 
uin:136542059                jid:Martin.Jansa@gmail.com
Jansa Martin                 sip:jamasip@voip.wengo.fr 
JaMa