[PATCH] Epoch support for bitbake (was: Re: RFC: Xorg Version Numbering)

[PATCH] Epoch support for bitbake (was: Re: RFC: Xorg Version Numbering)

[pHilipp Zabel]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

On 1/31/07, Richard Purdie <rpurdie@rpsys.net> wrote:
> Can anyone remember the specifics of the problem with bitbake? I guess
> its easy enough to test and find out...
>
> /me adds looking at this to his todo list...
> (unless anyone beats me to it ;-)

After some discussion on IRC, I came up with the attached patch.
It adds support for a new integer variable PE (package epoch) in the
version comparison code and bitbake cache.
It also changes the code that handles PREFERRED_VERSION to optionally
allow specifying PE (similar to how it is done with PR already).
Especially the provider.py part is just a quick proof of concept. It
always selects
the package with the highest epoch first if the epoch is not
explicitly mentioned
in PREFERRED_VERSION.

So given we have a provider foo with PE="0" (default), PV="1.0",
PR="r0" (1.0-r0)
and a second provider foo with PE="1", PV="1.0", PR="r0" (1:1.0-r0), setting
PREFERRED_VERSION_foo = "0:1.0_r0" will select foo 1.0-r0, setting
PREFERRED_VERSION_foo = "1:1.0_r0" will select foo 1:1.0-r0, and setting
PREFERRED_VERSION_foo = "1.0" will select foo 1:1.0-r0. The last example
is open to discussion. Should an omitted epoch in PREFERRED_VERSION
be considered as PE="0", or should the provider with the greatest PE
be selected?

regards
Philipp

[-- Attachment #2: bitbake-epoch.patch --]
[-- Type: text/x-patch, Size: 5095 bytes --]

Index: lib/bb/utils.py
===================================================================
--- lib/bb/utils.py	(revision 753)
+++ lib/bb/utils.py	(working copy)
@@ -62,11 +62,13 @@
             return -1
 
 def vercmp(ta, tb):
-    (va, ra) = ta
-    (vb, rb) = tb
+    (ea, va, ra) = ta
+    (eb, vb, rb) = tb
 
-    r = vercmp_part(va, vb)
+    r = ea-eb
     if (r == 0):
+        r = vercmp_part(va, vb)
+    if (r == 0):
         r = vercmp_part(ra, rb)
     return r
 
Index: lib/bb/cache.py
===================================================================
--- lib/bb/cache.py	(revision 753)
+++ lib/bb/cache.py	(working copy)
@@ -39,7 +39,7 @@
     import pickle
     bb.msg.note(1, bb.msg.domain.Cache, "Importing cPickle failed. Falling back to a very slow implementation.")
 
-__cache_version__ = "125"
+__cache_version__ = "126"
 
 class Cache:
     """
@@ -251,6 +251,7 @@
         """
 
         pn       = self.getVar('PN', file_name, True)
+        pe       = int(self.getVar('PE', file_name, True) or "0")
         pv       = self.getVar('PV', file_name, True)
         pr       = self.getVar('PR', file_name, True)
         dp       = int(self.getVar('DEFAULT_PREFERENCE', file_name, True) or "0")
@@ -272,7 +273,7 @@
 
         # build FileName to PackageName lookup table
         cacheData.pkg_fn[file_name] = pn
-        cacheData.pkg_pvpr[file_name] = (pv,pr)
+        cacheData.pkg_pepvpr[file_name] = (pe,pv,pr)
         cacheData.pkg_dp[file_name] = dp
 
         # Build forward and reverse provider hashes
@@ -407,7 +408,7 @@
         self.possible_world = []
         self.pkg_pn = {}
         self.pkg_fn = {}
-        self.pkg_pvpr = {}
+        self.pkg_pepvpr = {}
         self.pkg_dp = {}
         self.pn_provides = {}
         self.all_depends = Set()
Index: lib/bb/providers.py
===================================================================
--- lib/bb/providers.py	(revision 753)
+++ lib/bb/providers.py	(working copy)
@@ -61,19 +61,27 @@
 
     preferred_v = bb.data.getVar('PREFERRED_VERSION_%s' % pn, localdata, True)
     if preferred_v:
-        m = re.match('(.*)_(.*)', preferred_v)
+        m = re.match('(\d+:)*(.*)(_.*)*', preferred_v)
         if m:
-            preferred_v = m.group(1)
-            preferred_r = m.group(2)
+            if m.group(1):
+                preferred_e = int(m.group(1)[:-1])
+            else:
+                preferred_e = None
+            preferred_v = m.group(2)
+            if m.group(3):
+                preferred_r = m.group(3)[1:]
+            else:
+                preferred_r = None
         else:
+            preferred_e = None
             preferred_r = None
 
         for file_set in tmp_pn:
             for f in file_set:
-                pv,pr = dataCache.pkg_pvpr[f]
-                if preferred_v == pv and (preferred_r == pr or preferred_r == None):
+                pe,pv,pr = dataCache.pkg_pepvpr[f]
+                if preferred_v == pv and (preferred_r == pr or preferred_r == None) and (preferred_e == pe or preferred_e == None):
                     preferred_file = f
-                    preferred_ver = (pv, pr)
+                    preferred_ver = (pe, pv, pr)
                     break
             if preferred_file:
                 break;
@@ -81,6 +89,8 @@
             pv_str = '%s-%s' % (preferred_v, preferred_r)
         else:
             pv_str = preferred_v
+        if not (preferred_e is None):
+            pv_str = '%d:%s' % (preferred_e, pv_str)
         itemstr = ""
         if item:
             itemstr = " (for item %s)" % item
@@ -97,11 +107,11 @@
     latest_p = 0
     latest_f = None
     for file_name in files:
-        pv,pr = dataCache.pkg_pvpr[file_name]
+        pe,pv,pr = dataCache.pkg_pepvpr[file_name]
         dp = dataCache.pkg_dp[file_name]
 
-        if (latest is None) or ((latest_p == dp) and (utils.vercmp(latest, (pv, pr)) < 0)) or (dp > latest_p):
-            latest = (pv, pr)
+        if (latest is None) or ((latest_p == dp) and (utils.vercmp(latest, (pe, pv, pr)) < 0)) or (dp > latest_p):
+            latest = (pe, pv, pr)
             latest_f = file_name
             latest_p = dp
     if preferred_file is None:
@@ -162,7 +172,7 @@
     # if so, bump it to the head of the queue
     for p in providers:
         pn = dataCache.pkg_fn[p]
-        pv, pr = dataCache.pkg_pvpr[p]
+        pe, pv, pr = dataCache.pkg_pepvpr[p]
 
         stamp = '%s.do_populate_staging' % dataCache.stamp[p]
         if os.path.exists(stamp):
@@ -171,7 +181,11 @@
                 # package was made ineligible by already-failed check
                 continue
             oldver = "%s-%s" % (pv, pr)
-            newver = '-'.join(newvers)
+            if pe > 0:
+                oldver = "%d:%s" % (pe, oldver)
+            newver = "%s-%s" % (newvers[1], newvers[2])
+            if newvers[0] > 0:
+                newver = "%d:%s" % (newvers[0], newver)
             if (newver != oldver):
                 extra_chat = "%s (%s) already staged but upgrading to %s to satisfy %s" % (pn, oldver, newver, item)
             else:

Re: [PATCH] Epoch support for bitbake (was: Re: RFC: Xorg Version Numbering)

[Richard Purdie]

On Wed, 2007-01-31 at 16:40 +0100, pHilipp Zabel wrote:
> On 1/31/07, Richard Purdie <rpurdie@rpsys.net> wrote:
> > Can anyone remember the specifics of the problem with bitbake? I guess
> > its easy enough to test and find out...
> >
> > /me adds looking at this to his todo list...
> > (unless anyone beats me to it ;-)
> 
> After some discussion on IRC, I came up with the attached patch.
> It adds support for a new integer variable PE (package epoch) in the
> version comparison code and bitbake cache.
> It also changes the code that handles PREFERRED_VERSION to optionally
> allow specifying PE (similar to how it is done with PR already).
> Especially the provider.py part is just a quick proof of concept. It
> always selects
> the package with the highest epoch first if the epoch is not
> explicitly mentioned
> in PREFERRED_VERSION.

Thanks for this, I just checked it in unchanged to bitbake trunk
(1.9.x). If people test it and are happy, I'm happy for it to be
backported for 1.8.2 too, at which point OE.dev can maybe start to rely
upon it.

Cheers,

Richard

Re: [PATCH] Epoch support for bitbake

[Rod Whitby]

Richard Purdie wrote:
> Thanks for this, I just checked it in unchanged to bitbake trunk
> (1.9.x). If people test it and are happy, I'm happy for it to be
> backported for 1.8.2 too, at which point OE.dev can maybe start to rely
> upon it.

Dunno whether this is important or not, but OpenMoko (which is now using
svn overlaying OE .dev, not a frozen OE version) uses bitbake 1.6.x with
custom svnnow patches.

So OE.dev starting to rely on epoch support may (I dunno, just asking
the experts) break OpenMoko until they port their svnnow patches to 1.8
and move everyone building openmoko from the official openmoko svn/OE
distro (not the OE.dev-only "openmoko" distro) over to that.

[Please refrain from comments about what OpenMoko should or should not
do with regard to SVN and OE.dev - that's not the issue here.  Or at
least please use a different subject header if you do.]

-- Rod

Re: [PATCH] Epoch support for bitbake

[Holger Freyther]

Am 20.03.2007 um 01:24 schrieb Rod Whitby:

> Richard Purdie wrote:
>> Thanks for this, I just checked it in unchanged to bitbake trunk
>> (1.9.x). If people test it and are happy, I'm happy for it to be
>> backported for 1.8.2 too, at which point OE.dev can maybe start to  
>> rely
>> upon it.
>
> Dunno whether this is important or not, but OpenMoko (which is now  
> using
> svn overlaying OE .dev, not a frozen OE version) uses bitbake 1.6.x  
> with
> custom svnnow patches.
>
> So OE.dev starting to rely on epoch support may (I dunno, just asking
> the experts) break OpenMoko until they port their svnnow patches to  
> 1.8
> and move everyone building openmoko from the official openmoko svn/OE
> distro (not the OE.dev-only "openmoko" distro) over to that.

Hi Rod,

I assume that henryk will port his work to bitbake trunk/1.8 and it  
is highly likely (actually pretty obvious) that we will accept the  
patch. And at this point it is a potential backporting candidate for  
1.8.2.

z.

Re: [PATCH] Epoch support for bitbake

[Rod Whitby]

Holger Freyther wrote:
> Am 20.03.2007 um 01:24 schrieb Rod Whitby:
>> Richard Purdie wrote:
>>> Thanks for this, I just checked it in unchanged to bitbake trunk
>>> (1.9.x). If people test it and are happy, I'm happy for it to be
>>> backported for 1.8.2 too, at which point OE.dev can maybe start to  
>>> rely
>>> upon it.
>> Dunno whether this is important or not, but OpenMoko (which is now  
>> using svn overlaying OE .dev, not a frozen OE version) uses bitbake 1.6.x  
>> with custom svnnow patches.
> I assume that henryk will port his work to bitbake trunk/1.8 and it  
> is highly likely (actually pretty obvious) that we will accept the  
> patch. And at this point it is a potential backporting candidate for  
> 1.8.2.

So does that mean that changing OE.dev to _rely_ on epoch support should
be held off until that happens?

-- Rod

Re: [PATCH] Epoch support for bitbake

[Richard Purdie]

On Tue, 2007-03-20 at 12:00 +1030, Rod Whitby wrote:
> Holger Freyther wrote:
> > Am 20.03.2007 um 01:24 schrieb Rod Whitby:
> >> Richard Purdie wrote:
> >>> Thanks for this, I just checked it in unchanged to bitbake trunk
> >>> (1.9.x). If people test it and are happy, I'm happy for it to be
> >>> backported for 1.8.2 too, at which point OE.dev can maybe start to  
> >>> rely
> >>> upon it.
> >> Dunno whether this is important or not, but OpenMoko (which is now  
> >> using svn overlaying OE .dev, not a frozen OE version) uses bitbake 1.6.x  
> >> with custom svnnow patches.
> > I assume that henryk will port his work to bitbake trunk/1.8 and it  
> > is highly likely (actually pretty obvious) that we will accept the  
> > patch. And at this point it is a potential backporting candidate for  
> > 1.8.2.
> 
> So does that mean that changing OE.dev to _rely_ on epoch support should
> be held off until that happens?

We won't rely on epoch support until our users have a stable version of
bitbake available to them. "users" includes the openmoko developers. The
indicator will be the bitbake minimum version requirement set in the
sanity checker - once the version that points to has epoch support, we
can rely on it.

I'm hoping omoko will update to use bitbake 1.8.x. If they don't, we
might consider backporting the epoch code to 1.6.x. Lets see how things
work out...

I do want to ultimately solve the X11R7 mess though.

Cheers,

Richard

Re: [PATCH] Epoch support for bitbake

[Holger Freyther]

Am 20.03.2007 um 17:37 schrieb Richard Purdie:

> I'm hoping omoko will update to use bitbake 1.8.x. If they don't, we
> might consider backporting the epoch code to 1.6.x. Lets see how  
> things
> work out...

Upgrade path:
	Make the metadata compile with bitbake-1.8 or verify that it compiles
	Compile everything with the old bitbake and save the image+ipk's
	Recompile everything (exact same source) with the new bitbake-1.8

	Run ipkg-diff from contrib/qa and see what changed. Evaluate changes  
and inform us and go with bitbake-1.8

z.