[PATCH 1/3] bitbake: Add MultiConfigParsed event

[PATCH 1/3] bitbake: Add MultiConfigParsed event

[Richard Purdie]

There are some cases where the metadata needs to be aware a multiconfig build
is happening and have access to the multiconfig data stores to merge data into
the common build. This adds such an event allowing access to these datastores.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cookerdata.py | 2 ++
 lib/bb/event.py      | 6 ++++++
 lib/bb/ui/knotty.py  | 1 +
 3 files changed, 9 insertions(+)

diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index 4d32aa0..644bb38 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -294,6 +294,8 @@ class CookerDataBuilder(object):
                 mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config)
                 bb.event.fire(bb.event.ConfigParsed(), mcdata)
                 self.mcdata[config] = mcdata
+            if multiconfig:
+                bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data)
 
         except (SyntaxError, bb.BBHandledException):
             raise bb.BBHandledException
diff --git a/lib/bb/event.py b/lib/bb/event.py
index 6ccd4bdc..1462382 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -358,6 +358,12 @@ class OperationProgress(Event):
 class ConfigParsed(Event):
     """Configuration Parsing Complete"""
 
+class MultiConfigParsed(Event):
+    """Multi-Config Parsing Complete"""
+    def __init__(self, mcdata):
+        self.mcdata = mcdata
+        Event.__init__(self)
+
 class RecipeEvent(Event):
     def __init__(self, fn):
         self.fn = fn
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index a19c3b3..936d5a4 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -664,6 +664,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
                                   bb.event.MetadataEvent,
                                   bb.event.StampUpdate,
                                   bb.event.ConfigParsed,
+                                  bb.event.MultiConfigParsed,
                                   bb.event.RecipeParsed,
                                   bb.event.RecipePreFinalise,
                                   bb.runqueue.runQueueEvent,
-- 
2.7.4

[PATCH 2/3] cooker: Experiment with multiple BuildStarted for multiconfig

[Richard Purdie]

Currently builds in multiple TMPDIRs with multiconfig can break
since the BuildStarted event is used to create directory strutures in several
cases (e.g. buildstats.bbclass) and there is only on BuildStarted event
generated in a multiconfig build.

We have two options, a) to add a new MultiConfigBuildStarted event which is
generated once per multiconfig, or b) allow multiple BuildStarted events.

Having reviewed the code and current users of BuildStarted, sending one event
per multiconfig seems like its the best way forward and the existing code looks
able to cope with the duplication of events. I did also check toaster and I think
that can handle this issue too (multiconfig builds may have other issues there).

I'm therefore proposing we send multiple BuildStarted events for multiconfig
and for consistency, send multiple BuildCompleted events too.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cooker.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 479dc5a..c140220 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1191,6 +1191,10 @@ class BBCooker:
         if siggen_cache:
             bb.parse.siggen.checksum_cache.mtime_cache.clear()
 
+    def copyBuildVars(self, d):
+        for i in ["BUILDNAME", "BUILDSTART", "DATE", "TIME"]:
+            d.setVar(i, self.data.getVar(i, False))
+
     def matchFiles(self, bf):
         """
         Find the .bb files which match the expression in 'buildfile'.
@@ -1362,7 +1366,10 @@ class BBCooker:
 
             if not retval:
                 try:
-                    bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, targets, failures, interrupted), self.data)
+                    for mc in self.multiconfigs:
+                        localdata = data.createCopy(self.databuilder.mcdata[mc])
+                        self.copyBuildVars(localdata)
+                        bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, targets, failures, interrupted), localdata)
                 finally:
                     self.command.finishAsyncCommand(msg)
                 return False
@@ -1395,7 +1402,10 @@ class BBCooker:
                 ntargets.append("multiconfig:%s:%s:%s" % (target[0], target[1], target[2]))
             ntargets.append("%s:%s" % (target[1], target[2]))
 
-        bb.event.fire(bb.event.BuildStarted(buildname, ntargets), self.data)
+        for mc in self.multiconfigs:
+            localdata = data.createCopy(self.databuilder.mcdata[mc])
+            self.copyBuildVars(localdata)
+            bb.event.fire(bb.event.BuildStarted(buildname, ntargets), localdata)
 
         rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
         if 'universe' in targets:
-- 
2.7.4

[PATCH 3/3] data: Micro performance optimisation tweak

[Richard Purdie]

The datastore can assume internal API, this just removes the function indirection
overhead involved in this very common codepath (800,000 calls in parsing OE-Core).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/data.py b/lib/bb/data.py
index 134afaa..80a7879 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -290,7 +290,7 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
             return deps, value
         varflags = d.getVarFlags(key, ["vardeps", "vardepvalue", "vardepsexclude", "exports", "postfuncs", "prefuncs", "lineno", "filename"]) or {}
         vardeps = varflags.get("vardeps")
-        value = d.getVar(key, False)
+        value = d.getVarFlag(key, "_content", False)
 
         def handle_contains(value, contains, d):
             newvalue = ""
-- 
2.7.4