[PATCH 0/3] runqueue changes

[PATCH 0/3] runqueue changes

[Jason Wessel]

This series is now about the runqueue changes around
invalidating the set scene tasks properly.

It adds new bitbake flags for the purpose of
invalidating setscene tasks.

[PATCH 1/3] bitbake, runqueue: Add --no-setscene to skip all setscene tasks

[Jason Wessel]

Mainly intended for the purpose of debugging or forcing builds
from source, the --no-setscene will prevent any setscene
tasks from running.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 bin/bitbake        |    2 ++
 lib/bb/runqueue.py |   13 +++++++------
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/bin/bitbake b/bin/bitbake
index 478ac06..d178a38 100755
--- a/bin/bitbake
+++ b/bin/bitbake
@@ -171,6 +171,8 @@ Default BBFILES are the .bb files in the current directory.""")
 
     parser.add_option("-B", "--bind", help = "The name/address for the bitbake server to bind to",
                action = "store", dest = "bind", default = False)
+    parser.add_option("", "--no-setscene", help = "Do not run any setscene tasks, forces builds",
+               action = "store_true", dest = "nosetscene", default = False)
     options, args = parser.parse_args(sys.argv)
 
     configuration = BBConfiguration(options)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index d925b4c..6596e83 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -699,13 +699,14 @@ class RunQueueData:
 
         # Interate over the task list looking for tasks with a 'setscene' function
         self.runq_setscene = []
-        for task in range(len(self.runq_fnid)):
-            setscene = taskData.gettask_id(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task] + "_setscene", False)
-            if not setscene:
-                continue
-            self.runq_setscene.append(task)
+        if not self.cooker.configuration.nosetscene:
+            for task in range(len(self.runq_fnid)):
+                setscene = taskData.gettask_id(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task] + "_setscene", False)
+                if not setscene:
+                    continue
+                self.runq_setscene.append(task)
 
-        # Interate over the task list and call into the siggen code
+        # Iterate over the task list and call into the siggen code
         dealtwith = set()
         todeal = set(range(len(self.runq_fnid)))
         while len(todeal) > 0:
-- 
1.7.10

[PATCH 2/3] bitbake, runqueue.py: Add -i to invalidate a stamp and rebuild the target

[Jason Wessel]

It is highly desirable to be able to invalidate a stamp used for
do_compile and to be able to continue on to build the entire
package's default target.

If invalidating a stamp, the setscene rules for the specified targets
should be automatically canceled so as to allow the full set of tasks
depending on any invalidated stap to run for the specified packages.
For example if you wanted to rebuild the "tinylogin" package and see
the results in a final image in one step for the core-image-minimal
you could do so with:

   bitbake -C compile tinylogin core-image-minimal

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 bin/bitbake        |    4 ++++
 lib/bb/runqueue.py |   29 +++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/bin/bitbake b/bin/bitbake
index d178a38..1ac6e8a 100755
--- a/bin/bitbake
+++ b/bin/bitbake
@@ -118,6 +118,10 @@ Default BBFILES are the .bb files in the current directory.""")
     parser.add_option("-c", "--cmd", help = "Specify task to execute. Note that this only executes the specified task for the providee and the packages it depends on, i.e. 'compile' does not implicitly call stage for the dependencies (IOW: use only if you know what you are doing). Depending on the base.bbclass a listtasks tasks is defined and will show available tasks",
                action = "store", dest = "cmd")
 
+    parser.add_option("-C", "--clear-stamp", help = "Invalidate the specified stamp for a task such as 'compile' and run the default task for and specified packages",
+                action = "store", dest = "invalidate_stamp")
+
+
     parser.add_option("-r", "--read", help = "read the specified file before bitbake.conf",
                action = "append", dest = "prefile", default = [])
 
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 6596e83..f1fabfa 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -719,6 +719,28 @@ class RunQueueData:
                         procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
                     self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
 
+        # Remove any setscene tasks from the queue for which we invalidated
+        # a stamp
+        try:
+            new_setscene = []
+            test_fns = []
+            if self.cooker.configuration.invalidate_stamp:
+                for tgt in self.target_pairs:
+                    test_fns.append(tgt[0])
+            for task in self.runq_setscene:
+                try:
+                    fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
+                    if (self.cooker.configuration.invalidate_stamp and fn in test_fns):
+                        logger.debug(2, 'Removing task %s due to invalidated build stamps', task)
+                    else:
+                        new_setscene.append(task)
+                except:
+                    logger.debug(2, 'Failed invalidate check for %s', task)
+            self.runq_setscene = new_setscene
+        except:
+            logger.debug(2, 'Failed to update runq_setscene')
+
+
         self.hashes = {}
         self.hash_deps = {}
         for task in xrange(len(self.runq_fnid)):
@@ -732,6 +754,13 @@ class RunQueueData:
                 deps.append(depidentifier)
             self.hash_deps[identifier] = deps
 
+        # Remove stamps for forced invalidation
+        if self.cooker.configuration.invalidate_stamp:
+            for (fn, target) in self.target_pairs:
+                for st in self.cooker.configuration.invalidate_stamp.split(','):
+                    logger.verbose("Remove stamp %s, %s", st, fn)
+                    bb.build.del_stamp("do_%s" % st, self.dataCache, fn)
+
         # Remove stamps for targets if force mode active
         if self.cooker.configuration.force:
             for (fn, target) in self.target_pairs:
-- 
1.7.10

[PATCH 3/3] runqueue.py, build.py: Invalidate setscene tasks based on do_unpack stamps

[Jason Wessel]

If you have a fully populated sstate cache and have used it to
execute a build, it is not possible to invalidate repackage
an intermediate build after you have forced a compiled

Example when you have build from a complete sstate cache build:
  bitbake core-image-sato
  bitbake -c patch acl
     *** Make some changes to the C files for experimentation.
  bitbake -f -c compile acl
  bitbake acl

The bitbake will refuse to build the acl package at this
point and instead keep populating it from the sstate.  Using
the cleanstate is no longer a good option because it will
also erase the scratch area.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/build.py    |    8 ++++++++
 lib/bb/runqueue.py |    2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index 055d382..9d78764 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -499,6 +499,14 @@ def del_stamp(task, d, file_name = None):
     stamp = stamp_internal(task, d, file_name)
     bb.utils.remove(stamp)
 
+def exists_stamp(task, d, file_name = None):
+    """
+    Removes a stamp for a given task
+    (d can be a data dict or dataCache)
+    """
+    stamp = stamp_internal(task, d, file_name)
+    return os.path.exists(stamp)
+
 def stampfile(taskname, d, file_name = None):
     """
     Return the stamp for a given task
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index f1fabfa..93d36c9 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -730,7 +730,7 @@ class RunQueueData:
             for task in self.runq_setscene:
                 try:
                     fn = self.taskData.fn_index[self.rq.rqdata.runq_fnid[task]]
-                    if (self.cooker.configuration.invalidate_stamp and fn in test_fns):
+                    if bb.build.exists_stamp("do_unpack", self.dataCache, fn) or (self.cooker.configuration.invalidate_stamp and fn in test_fns):
                         logger.debug(2, 'Removing task %s due to invalidated build stamps', task)
                     else:
                         new_setscene.append(task)
-- 
1.7.10