* [PATCH 02/26] runqueue: Drop unused BB_SETSCENE_VERIFY_FUNCTION2
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
@ 2019-07-10 23:53 ` Richard Purdie
2019-07-10 23:53 ` [PATCH 03/26] runqueue: Remove now uneeded code Richard Purdie
` (24 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:53 UTC (permalink / raw)
To: bitbake-devel
Nothing in OE-Core uses this and hasn't since 2017. It wasn't needed by core
metadata since the switch to recipe specific sysroots.
Since this function would be hard to implement with the planned changes to
runqueue, drop it which allows simplification and further code cleanup.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 33 +++------------------------------
1 file changed, 3 insertions(+), 30 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index b19b524e77..19af73cb3c 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1194,7 +1194,6 @@ class RunQueue:
self.stamppolicy = cfgData.getVar("BB_STAMP_POLICY") or "perfile"
self.hashvalidate = cfgData.getVar("BB_HASHCHECK_FUNCTION") or None
- self.setsceneverify = cfgData.getVar("BB_SETSCENE_VERIFY_FUNCTION2") or None
self.depvalidate = cfgData.getVar("BB_SETSCENE_DEPVALID") or None
self.state = runQueuePrepare
@@ -1819,32 +1818,7 @@ class RunQueueExecuteTasks(RunQueueExecute):
found = True
self.rq.scenequeue_covered.add(tid)
- logger.debug(1, 'Skip list (pre setsceneverify) %s', sorted(self.rq.scenequeue_covered))
-
- # Allow the metadata to elect for setscene tasks to run anyway
- covered_remove = set()
- if self.rq.setsceneverify:
- invalidtasks = []
- tasknames = {}
- fns = {}
- for tid in self.rqdata.runtaskentries:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
- fns[tid] = taskfn
- tasknames[tid] = taskname
- if 'noexec' in taskdep and taskname in taskdep['noexec']:
- continue
- if self.rq.check_stamp_task(tid, taskname + "_setscene", cache=self.stampcache):
- logger.debug(2, 'Setscene stamp current for task %s', tid)
- continue
- if self.rq.check_stamp_task(tid, taskname, recurse = True, cache=self.stampcache):
- logger.debug(2, 'Normal stamp current for task %s', tid)
- continue
- invalidtasks.append(tid)
-
- call = self.rq.setsceneverify + "(covered, tasknames, fns, d, invalidtasks=invalidtasks)"
- locs = { "covered" : self.rq.scenequeue_covered, "tasknames" : tasknames, "fns" : fns, "d" : self.cooker.data, "invalidtasks" : invalidtasks }
- covered_remove = bb.utils.better_eval(call, locs)
+ logger.debug(1, 'Skip list %s', sorted(self.rq.scenequeue_covered))
def removecoveredtask(tid):
(mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
@@ -1852,9 +1826,9 @@ class RunQueueExecuteTasks(RunQueueExecute):
bb.build.del_stamp(taskname, self.rqdata.dataCaches[mc], taskfn)
self.rq.scenequeue_covered.remove(tid)
- toremove = covered_remove | self.rq.scenequeue_notcovered
+ toremove = self.rq.scenequeue_notcovered
for task in toremove:
- logger.debug(1, 'Not skipping task %s due to setsceneverify', task)
+ logger.debug(1, 'Not skipping task %s', task)
while toremove:
covered_remove = []
for task in toremove:
@@ -1871,7 +1845,6 @@ class RunQueueExecuteTasks(RunQueueExecute):
logger.debug(1, 'Full skip list %s', self.rq.scenequeue_covered)
-
for mc in self.rqdata.dataCaches:
target_pairs = []
for tid in self.rqdata.target_tids:
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 03/26] runqueue: Remove now uneeded code
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
2019-07-10 23:53 ` [PATCH 02/26] runqueue: Drop unused BB_SETSCENE_VERIFY_FUNCTION2 Richard Purdie
@ 2019-07-10 23:53 ` Richard Purdie
2019-07-10 23:53 ` [PATCH 04/26] runqueue: Move scenequeue data generation to a separate function Richard Purdie
` (23 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:53 UTC (permalink / raw)
To: bitbake-devel
With the removal of the setcene verify code, this additional code block
is also now unneeded since tasks can't be forced at this point in the code
any move. This effectively reverts f21910157d873c030b149c4cdc5b57c5062ab5a6.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 26 +-------------------------
1 file changed, 1 insertion(+), 25 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 19af73cb3c..6ac9fb1678 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1795,8 +1795,6 @@ class RunQueueExecuteTasks(RunQueueExecute):
self.stampcache = {}
- initial_covered = self.rq.scenequeue_covered.copy()
-
# Mark initial buildable tasks
for tid in self.rqdata.runtaskentries:
if len(self.rqdata.runtaskentries[tid].depends) == 0:
@@ -1820,30 +1818,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
logger.debug(1, 'Skip list %s', sorted(self.rq.scenequeue_covered))
- def removecoveredtask(tid):
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- taskname = taskname + '_setscene'
- bb.build.del_stamp(taskname, self.rqdata.dataCaches[mc], taskfn)
- self.rq.scenequeue_covered.remove(tid)
-
- toremove = self.rq.scenequeue_notcovered
- for task in toremove:
+ for task in self.rq.scenequeue_notcovered:
logger.debug(1, 'Not skipping task %s', task)
- while toremove:
- covered_remove = []
- for task in toremove:
- if task in self.rq.scenequeue_covered:
- removecoveredtask(task)
- for deptask in self.rqdata.runtaskentries[task].depends:
- if deptask not in self.rq.scenequeue_covered:
- continue
- if deptask in toremove or deptask in covered_remove or deptask in initial_covered:
- continue
- logger.debug(1, 'Task %s depends on task %s so not skipping' % (task, deptask))
- covered_remove.append(deptask)
- toremove = covered_remove
-
- logger.debug(1, 'Full skip list %s', self.rq.scenequeue_covered)
for mc in self.rqdata.dataCaches:
target_pairs = []
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 04/26] runqueue: Move scenequeue data generation to a separate function
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
2019-07-10 23:53 ` [PATCH 02/26] runqueue: Drop unused BB_SETSCENE_VERIFY_FUNCTION2 Richard Purdie
2019-07-10 23:53 ` [PATCH 03/26] runqueue: Remove now uneeded code Richard Purdie
@ 2019-07-10 23:53 ` Richard Purdie
2019-07-10 23:53 ` [PATCH 05/26] runqueue: Remove unused function parameter Richard Purdie
` (22 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:53 UTC (permalink / raw)
To: bitbake-devel
Move the bulk of the scenequeue data generation to its own function
allowing for refactoring of the code.
Create the start of an object to represent this data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 457 +++++++++++++++++++++++----------------------
1 file changed, 233 insertions(+), 224 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 6ac9fb1678..b3116bce2c 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2080,267 +2080,276 @@ class RunQueueExecuteTasks(RunQueueExecute):
#bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n"))
return taskdepdata
-class RunQueueExecuteScenequeue(RunQueueExecute):
- def __init__(self, rq):
- RunQueueExecute.__init__(self, rq)
-
- self.scenequeue_covered = set()
- self.scenequeue_notcovered = set()
- self.scenequeue_notneeded = set()
-
- # If we don't have any setscene functions, skip this step
- if len(self.rqdata.runq_setscene_tids) == 0:
- rq.scenequeue_covered = set()
- rq.scenequeue_notcovered = set()
- rq.state = runQueueRunInit
- return
-
- self.stats = RunQueueStats(len(self.rqdata.runq_setscene_tids))
-
- sq_revdeps = {}
- sq_revdeps_new = {}
- sq_revdeps_squash = {}
+class SQData(object):
+ def __init__(self):
self.sq_harddeps = {}
self.stamps = {}
+ self.unskippable = []
- # We need to construct a dependency graph for the setscene functions. Intermediate
- # dependencies between the setscene tasks only complicate the code. This code
- # therefore aims to collapse the huge runqueue dependency tree into a smaller one
- # only containing the setscene functions.
+def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
+
+ sq_revdeps = {}
+ sq_revdeps_new = {}
+ sq_revdeps_squash = {}
+
+ # We need to construct a dependency graph for the setscene functions. Intermediate
+ # dependencies between the setscene tasks only complicate the code. This code
+ # therefore aims to collapse the huge runqueue dependency tree into a smaller one
+ # only containing the setscene functions.
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # First process the chains up to the first setscene task.
+ endpoints = {}
+ for tid in rqdata.runtaskentries:
+ sq_revdeps[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps)
+ sq_revdeps_new[tid] = set()
+ if (len(sq_revdeps[tid]) == 0) and tid not in rqdata.runq_setscene_tids:
+ #bb.warn("Added endpoint %s" % (tid))
+ endpoints[tid] = set()
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # Secondly process the chains between setscene tasks.
+ for tid in rqdata.runq_setscene_tids:
+ #bb.warn("Added endpoint 2 %s" % (tid))
+ for dep in rqdata.runtaskentries[tid].depends:
+ if tid in sq_revdeps[dep]:
+ sq_revdeps[dep].remove(tid)
+ if dep not in endpoints:
+ endpoints[dep] = set()
+ #bb.warn(" Added endpoint 3 %s" % (dep))
+ endpoints[dep].add(tid)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ def process_endpoints(endpoints):
+ newendpoints = {}
+ for point, task in endpoints.items():
+ tasks = set()
+ if task:
+ tasks |= task
+ if sq_revdeps_new[point]:
+ tasks |= sq_revdeps_new[point]
+ sq_revdeps_new[point] = set()
+ if point in rqdata.runq_setscene_tids:
+ sq_revdeps_new[point] = tasks
+ tasks = set()
+ continue
+ for dep in rqdata.runtaskentries[point].depends:
+ if point in sq_revdeps[dep]:
+ sq_revdeps[dep].remove(point)
+ if tasks:
+ sq_revdeps_new[dep] |= tasks
+ if len(sq_revdeps[dep]) == 0 and dep not in rqdata.runq_setscene_tids:
+ newendpoints[dep] = task
+ if len(newendpoints) != 0:
+ process_endpoints(newendpoints)
+
+ process_endpoints(endpoints)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # Build a list of setscene tasks which are "unskippable"
+ # These are direct endpoints referenced by the build
+ endpoints2 = {}
+ sq_revdeps2 = {}
+ sq_revdeps_new2 = {}
+ def process_endpoints2(endpoints):
+ newendpoints = {}
+ for point, task in endpoints.items():
+ tasks = set([point])
+ if task:
+ tasks |= task
+ if sq_revdeps_new2[point]:
+ tasks |= sq_revdeps_new2[point]
+ sq_revdeps_new2[point] = set()
+ if point in rqdata.runq_setscene_tids:
+ sq_revdeps_new2[point] = tasks
+ for dep in rqdata.runtaskentries[point].depends:
+ if point in sq_revdeps2[dep]:
+ sq_revdeps2[dep].remove(point)
+ if tasks:
+ sq_revdeps_new2[dep] |= tasks
+ if (len(sq_revdeps2[dep]) == 0 or len(sq_revdeps_new2[dep]) != 0) and dep not in rqdata.runq_setscene_tids:
+ newendpoints[dep] = tasks
+ if len(newendpoints) != 0:
+ process_endpoints2(newendpoints)
+ for tid in rqdata.runtaskentries:
+ sq_revdeps2[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps)
+ sq_revdeps_new2[tid] = set()
+ if (len(sq_revdeps2[tid]) == 0) and tid not in rqdata.runq_setscene_tids:
+ endpoints2[tid] = set()
+ process_endpoints2(endpoints2)
+ for tid in rqdata.runq_setscene_tids:
+ if sq_revdeps_new2[tid]:
+ sqdata.unskippable.append(tid)
+
+ rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries))
+
+ for taskcounter, tid in enumerate(rqdata.runtaskentries):
+ if tid in rqdata.runq_setscene_tids:
+ deps = set()
+ for dep in sq_revdeps_new[tid]:
+ deps.add(dep)
+ sq_revdeps_squash[tid] = deps
+ elif len(sq_revdeps_new[tid]) != 0:
+ bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
+ rqdata.init_progress_reporter.update(taskcounter)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # Resolve setscene inter-task dependencies
+ # e.g. do_sometask_setscene[depends] = "targetname:do_someothertask_setscene"
+ # Note that anything explicitly depended upon will have its reverse dependencies removed to avoid circular dependencies
+ for tid in rqdata.runq_setscene_tids:
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ realtid = tid + "_setscene"
+ idepends = rqdata.taskData[mc].taskentries[realtid].idepends
+ sqdata.stamps[tid] = bb.build.stampfile(taskname + "_setscene", rqdata.dataCaches[mc], taskfn, noextra=True)
+ for (depname, idependtask) in idepends:
- self.rqdata.init_progress_reporter.next_stage()
+ if depname not in rqdata.taskData[mc].build_targets:
+ continue
- # First process the chains up to the first setscene task.
- endpoints = {}
- for tid in self.rqdata.runtaskentries:
- sq_revdeps[tid] = copy.copy(self.rqdata.runtaskentries[tid].revdeps)
- sq_revdeps_new[tid] = set()
- if (len(sq_revdeps[tid]) == 0) and tid not in self.rqdata.runq_setscene_tids:
- #bb.warn("Added endpoint %s" % (tid))
- endpoints[tid] = set()
+ depfn = rqdata.taskData[mc].build_targets[depname][0]
+ if depfn is None:
+ continue
+ deptid = depfn + ":" + idependtask.replace("_setscene", "")
+ if deptid not in rqdata.runtaskentries:
+ bb.msg.fatal("RunQueue", "Task %s depends upon non-existent task %s:%s" % (realtid, depfn, idependtask))
- self.rqdata.init_progress_reporter.next_stage()
+ if not deptid in sqdata.sq_harddeps:
+ sqdata.sq_harddeps[deptid] = set()
+ sqdata.sq_harddeps[deptid].add(tid)
- # Secondly process the chains between setscene tasks.
- for tid in self.rqdata.runq_setscene_tids:
- #bb.warn("Added endpoint 2 %s" % (tid))
- for dep in self.rqdata.runtaskentries[tid].depends:
- if tid in sq_revdeps[dep]:
- sq_revdeps[dep].remove(tid)
- if dep not in endpoints:
- endpoints[dep] = set()
- #bb.warn(" Added endpoint 3 %s" % (dep))
- endpoints[dep].add(tid)
-
- self.rqdata.init_progress_reporter.next_stage()
-
- def process_endpoints(endpoints):
- newendpoints = {}
- for point, task in endpoints.items():
- tasks = set()
- if task:
- tasks |= task
- if sq_revdeps_new[point]:
- tasks |= sq_revdeps_new[point]
- sq_revdeps_new[point] = set()
- if point in self.rqdata.runq_setscene_tids:
- sq_revdeps_new[point] = tasks
- tasks = set()
- continue
- for dep in self.rqdata.runtaskentries[point].depends:
- if point in sq_revdeps[dep]:
- sq_revdeps[dep].remove(point)
- if tasks:
- sq_revdeps_new[dep] |= tasks
- if len(sq_revdeps[dep]) == 0 and dep not in self.rqdata.runq_setscene_tids:
- newendpoints[dep] = task
- if len(newendpoints) != 0:
- process_endpoints(newendpoints)
-
- process_endpoints(endpoints)
-
- self.rqdata.init_progress_reporter.next_stage()
-
- # Build a list of setscene tasks which are "unskippable"
- # These are direct endpoints referenced by the build
- endpoints2 = {}
- sq_revdeps2 = {}
- sq_revdeps_new2 = {}
- def process_endpoints2(endpoints):
- newendpoints = {}
- for point, task in endpoints.items():
- tasks = set([point])
- if task:
- tasks |= task
- if sq_revdeps_new2[point]:
- tasks |= sq_revdeps_new2[point]
- sq_revdeps_new2[point] = set()
- if point in self.rqdata.runq_setscene_tids:
- sq_revdeps_new2[point] = tasks
- for dep in self.rqdata.runtaskentries[point].depends:
- if point in sq_revdeps2[dep]:
- sq_revdeps2[dep].remove(point)
- if tasks:
- sq_revdeps_new2[dep] |= tasks
- if (len(sq_revdeps2[dep]) == 0 or len(sq_revdeps_new2[dep]) != 0) and dep not in self.rqdata.runq_setscene_tids:
- newendpoints[dep] = tasks
- if len(newendpoints) != 0:
- process_endpoints2(newendpoints)
- for tid in self.rqdata.runtaskentries:
- sq_revdeps2[tid] = copy.copy(self.rqdata.runtaskentries[tid].revdeps)
- sq_revdeps_new2[tid] = set()
- if (len(sq_revdeps2[tid]) == 0) and tid not in self.rqdata.runq_setscene_tids:
- endpoints2[tid] = set()
- process_endpoints2(endpoints2)
- self.unskippable = []
- for tid in self.rqdata.runq_setscene_tids:
- if sq_revdeps_new2[tid]:
- self.unskippable.append(tid)
+ sq_revdeps_squash[tid].add(deptid)
+ # Have to zero this to avoid circular dependencies
+ sq_revdeps_squash[deptid] = set()
- self.rqdata.init_progress_reporter.next_stage(len(self.rqdata.runtaskentries))
+ rqdata.init_progress_reporter.next_stage()
- for taskcounter, tid in enumerate(self.rqdata.runtaskentries):
- if tid in self.rqdata.runq_setscene_tids:
- deps = set()
- for dep in sq_revdeps_new[tid]:
- deps.add(dep)
- sq_revdeps_squash[tid] = deps
- elif len(sq_revdeps_new[tid]) != 0:
- bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
- self.rqdata.init_progress_reporter.update(taskcounter)
-
- self.rqdata.init_progress_reporter.next_stage()
-
- # Resolve setscene inter-task dependencies
- # e.g. do_sometask_setscene[depends] = "targetname:do_someothertask_setscene"
- # Note that anything explicitly depended upon will have its reverse dependencies removed to avoid circular dependencies
- for tid in self.rqdata.runq_setscene_tids:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- realtid = tid + "_setscene"
- idepends = self.rqdata.taskData[mc].taskentries[realtid].idepends
- self.stamps[tid] = bb.build.stampfile(taskname + "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
- for (depname, idependtask) in idepends:
+ for task in sqdata.sq_harddeps:
+ for dep in sqdata.sq_harddeps[task]:
+ sq_revdeps_squash[dep].add(task)
- if depname not in self.rqdata.taskData[mc].build_targets:
- continue
+ rqdata.init_progress_reporter.next_stage()
- depfn = self.rqdata.taskData[mc].build_targets[depname][0]
- if depfn is None:
- continue
- deptid = depfn + ":" + idependtask.replace("_setscene", "")
- if deptid not in self.rqdata.runtaskentries:
- bb.msg.fatal("RunQueue", "Task %s depends upon non-existent task %s:%s" % (realtid, depfn, idependtask))
+ #for tid in sq_revdeps_squash:
+ # for dep in sq_revdeps_squash[tid]:
+ # data = data + "\n %s" % dep
+ # bb.warn("Task %s_setscene: is %s " % (tid, data
- if not deptid in self.sq_harddeps:
- self.sq_harddeps[deptid] = set()
- self.sq_harddeps[deptid].add(tid)
+ sqdata.sq_deps = {}
+ sqdata.sq_revdeps = sq_revdeps_squash
+ sqdata.sq_revdeps2 = copy.deepcopy(sqdata.sq_revdeps)
- sq_revdeps_squash[tid].add(deptid)
- # Have to zero this to avoid circular dependencies
- sq_revdeps_squash[deptid] = set()
+ for tid in sqdata.sq_revdeps:
+ sqdata.sq_deps[tid] = set()
+ for tid in sqdata.sq_revdeps:
+ for dep in sqdata.sq_revdeps[tid]:
+ sqdata.sq_deps[dep].add(tid)
- self.rqdata.init_progress_reporter.next_stage()
+ rqdata.init_progress_reporter.next_stage()
- for task in self.sq_harddeps:
- for dep in self.sq_harddeps[task]:
- sq_revdeps_squash[dep].add(task)
+ for tid in sqdata.sq_revdeps:
+ if len(sqdata.sq_revdeps[tid]) == 0:
+ sqrq.runq_buildable.add(tid)
- self.rqdata.init_progress_reporter.next_stage()
+ rqdata.init_progress_reporter.finish()
- #for tid in sq_revdeps_squash:
- # for dep in sq_revdeps_squash[tid]:
- # data = data + "\n %s" % dep
- # bb.warn("Task %s_setscene: is %s " % (tid, data
+ sqdata.outrightfail = []
+ if rq.hashvalidate:
+ sq_hash = []
+ sq_hashfn = []
+ sq_unihash = []
+ sq_fn = []
+ sq_taskname = []
+ sq_task = []
+ noexec = []
+ stamppresent = []
+ for tid in sqdata.sq_revdeps:
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- self.sq_deps = {}
- self.sq_revdeps = sq_revdeps_squash
- self.sq_revdeps2 = copy.deepcopy(self.sq_revdeps)
+ taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
- for tid in self.sq_revdeps:
- self.sq_deps[tid] = set()
- for tid in self.sq_revdeps:
- for dep in self.sq_revdeps[tid]:
- self.sq_deps[dep].add(tid)
+ if 'noexec' in taskdep and taskname in taskdep['noexec']:
+ noexec.append(tid)
+ sqrq.task_skip(tid)
+ bb.build.make_stamp(taskname + "_setscene", rqdata.dataCaches[mc], taskfn)
+ continue
- self.rqdata.init_progress_reporter.next_stage()
+ if rq.check_stamp_task(tid, taskname + "_setscene", cache=stampcache):
+ logger.debug(2, 'Setscene stamp current for task %s', tid)
+ stamppresent.append(tid)
+ sqrq.task_skip(tid)
+ continue
- for tid in self.sq_revdeps:
- if len(self.sq_revdeps[tid]) == 0:
- self.runq_buildable.add(tid)
+ if rq.check_stamp_task(tid, taskname, recurse = True, cache=stampcache):
+ logger.debug(2, 'Normal stamp current for task %s', tid)
+ stamppresent.append(tid)
+ sqrq.task_skip(tid)
+ continue
- self.rqdata.init_progress_reporter.finish()
-
- self.outrightfail = []
- if self.rq.hashvalidate:
- sq_hash = []
- sq_hashfn = []
- sq_unihash = []
- sq_fn = []
- sq_taskname = []
- sq_task = []
- noexec = []
- stamppresent = []
- for tid in self.sq_revdeps:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ sq_fn.append(fn)
+ sq_hashfn.append(rqdata.dataCaches[mc].hashfn[taskfn])
+ sq_hash.append(rqdata.runtaskentries[tid].hash)
+ sq_unihash.append(rqdata.runtaskentries[tid].unihash)
+ sq_taskname.append(taskname)
+ sq_task.append(tid)
- taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
+ cooker.data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT", len(stamppresent))
- if 'noexec' in taskdep and taskname in taskdep['noexec']:
- noexec.append(tid)
- self.task_skip(tid)
- bb.build.make_stamp(taskname + "_setscene", self.rqdata.dataCaches[mc], taskfn)
- continue
+ valid = rq.validate_hash(sq_fn=sq_fn, sq_task=sq_taskname, sq_hash=sq_hash, sq_hashfn=sq_hashfn,
+ siginfo=False, sq_unihash=sq_unihash, d=cooker.data)
- if self.rq.check_stamp_task(tid, taskname + "_setscene", cache=self.stampcache):
- logger.debug(2, 'Setscene stamp current for task %s', tid)
- stamppresent.append(tid)
- self.task_skip(tid)
- continue
+ cooker.data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT")
- if self.rq.check_stamp_task(tid, taskname, recurse = True, cache=self.stampcache):
- logger.debug(2, 'Normal stamp current for task %s', tid)
- stamppresent.append(tid)
- self.task_skip(tid)
- continue
+ valid_new = stamppresent
+ for v in valid:
+ valid_new.append(sq_task[v])
- sq_fn.append(fn)
- sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
- sq_hash.append(self.rqdata.runtaskentries[tid].hash)
- sq_unihash.append(self.rqdata.runtaskentries[tid].unihash)
- sq_taskname.append(taskname)
- sq_task.append(tid)
+ for tid in sqdata.sq_revdeps:
+ if tid not in valid_new and tid not in noexec:
+ logger.debug(2, 'No package found, so skipping setscene task %s', tid)
+ sqdata.outrightfail.append(tid)
- self.cooker.data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT", len(stamppresent))
- valid = self.rq.validate_hash(sq_fn=sq_fn, sq_task=sq_taskname, sq_hash=sq_hash, sq_hashfn=sq_hashfn,
- siginfo=False, sq_unihash=sq_unihash, d=self.cooker.data)
+class RunQueueExecuteScenequeue(RunQueueExecute):
+ def __init__(self, rq):
+ RunQueueExecute.__init__(self, rq)
- self.cooker.data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT")
+ self.scenequeue_covered = set()
+ self.scenequeue_notcovered = set()
+ self.scenequeue_notneeded = set()
- valid_new = stamppresent
- for v in valid:
- valid_new.append(sq_task[v])
+ # If we don't have any setscene functions, skip this step
+ if len(self.rqdata.runq_setscene_tids) == 0:
+ rq.scenequeue_covered = set()
+ rq.scenequeue_notcovered = set()
+ rq.state = runQueueRunInit
+ return
+
+ self.stats = RunQueueStats(len(self.rqdata.runq_setscene_tids))
- for tid in self.sq_revdeps:
- if tid not in valid_new and tid not in noexec:
- logger.debug(2, 'No package found, so skipping setscene task %s', tid)
- self.outrightfail.append(tid)
+ self.sqdata = SQData()
+ build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self)
logger.info('Executing SetScene Tasks')
self.rq.state = runQueueSceneRun
def scenequeue_updatecounters(self, task, fail = False):
- for dep in self.sq_deps[task]:
- if fail and task in self.sq_harddeps and dep in self.sq_harddeps[task]:
+ for dep in self.sqdata.sq_deps[task]:
+ if fail and task in self.sqdata.sq_harddeps and dep in self.sqdata.sq_harddeps[task]:
logger.debug(2, "%s was unavailable and is a hard dependency of %s so skipping" % (task, dep))
self.scenequeue_updatecounters(dep, fail)
continue
- if task not in self.sq_revdeps2[dep]:
+ if task not in self.sqdata.sq_revdeps2[dep]:
# May already have been removed by the fail case above
continue
- self.sq_revdeps2[dep].remove(task)
- if len(self.sq_revdeps2[dep]) == 0:
+ self.sqdata.sq_revdeps2[dep].remove(task)
+ if len(self.sqdata.sq_revdeps2[dep]) == 0:
self.runq_buildable.add(dep)
def task_completeoutright(self, task):
@@ -2401,10 +2410,10 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
if self.can_start_task():
# Find the next setscene to run
for nexttask in self.rqdata.runq_setscene_tids:
- if nexttask in self.runq_buildable and nexttask not in self.runq_running and self.stamps[nexttask] not in self.build_stamps.values():
- if nexttask in self.unskippable:
+ if nexttask in self.runq_buildable and nexttask not in self.runq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
+ if nexttask in self.sqdata.unskippable:
logger.debug(2, "Setscene task %s is unskippable" % nexttask)
- if nexttask not in self.unskippable and len(self.sq_revdeps[nexttask]) > 0 and self.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sq_revdeps[nexttask], True):
+ if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask], True):
fn = fn_from_tid(nexttask)
foundtarget = False
@@ -2415,7 +2424,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.task_skip(nexttask)
self.scenequeue_notneeded.add(nexttask)
return True
- if nexttask in self.outrightfail:
+ if nexttask in self.sqdata.outrightfail:
self.task_failoutright(nexttask)
return True
task = nexttask
@@ -2471,10 +2480,10 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.rq.read_workers()
return self.rq.active_fds()
- #for tid in self.sq_revdeps:
+ #for tid in self.sqdata.sq_revdeps:
# if tid not in self.runq_running:
# buildable = tid in self.runq_buildable
- # revdeps = self.sq_revdeps[tid]
+ # revdeps = self.sqdata.sq_revdeps[tid]
# bb.warn("Found we didn't run %s %s %s" % (tid, buildable, str(revdeps)))
self.rq.scenequeue_covered = self.scenequeue_covered
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 05/26] runqueue: Remove unused function parameter
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (2 preceding siblings ...)
2019-07-10 23:53 ` [PATCH 04/26] runqueue: Move scenequeue data generation to a separate function Richard Purdie
@ 2019-07-10 23:53 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 06/26] runqueue: Factor out the process_setscene_whitelist checks Richard Purdie
` (21 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:53 UTC (permalink / raw)
To: bitbake-devel
The function is only used by setscene code so the parameter is pointless,
remove it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index b3116bce2c..b6943f238f 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1759,7 +1759,8 @@ class RunQueueExecute:
self.rq.state = runQueueComplete
return True
- def check_dependencies(self, task, taskdeps, setscene = False):
+ # Used by setscene only
+ def check_dependencies(self, task, taskdeps):
if not self.rq.depvalidate:
return False
@@ -2413,7 +2414,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
if nexttask in self.runq_buildable and nexttask not in self.runq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
if nexttask in self.sqdata.unskippable:
logger.debug(2, "Setscene task %s is unskippable" % nexttask)
- if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask], True):
+ if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]):
fn = fn_from_tid(nexttask)
foundtarget = False
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 06/26] runqueue: Factor out the process_setscene_whitelist checks
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (3 preceding siblings ...)
2019-07-10 23:53 ` [PATCH 05/26] runqueue: Remove unused function parameter Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 07/26] runqueue: Uniquely namespace the scenequeue functions Richard Purdie
` (20 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
For ease of refactoring, move this code to its own separate function
until it becomes clear what we should do with it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 85 ++++++++++++++++++++++++----------------------
1 file changed, 45 insertions(+), 40 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index b6943f238f..b8b35ecabe 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1679,6 +1679,50 @@ class RunQueue:
output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb)
bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, closest matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output))
+def process_setscene_whitelist(rq, rqdata, stampcache, sched, rqex):
+ # Check tasks that are going to run against the whitelist
+ def check_norun_task(tid, showerror=False):
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ # Ignore covered tasks
+ if tid in rq.scenequeue_covered:
+ return False
+ # Ignore stamped tasks
+ if rq.check_stamp_task(tid, taskname, cache=stampcache):
+ return False
+ # Ignore noexec tasks
+ taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
+ if 'noexec' in taskdep and taskname in taskdep['noexec']:
+ return False
+
+ pn = rqdata.dataCaches[mc].pkg_fn[taskfn]
+ if not check_setscene_enforce_whitelist(pn, taskname, rqdata.setscenewhitelist):
+ if showerror:
+ if tid in rqdata.runq_setscene_tids:
+ logger.error('Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname))
+ else:
+ logger.error('Task %s.%s attempted to execute unexpectedly' % (pn, taskname))
+ return True
+ return False
+ # Look to see if any tasks that we think shouldn't run are going to
+ unexpected = False
+ for tid in rqdata.runtaskentries:
+ if check_norun_task(tid):
+ unexpected = True
+ break
+ if unexpected:
+ # Run through the tasks in the rough order they'd have executed and print errors
+ # (since the order can be useful - usually missing sstate for the last few tasks
+ # is the cause of the problem)
+ task = sched.next()
+ while task is not None:
+ check_norun_task(task, showerror=True)
+ rqex.task_skip(task, 'Setscene enforcement check')
+ task = sched.next()
+
+ rq.state = runQueueCleanUp
+ return True
+
+
class RunQueueExecute:
def __init__(self, rq):
@@ -1920,46 +1964,7 @@ class RunQueueExecuteTasks(RunQueueExecute):
if self.rqdata.setscenewhitelist is not None and not self.rqdata.setscenewhitelist_checked:
self.rqdata.setscenewhitelist_checked = True
- # Check tasks that are going to run against the whitelist
- def check_norun_task(tid, showerror=False):
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- # Ignore covered tasks
- if tid in self.rq.scenequeue_covered:
- return False
- # Ignore stamped tasks
- if self.rq.check_stamp_task(tid, taskname, cache=self.stampcache):
- return False
- # Ignore noexec tasks
- taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
- if 'noexec' in taskdep and taskname in taskdep['noexec']:
- return False
-
- pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
- if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
- if showerror:
- if tid in self.rqdata.runq_setscene_tids:
- logger.error('Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname))
- else:
- logger.error('Task %s.%s attempted to execute unexpectedly' % (pn, taskname))
- return True
- return False
- # Look to see if any tasks that we think shouldn't run are going to
- unexpected = False
- for tid in self.rqdata.runtaskentries:
- if check_norun_task(tid):
- unexpected = True
- break
- if unexpected:
- # Run through the tasks in the rough order they'd have executed and print errors
- # (since the order can be useful - usually missing sstate for the last few tasks
- # is the cause of the problem)
- task = self.sched.next()
- while task is not None:
- check_norun_task(task, showerror=True)
- self.task_skip(task, 'Setscene enforcement check')
- task = self.sched.next()
-
- self.rq.state = runQueueCleanUp
+ if process_setscenewhitelist(self.rq, self.rqdata, self.stampcache, self.sched, self):
return True
self.rq.read_workers()
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 07/26] runqueue: Uniquely namespace the scenequeue functions
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (4 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 06/26] runqueue: Factor out the process_setscene_whitelist checks Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 08/26] runqueue: Merge stats handling together for setscene/real tasks Richard Purdie
` (19 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
In preparation for merging the setscene and normal task execution,
uniquely namespace the scenequeue specific functions.
For the one shared function, add the "sq_live" variable so we know
which functions to send the results to.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 86 +++++++++++++++++++++++++---------------------
1 file changed, 47 insertions(+), 39 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index b8b35ecabe..c512920a6d 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1452,7 +1452,7 @@ class RunQueue:
self.rqexe = RunQueueExecuteScenequeue(self)
if self.state is runQueueSceneRun:
- retval = self.rqexe.execute()
+ retval = self.rqexe.sq_execute()
if self.state is runQueueRunInit:
if self.cooker.configuration.setsceneonly:
@@ -1734,6 +1734,10 @@ class RunQueueExecute:
self.number_tasks = int(self.cfgData.getVar("BB_NUMBER_THREADS") or 1)
self.scheduler = self.cfgData.getVar("BB_SCHEDULER") or "speed"
+ self.sq_buildable = set()
+ self.sq_running = set()
+ self.sq_live = set()
+
self.runq_buildable = set()
self.runq_running = set()
self.runq_complete = set()
@@ -1759,10 +1763,17 @@ class RunQueueExecute:
self.build_stamps2.remove(self.build_stamps[task])
del self.build_stamps[task]
- if status != 0:
- self.task_fail(task, status)
+ if task in self.sq_live:
+ if status != 0:
+ self.sq_task_fail(task, status)
+ else:
+ self.sq_task_complete(task)
+ self.sq_live.remove(task)
else:
- self.task_complete(task)
+ if status != 0:
+ self.task_fail(task, status)
+ else:
+ self.task_complete(task)
return True
def finish_now(self):
@@ -2260,7 +2271,7 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
for tid in sqdata.sq_revdeps:
if len(sqdata.sq_revdeps[tid]) == 0:
- sqrq.runq_buildable.add(tid)
+ sqrq.sq_buildable.add(tid)
rqdata.init_progress_reporter.finish()
@@ -2281,20 +2292,20 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
if 'noexec' in taskdep and taskname in taskdep['noexec']:
noexec.append(tid)
- sqrq.task_skip(tid)
+ sqrq.sq_task_skip(tid)
bb.build.make_stamp(taskname + "_setscene", rqdata.dataCaches[mc], taskfn)
continue
if rq.check_stamp_task(tid, taskname + "_setscene", cache=stampcache):
logger.debug(2, 'Setscene stamp current for task %s', tid)
stamppresent.append(tid)
- sqrq.task_skip(tid)
+ sqrq.sq_task_skip(tid)
continue
if rq.check_stamp_task(tid, taskname, recurse = True, cache=stampcache):
logger.debug(2, 'Normal stamp current for task %s', tid)
stamppresent.append(tid)
- sqrq.task_skip(tid)
+ sqrq.sq_task_skip(tid)
continue
sq_fn.append(fn)
@@ -2356,9 +2367,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
continue
self.sqdata.sq_revdeps2[dep].remove(task)
if len(self.sqdata.sq_revdeps2[dep]) == 0:
- self.runq_buildable.add(dep)
+ self.sq_buildable.add(dep)
- def task_completeoutright(self, task):
+ def sq_task_completeoutright(self, task):
"""
Mark a task as completed
Look at the reverse dependencies and mark any task with
@@ -2369,7 +2380,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.scenequeue_covered.add(task)
self.scenequeue_updatecounters(task)
- def check_taskfail(self, task):
+ def sq_check_taskfail(self, task):
if self.rqdata.setscenewhitelist is not None:
realtask = task.split('_setscene')[0]
(mc, fn, taskname, taskfn) = split_tid_mcfn(realtask)
@@ -2378,34 +2389,34 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
logger.error('Task %s.%s failed' % (pn, taskname + "_setscene"))
self.rq.state = runQueueCleanUp
- def task_complete(self, task):
+ def sq_task_complete(self, task):
self.stats.taskCompleted()
bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
- self.task_completeoutright(task)
+ self.sq_task_completeoutright(task)
- def task_fail(self, task, result):
+ def sq_task_fail(self, task, result):
self.stats.taskFailed()
bb.event.fire(sceneQueueTaskFailed(task, self.stats, result, self), self.cfgData)
self.scenequeue_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
- self.check_taskfail(task)
+ self.sq_check_taskfail(task)
- def task_failoutright(self, task):
- self.runq_running.add(task)
- self.runq_buildable.add(task)
+ def sq_task_failoutright(self, task):
+ self.sq_running.add(task)
+ self.sq_buildable.add(task)
self.stats.taskSkipped()
self.stats.taskCompleted()
self.scenequeue_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
- def task_skip(self, task):
- self.runq_running.add(task)
- self.runq_buildable.add(task)
- self.task_completeoutright(task)
+ def sq_task_skip(self, task):
+ self.sq_running.add(task)
+ self.sq_buildable.add(task)
+ self.sq_task_completeoutright(task)
self.stats.taskSkipped()
self.stats.taskCompleted()
- def execute(self):
+ def sq_execute(self):
"""
Run the tasks in a queue prepared by prepare_runqueue
"""
@@ -2416,7 +2427,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
if self.can_start_task():
# Find the next setscene to run
for nexttask in self.rqdata.runq_setscene_tids:
- if nexttask in self.runq_buildable and nexttask not in self.runq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
+ if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
if nexttask in self.sqdata.unskippable:
logger.debug(2, "Setscene task %s is unskippable" % nexttask)
if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]):
@@ -2427,11 +2438,11 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
foundtarget = True
if not foundtarget:
logger.debug(2, "Skipping setscene for task %s" % nexttask)
- self.task_skip(nexttask)
+ self.sq_task_skip(nexttask)
self.scenequeue_notneeded.add(nexttask)
return True
if nexttask in self.sqdata.outrightfail:
- self.task_failoutright(nexttask)
+ self.sq_task_failoutright(nexttask)
return True
task = nexttask
break
@@ -2440,28 +2451,28 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
taskname = taskname + "_setscene"
if self.rq.check_stamp_task(task, taskname_from_tid(task), recurse = True, cache=self.stampcache):
logger.debug(2, 'Stamp for underlying task %s is current, so skipping setscene variant', task)
- self.task_failoutright(task)
+ self.sq_task_failoutright(task)
return True
if self.cooker.configuration.force:
if task in self.rqdata.target_tids:
- self.task_failoutright(task)
+ self.sq_task_failoutright(task)
return True
if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
logger.debug(2, 'Setscene stamp current task %s, so skip it and its dependencies', task)
- self.task_skip(task)
+ self.sq_task_skip(task)
return True
if self.cooker.configuration.skipsetscene:
logger.debug(2, 'No setscene tasks should be executed. Skipping %s', task)
- self.task_failoutright(task)
+ self.sq_task_failoutright(task)
return True
startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
bb.event.fire(startevent, self.cfgData)
- taskdepdata = self.build_taskdepdata(task)
+ taskdepdata = self.sq_build_taskdepdata(task)
taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
taskhash = self.rqdata.get_task_hash(task)
@@ -2477,7 +2488,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True)
self.build_stamps2.append(self.build_stamps[task])
- self.runq_running.add(task)
+ self.sq_running.add(task)
+ self.sq_live.add(task)
self.stats.taskActive()
if self.can_start_task():
return True
@@ -2487,8 +2499,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
return self.rq.active_fds()
#for tid in self.sqdata.sq_revdeps:
- # if tid not in self.runq_running:
- # buildable = tid in self.runq_buildable
+ # if tid not in self.sq_running:
+ # buildable = tid in self.sq_buildable
# revdeps = self.sqdata.sq_revdeps[tid]
# bb.warn("Found we didn't run %s %s %s" % (tid, buildable, str(revdeps)))
@@ -2504,11 +2516,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
return True
- def runqueue_process_waitpid(self, task, status):
- RunQueueExecute.runqueue_process_waitpid(self, task, status)
-
-
- def build_taskdepdata(self, task):
+ def sq_build_taskdepdata(self, task):
def getsetscenedeps(tid):
deps = set()
(mc, fn, taskname, _) = split_tid_mcfn(tid)
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 08/26] runqueue: Merge stats handling together for setscene/real tasks
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (5 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 07/26] runqueue: Uniquely namespace the scenequeue functions Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 09/26] runqueue: Merge scenequeue and real task queue code together Richard Purdie
` (18 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Use a seperate stats class for scenequeue tasks and move the setup
into the base class. Update references accordingly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 39 ++++++++++++++++++++-------------------
1 file changed, 20 insertions(+), 19 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index c512920a6d..7de27ec315 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1748,6 +1748,9 @@ class RunQueueExecute:
self.stampcache = {}
+ self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
+ self.sq_stats = RunQueueStats(len(self.rqdata.runq_setscene_tids))
+
for mc in rq.worker:
rq.worker[mc].pipe.setrunqueueexec(self)
for mc in rq.fakeworker:
@@ -1802,8 +1805,9 @@ class RunQueueExecute:
def finish(self):
self.rq.state = runQueueCleanUp
- if self.stats.active > 0:
- bb.event.fire(runQueueExitWait(self.stats.active), self.cfgData)
+ active = self.stats.active + self.sq_stats.active
+ if active > 0:
+ bb.event.fire(runQueueExitWait(active), self.cfgData)
self.rq.read_workers()
return self.rq.active_fds()
@@ -1831,7 +1835,8 @@ class RunQueueExecute:
return valid
def can_start_task(self):
- can_start = self.stats.active < self.number_tasks
+ active = self.stats.active + self.sq_stats.active
+ can_start = active < self.number_tasks
return can_start
class RunQueueExecuteDummy(RunQueueExecute):
@@ -1847,8 +1852,6 @@ class RunQueueExecuteTasks(RunQueueExecute):
def __init__(self, rq):
RunQueueExecute.__init__(self, rq)
- self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
-
self.stampcache = {}
# Mark initial buildable tasks
@@ -2347,8 +2350,6 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
rq.state = runQueueRunInit
return
- self.stats = RunQueueStats(len(self.rqdata.runq_setscene_tids))
-
self.sqdata = SQData()
build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self)
@@ -2390,13 +2391,13 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.rq.state = runQueueCleanUp
def sq_task_complete(self, task):
- self.stats.taskCompleted()
- bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
+ self.sq_stats.taskCompleted()
+ bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats, self.rq), self.cfgData)
self.sq_task_completeoutright(task)
def sq_task_fail(self, task, result):
- self.stats.taskFailed()
- bb.event.fire(sceneQueueTaskFailed(task, self.stats, result, self), self.cfgData)
+ self.sq_stats.taskFailed()
+ bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats, result, self), self.cfgData)
self.scenequeue_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
self.sq_check_taskfail(task)
@@ -2404,8 +2405,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
def sq_task_failoutright(self, task):
self.sq_running.add(task)
self.sq_buildable.add(task)
- self.stats.taskSkipped()
- self.stats.taskCompleted()
+ self.sq_stats.taskSkipped()
+ self.sq_stats.taskCompleted()
self.scenequeue_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
@@ -2413,8 +2414,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.sq_running.add(task)
self.sq_buildable.add(task)
self.sq_task_completeoutright(task)
- self.stats.taskSkipped()
- self.stats.taskCompleted()
+ self.sq_stats.taskSkipped()
+ self.sq_stats.taskCompleted()
def sq_execute(self):
"""
@@ -2469,7 +2470,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.sq_task_failoutright(task)
return True
- startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
+ startevent = sceneQueueTaskStarted(task, self.sq_stats, self.rq)
bb.event.fire(startevent, self.cfgData)
taskdepdata = self.sq_build_taskdepdata(task)
@@ -2490,11 +2491,11 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.build_stamps2.append(self.build_stamps[task])
self.sq_running.add(task)
self.sq_live.add(task)
- self.stats.taskActive()
+ self.sq_stats.taskActive()
if self.can_start_task():
return True
- if self.stats.active > 0:
+ if self.sq_stats.active > 0:
self.rq.read_workers()
return self.rq.active_fds()
@@ -2511,7 +2512,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.rq.state = runQueueRunInit
- completeevent = sceneQueueComplete(self.stats, self.rq)
+ completeevent = sceneQueueComplete(self.sq_stats, self.rq)
bb.event.fire(completeevent, self.cfgData)
return True
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 09/26] runqueue: Merge scenequeue and real task queue code together
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (6 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 08/26] runqueue: Merge stats handling together for setscene/real tasks Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 10/26] runqueue: Fix counter/task updating glitch Richard Purdie
` (17 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Merge the unique functions from the Tasks and Scenequeue Tasks classes
into the common base class.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 720 ++++++++++++++++++++++-----------------------
1 file changed, 360 insertions(+), 360 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 7de27ec315..9ec2f7d2d4 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1839,66 +1839,6 @@ class RunQueueExecute:
can_start = active < self.number_tasks
return can_start
-class RunQueueExecuteDummy(RunQueueExecute):
- def __init__(self, rq):
- self.rq = rq
- self.stats = RunQueueStats(0)
-
- def finish(self):
- self.rq.state = runQueueComplete
- return
-
-class RunQueueExecuteTasks(RunQueueExecute):
- def __init__(self, rq):
- RunQueueExecute.__init__(self, rq)
-
- self.stampcache = {}
-
- # Mark initial buildable tasks
- for tid in self.rqdata.runtaskentries:
- if len(self.rqdata.runtaskentries[tid].depends) == 0:
- self.runq_buildable.add(tid)
- if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
- self.rq.scenequeue_covered.add(tid)
-
- found = True
- while found:
- found = False
- for tid in self.rqdata.runtaskentries:
- if tid in self.rq.scenequeue_covered:
- continue
- logger.debug(1, 'Considering %s: %s' % (tid, str(self.rqdata.runtaskentries[tid].revdeps)))
-
- if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
- if tid in self.rq.scenequeue_notcovered:
- continue
- found = True
- self.rq.scenequeue_covered.add(tid)
-
- logger.debug(1, 'Skip list %s', sorted(self.rq.scenequeue_covered))
-
- for task in self.rq.scenequeue_notcovered:
- logger.debug(1, 'Not skipping task %s', task)
-
- for mc in self.rqdata.dataCaches:
- target_pairs = []
- for tid in self.rqdata.target_tids:
- (tidmc, fn, taskname, _) = split_tid_mcfn(tid)
- if tidmc == mc:
- target_pairs.append((fn, taskname))
-
- event.fire(bb.event.StampUpdate(target_pairs, self.rqdata.dataCaches[mc].stamp), self.cfgData)
-
- schedulers = self.get_schedulers()
- for scheduler in schedulers:
- if self.scheduler == scheduler.name:
- self.sched = scheduler(self, self.rqdata)
- logger.debug(1, "Using runqueue scheduler '%s'", scheduler.name)
- break
- else:
- bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" %
- (self.scheduler, ", ".join(obj.name for obj in schedulers)))
-
def get_schedulers(self):
schedulers = set(obj for obj in globals().values()
if type(obj) is type and
@@ -2100,122 +2040,320 @@ class RunQueueExecuteTasks(RunQueueExecute):
#bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n"))
return taskdepdata
-class SQData(object):
- def __init__(self):
- self.sq_harddeps = {}
- self.stamps = {}
- self.unskippable = []
+ def scenequeue_updatecounters(self, task, fail = False):
+ for dep in self.sqdata.sq_deps[task]:
+ if fail and task in self.sqdata.sq_harddeps and dep in self.sqdata.sq_harddeps[task]:
+ logger.debug(2, "%s was unavailable and is a hard dependency of %s so skipping" % (task, dep))
+ self.scenequeue_updatecounters(dep, fail)
+ continue
+ if task not in self.sqdata.sq_revdeps2[dep]:
+ # May already have been removed by the fail case above
+ continue
+ self.sqdata.sq_revdeps2[dep].remove(task)
+ if len(self.sqdata.sq_revdeps2[dep]) == 0:
+ self.sq_buildable.add(dep)
-def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
+ def sq_task_completeoutright(self, task):
+ """
+ Mark a task as completed
+ Look at the reverse dependencies and mark any task with
+ completed dependencies as buildable
+ """
- sq_revdeps = {}
- sq_revdeps_new = {}
- sq_revdeps_squash = {}
+ logger.debug(1, 'Found task %s which could be accelerated', task)
+ self.scenequeue_covered.add(task)
+ self.scenequeue_updatecounters(task)
- # We need to construct a dependency graph for the setscene functions. Intermediate
- # dependencies between the setscene tasks only complicate the code. This code
- # therefore aims to collapse the huge runqueue dependency tree into a smaller one
- # only containing the setscene functions.
+ def sq_check_taskfail(self, task):
+ if self.rqdata.setscenewhitelist is not None:
+ realtask = task.split('_setscene')[0]
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(realtask)
+ pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
+ if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
+ logger.error('Task %s.%s failed' % (pn, taskname + "_setscene"))
+ self.rq.state = runQueueCleanUp
- rqdata.init_progress_reporter.next_stage()
+ def sq_task_complete(self, task):
+ self.sq_stats.taskCompleted()
+ bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats, self.rq), self.cfgData)
+ self.sq_task_completeoutright(task)
- # First process the chains up to the first setscene task.
- endpoints = {}
- for tid in rqdata.runtaskentries:
- sq_revdeps[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps)
- sq_revdeps_new[tid] = set()
- if (len(sq_revdeps[tid]) == 0) and tid not in rqdata.runq_setscene_tids:
- #bb.warn("Added endpoint %s" % (tid))
- endpoints[tid] = set()
+ def sq_task_fail(self, task, result):
+ self.sq_stats.taskFailed()
+ bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats, result, self), self.cfgData)
+ self.scenequeue_notcovered.add(task)
+ self.scenequeue_updatecounters(task, True)
+ self.sq_check_taskfail(task)
- rqdata.init_progress_reporter.next_stage()
+ def sq_task_failoutright(self, task):
+ self.sq_running.add(task)
+ self.sq_buildable.add(task)
+ self.sq_stats.taskSkipped()
+ self.sq_stats.taskCompleted()
+ self.scenequeue_notcovered.add(task)
+ self.scenequeue_updatecounters(task, True)
- # Secondly process the chains between setscene tasks.
- for tid in rqdata.runq_setscene_tids:
- #bb.warn("Added endpoint 2 %s" % (tid))
- for dep in rqdata.runtaskentries[tid].depends:
- if tid in sq_revdeps[dep]:
- sq_revdeps[dep].remove(tid)
- if dep not in endpoints:
- endpoints[dep] = set()
- #bb.warn(" Added endpoint 3 %s" % (dep))
- endpoints[dep].add(tid)
+ def sq_task_skip(self, task):
+ self.sq_running.add(task)
+ self.sq_buildable.add(task)
+ self.sq_task_completeoutright(task)
+ self.sq_stats.taskSkipped()
+ self.sq_stats.taskCompleted()
- rqdata.init_progress_reporter.next_stage()
+ def sq_execute(self):
+ """
+ Run the tasks in a queue prepared by prepare_runqueue
+ """
- def process_endpoints(endpoints):
- newendpoints = {}
- for point, task in endpoints.items():
- tasks = set()
- if task:
- tasks |= task
- if sq_revdeps_new[point]:
- tasks |= sq_revdeps_new[point]
- sq_revdeps_new[point] = set()
- if point in rqdata.runq_setscene_tids:
- sq_revdeps_new[point] = tasks
- tasks = set()
- continue
- for dep in rqdata.runtaskentries[point].depends:
- if point in sq_revdeps[dep]:
- sq_revdeps[dep].remove(point)
- if tasks:
- sq_revdeps_new[dep] |= tasks
- if len(sq_revdeps[dep]) == 0 and dep not in rqdata.runq_setscene_tids:
- newendpoints[dep] = task
- if len(newendpoints) != 0:
- process_endpoints(newendpoints)
+ self.rq.read_workers()
- process_endpoints(endpoints)
+ task = None
+ if self.can_start_task():
+ # Find the next setscene to run
+ for nexttask in self.rqdata.runq_setscene_tids:
+ if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
+ if nexttask in self.sqdata.unskippable:
+ logger.debug(2, "Setscene task %s is unskippable" % nexttask)
+ if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]):
+ fn = fn_from_tid(nexttask)
+ foundtarget = False
- rqdata.init_progress_reporter.next_stage()
+ if nexttask in self.rqdata.target_tids:
+ foundtarget = True
+ if not foundtarget:
+ logger.debug(2, "Skipping setscene for task %s" % nexttask)
+ self.sq_task_skip(nexttask)
+ self.scenequeue_notneeded.add(nexttask)
+ return True
+ if nexttask in self.sqdata.outrightfail:
+ self.sq_task_failoutright(nexttask)
+ return True
+ task = nexttask
+ break
+ if task is not None:
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
+ taskname = taskname + "_setscene"
+ if self.rq.check_stamp_task(task, taskname_from_tid(task), recurse = True, cache=self.stampcache):
+ logger.debug(2, 'Stamp for underlying task %s is current, so skipping setscene variant', task)
+ self.sq_task_failoutright(task)
+ return True
- # Build a list of setscene tasks which are "unskippable"
- # These are direct endpoints referenced by the build
- endpoints2 = {}
- sq_revdeps2 = {}
- sq_revdeps_new2 = {}
- def process_endpoints2(endpoints):
- newendpoints = {}
- for point, task in endpoints.items():
- tasks = set([point])
- if task:
- tasks |= task
- if sq_revdeps_new2[point]:
- tasks |= sq_revdeps_new2[point]
- sq_revdeps_new2[point] = set()
- if point in rqdata.runq_setscene_tids:
- sq_revdeps_new2[point] = tasks
- for dep in rqdata.runtaskentries[point].depends:
- if point in sq_revdeps2[dep]:
- sq_revdeps2[dep].remove(point)
- if tasks:
- sq_revdeps_new2[dep] |= tasks
- if (len(sq_revdeps2[dep]) == 0 or len(sq_revdeps_new2[dep]) != 0) and dep not in rqdata.runq_setscene_tids:
- newendpoints[dep] = tasks
- if len(newendpoints) != 0:
- process_endpoints2(newendpoints)
- for tid in rqdata.runtaskentries:
- sq_revdeps2[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps)
- sq_revdeps_new2[tid] = set()
- if (len(sq_revdeps2[tid]) == 0) and tid not in rqdata.runq_setscene_tids:
- endpoints2[tid] = set()
- process_endpoints2(endpoints2)
- for tid in rqdata.runq_setscene_tids:
- if sq_revdeps_new2[tid]:
- sqdata.unskippable.append(tid)
+ if self.cooker.configuration.force:
+ if task in self.rqdata.target_tids:
+ self.sq_task_failoutright(task)
+ return True
- rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries))
+ if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
+ logger.debug(2, 'Setscene stamp current task %s, so skip it and its dependencies', task)
+ self.sq_task_skip(task)
+ return True
- for taskcounter, tid in enumerate(rqdata.runtaskentries):
- if tid in rqdata.runq_setscene_tids:
- deps = set()
- for dep in sq_revdeps_new[tid]:
- deps.add(dep)
- sq_revdeps_squash[tid] = deps
- elif len(sq_revdeps_new[tid]) != 0:
- bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
- rqdata.init_progress_reporter.update(taskcounter)
+ if self.cooker.configuration.skipsetscene:
+ logger.debug(2, 'No setscene tasks should be executed. Skipping %s', task)
+ self.sq_task_failoutright(task)
+ return True
+
+ startevent = sceneQueueTaskStarted(task, self.sq_stats, self.rq)
+ bb.event.fire(startevent, self.cfgData)
+
+ taskdepdata = self.sq_build_taskdepdata(task)
+
+ taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
+ taskhash = self.rqdata.get_task_hash(task)
+ unihash = self.rqdata.get_task_unihash(task)
+ if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
+ if not mc in self.rq.fakeworker:
+ self.rq.start_fakeworker(self, mc)
+ self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
+ self.rq.fakeworker[mc].process.stdin.flush()
+ else:
+ self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
+ self.rq.worker[mc].process.stdin.flush()
+
+ self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True)
+ self.build_stamps2.append(self.build_stamps[task])
+ self.sq_running.add(task)
+ self.sq_live.add(task)
+ self.sq_stats.taskActive()
+ if self.can_start_task():
+ return True
+
+ if self.sq_stats.active > 0:
+ self.rq.read_workers()
+ return self.rq.active_fds()
+
+ #for tid in self.sqdata.sq_revdeps:
+ # if tid not in self.sq_running:
+ # buildable = tid in self.sq_buildable
+ # revdeps = self.sqdata.sq_revdeps[tid]
+ # bb.warn("Found we didn't run %s %s %s" % (tid, buildable, str(revdeps)))
+
+ self.rq.scenequeue_covered = self.scenequeue_covered
+ self.rq.scenequeue_notcovered = self.scenequeue_notcovered
+
+ logger.debug(1, 'We can skip tasks %s', "\n".join(sorted(self.rq.scenequeue_covered)))
+
+ self.rq.state = runQueueRunInit
+
+ completeevent = sceneQueueComplete(self.sq_stats, self.rq)
+ bb.event.fire(completeevent, self.cfgData)
+
+ return True
+
+ def sq_build_taskdepdata(self, task):
+ def getsetscenedeps(tid):
+ deps = set()
+ (mc, fn, taskname, _) = split_tid_mcfn(tid)
+ realtid = tid + "_setscene"
+ idepends = self.rqdata.taskData[mc].taskentries[realtid].idepends
+ for (depname, idependtask) in idepends:
+ if depname not in self.rqdata.taskData[mc].build_targets:
+ continue
+
+ depfn = self.rqdata.taskData[mc].build_targets[depname][0]
+ if depfn is None:
+ continue
+ deptid = depfn + ":" + idependtask.replace("_setscene", "")
+ deps.add(deptid)
+ return deps
+
+ taskdepdata = {}
+ next = getsetscenedeps(task)
+ next.add(task)
+ while next:
+ additional = []
+ for revdep in next:
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(revdep)
+ pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
+ deps = getsetscenedeps(revdep)
+ provides = self.rqdata.dataCaches[mc].fn_provides[taskfn]
+ taskhash = self.rqdata.runtaskentries[revdep].hash
+ unihash = self.rqdata.runtaskentries[revdep].unihash
+ taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, unihash]
+ for revdep2 in deps:
+ if revdep2 not in taskdepdata:
+ additional.append(revdep2)
+ next = additional
+
+ #bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n"))
+ return taskdepdata
+
+class SQData(object):
+ def __init__(self):
+ self.sq_harddeps = {}
+ self.stamps = {}
+ self.unskippable = []
+
+def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
+
+ sq_revdeps = {}
+ sq_revdeps_new = {}
+ sq_revdeps_squash = {}
+
+ # We need to construct a dependency graph for the setscene functions. Intermediate
+ # dependencies between the setscene tasks only complicate the code. This code
+ # therefore aims to collapse the huge runqueue dependency tree into a smaller one
+ # only containing the setscene functions.
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # First process the chains up to the first setscene task.
+ endpoints = {}
+ for tid in rqdata.runtaskentries:
+ sq_revdeps[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps)
+ sq_revdeps_new[tid] = set()
+ if (len(sq_revdeps[tid]) == 0) and tid not in rqdata.runq_setscene_tids:
+ #bb.warn("Added endpoint %s" % (tid))
+ endpoints[tid] = set()
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # Secondly process the chains between setscene tasks.
+ for tid in rqdata.runq_setscene_tids:
+ #bb.warn("Added endpoint 2 %s" % (tid))
+ for dep in rqdata.runtaskentries[tid].depends:
+ if tid in sq_revdeps[dep]:
+ sq_revdeps[dep].remove(tid)
+ if dep not in endpoints:
+ endpoints[dep] = set()
+ #bb.warn(" Added endpoint 3 %s" % (dep))
+ endpoints[dep].add(tid)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ def process_endpoints(endpoints):
+ newendpoints = {}
+ for point, task in endpoints.items():
+ tasks = set()
+ if task:
+ tasks |= task
+ if sq_revdeps_new[point]:
+ tasks |= sq_revdeps_new[point]
+ sq_revdeps_new[point] = set()
+ if point in rqdata.runq_setscene_tids:
+ sq_revdeps_new[point] = tasks
+ tasks = set()
+ continue
+ for dep in rqdata.runtaskentries[point].depends:
+ if point in sq_revdeps[dep]:
+ sq_revdeps[dep].remove(point)
+ if tasks:
+ sq_revdeps_new[dep] |= tasks
+ if len(sq_revdeps[dep]) == 0 and dep not in rqdata.runq_setscene_tids:
+ newendpoints[dep] = task
+ if len(newendpoints) != 0:
+ process_endpoints(newendpoints)
+
+ process_endpoints(endpoints)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # Build a list of setscene tasks which are "unskippable"
+ # These are direct endpoints referenced by the build
+ endpoints2 = {}
+ sq_revdeps2 = {}
+ sq_revdeps_new2 = {}
+ def process_endpoints2(endpoints):
+ newendpoints = {}
+ for point, task in endpoints.items():
+ tasks = set([point])
+ if task:
+ tasks |= task
+ if sq_revdeps_new2[point]:
+ tasks |= sq_revdeps_new2[point]
+ sq_revdeps_new2[point] = set()
+ if point in rqdata.runq_setscene_tids:
+ sq_revdeps_new2[point] = tasks
+ for dep in rqdata.runtaskentries[point].depends:
+ if point in sq_revdeps2[dep]:
+ sq_revdeps2[dep].remove(point)
+ if tasks:
+ sq_revdeps_new2[dep] |= tasks
+ if (len(sq_revdeps2[dep]) == 0 or len(sq_revdeps_new2[dep]) != 0) and dep not in rqdata.runq_setscene_tids:
+ newendpoints[dep] = tasks
+ if len(newendpoints) != 0:
+ process_endpoints2(newendpoints)
+ for tid in rqdata.runtaskentries:
+ sq_revdeps2[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps)
+ sq_revdeps_new2[tid] = set()
+ if (len(sq_revdeps2[tid]) == 0) and tid not in rqdata.runq_setscene_tids:
+ endpoints2[tid] = set()
+ process_endpoints2(endpoints2)
+ for tid in rqdata.runq_setscene_tids:
+ if sq_revdeps_new2[tid]:
+ sqdata.unskippable.append(tid)
+
+ rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries))
+
+ for taskcounter, tid in enumerate(rqdata.runtaskentries):
+ if tid in rqdata.runq_setscene_tids:
+ deps = set()
+ for dep in sq_revdeps_new[tid]:
+ deps.add(dep)
+ sq_revdeps_squash[tid] = deps
+ elif len(sq_revdeps_new[tid]) != 0:
+ bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
+ rqdata.init_progress_reporter.update(taskcounter)
rqdata.init_progress_reporter.next_stage()
@@ -2335,6 +2473,66 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
sqdata.outrightfail.append(tid)
+class RunQueueExecuteDummy(RunQueueExecute):
+ def __init__(self, rq):
+ self.rq = rq
+ self.stats = RunQueueStats(0)
+
+ def finish(self):
+ self.rq.state = runQueueComplete
+ return
+
+class RunQueueExecuteTasks(RunQueueExecute):
+ def __init__(self, rq):
+ RunQueueExecute.__init__(self, rq)
+
+ self.stampcache = {}
+
+ # Mark initial buildable tasks
+ for tid in self.rqdata.runtaskentries:
+ if len(self.rqdata.runtaskentries[tid].depends) == 0:
+ self.runq_buildable.add(tid)
+ if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
+ self.rq.scenequeue_covered.add(tid)
+
+ found = True
+ while found:
+ found = False
+ for tid in self.rqdata.runtaskentries:
+ if tid in self.rq.scenequeue_covered:
+ continue
+ logger.debug(1, 'Considering %s: %s' % (tid, str(self.rqdata.runtaskentries[tid].revdeps)))
+
+ if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
+ if tid in self.rq.scenequeue_notcovered:
+ continue
+ found = True
+ self.rq.scenequeue_covered.add(tid)
+
+ logger.debug(1, 'Skip list %s', sorted(self.rq.scenequeue_covered))
+
+ for task in self.rq.scenequeue_notcovered:
+ logger.debug(1, 'Not skipping task %s', task)
+
+ for mc in self.rqdata.dataCaches:
+ target_pairs = []
+ for tid in self.rqdata.target_tids:
+ (tidmc, fn, taskname, _) = split_tid_mcfn(tid)
+ if tidmc == mc:
+ target_pairs.append((fn, taskname))
+
+ event.fire(bb.event.StampUpdate(target_pairs, self.rqdata.dataCaches[mc].stamp), self.cfgData)
+
+ schedulers = self.get_schedulers()
+ for scheduler in schedulers:
+ if self.scheduler == scheduler.name:
+ self.sched = scheduler(self, self.rqdata)
+ logger.debug(1, "Using runqueue scheduler '%s'", scheduler.name)
+ break
+ else:
+ bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" %
+ (self.scheduler, ", ".join(obj.name for obj in schedulers)))
+
class RunQueueExecuteScenequeue(RunQueueExecute):
def __init__(self, rq):
RunQueueExecute.__init__(self, rq)
@@ -2357,204 +2555,6 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.rq.state = runQueueSceneRun
- def scenequeue_updatecounters(self, task, fail = False):
- for dep in self.sqdata.sq_deps[task]:
- if fail and task in self.sqdata.sq_harddeps and dep in self.sqdata.sq_harddeps[task]:
- logger.debug(2, "%s was unavailable and is a hard dependency of %s so skipping" % (task, dep))
- self.scenequeue_updatecounters(dep, fail)
- continue
- if task not in self.sqdata.sq_revdeps2[dep]:
- # May already have been removed by the fail case above
- continue
- self.sqdata.sq_revdeps2[dep].remove(task)
- if len(self.sqdata.sq_revdeps2[dep]) == 0:
- self.sq_buildable.add(dep)
-
- def sq_task_completeoutright(self, task):
- """
- Mark a task as completed
- Look at the reverse dependencies and mark any task with
- completed dependencies as buildable
- """
-
- logger.debug(1, 'Found task %s which could be accelerated', task)
- self.scenequeue_covered.add(task)
- self.scenequeue_updatecounters(task)
-
- def sq_check_taskfail(self, task):
- if self.rqdata.setscenewhitelist is not None:
- realtask = task.split('_setscene')[0]
- (mc, fn, taskname, taskfn) = split_tid_mcfn(realtask)
- pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
- if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
- logger.error('Task %s.%s failed' % (pn, taskname + "_setscene"))
- self.rq.state = runQueueCleanUp
-
- def sq_task_complete(self, task):
- self.sq_stats.taskCompleted()
- bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats, self.rq), self.cfgData)
- self.sq_task_completeoutright(task)
-
- def sq_task_fail(self, task, result):
- self.sq_stats.taskFailed()
- bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats, result, self), self.cfgData)
- self.scenequeue_notcovered.add(task)
- self.scenequeue_updatecounters(task, True)
- self.sq_check_taskfail(task)
-
- def sq_task_failoutright(self, task):
- self.sq_running.add(task)
- self.sq_buildable.add(task)
- self.sq_stats.taskSkipped()
- self.sq_stats.taskCompleted()
- self.scenequeue_notcovered.add(task)
- self.scenequeue_updatecounters(task, True)
-
- def sq_task_skip(self, task):
- self.sq_running.add(task)
- self.sq_buildable.add(task)
- self.sq_task_completeoutright(task)
- self.sq_stats.taskSkipped()
- self.sq_stats.taskCompleted()
-
- def sq_execute(self):
- """
- Run the tasks in a queue prepared by prepare_runqueue
- """
-
- self.rq.read_workers()
-
- task = None
- if self.can_start_task():
- # Find the next setscene to run
- for nexttask in self.rqdata.runq_setscene_tids:
- if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
- if nexttask in self.sqdata.unskippable:
- logger.debug(2, "Setscene task %s is unskippable" % nexttask)
- if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]):
- fn = fn_from_tid(nexttask)
- foundtarget = False
-
- if nexttask in self.rqdata.target_tids:
- foundtarget = True
- if not foundtarget:
- logger.debug(2, "Skipping setscene for task %s" % nexttask)
- self.sq_task_skip(nexttask)
- self.scenequeue_notneeded.add(nexttask)
- return True
- if nexttask in self.sqdata.outrightfail:
- self.sq_task_failoutright(nexttask)
- return True
- task = nexttask
- break
- if task is not None:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
- taskname = taskname + "_setscene"
- if self.rq.check_stamp_task(task, taskname_from_tid(task), recurse = True, cache=self.stampcache):
- logger.debug(2, 'Stamp for underlying task %s is current, so skipping setscene variant', task)
- self.sq_task_failoutright(task)
- return True
-
- if self.cooker.configuration.force:
- if task in self.rqdata.target_tids:
- self.sq_task_failoutright(task)
- return True
-
- if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
- logger.debug(2, 'Setscene stamp current task %s, so skip it and its dependencies', task)
- self.sq_task_skip(task)
- return True
-
- if self.cooker.configuration.skipsetscene:
- logger.debug(2, 'No setscene tasks should be executed. Skipping %s', task)
- self.sq_task_failoutright(task)
- return True
-
- startevent = sceneQueueTaskStarted(task, self.sq_stats, self.rq)
- bb.event.fire(startevent, self.cfgData)
-
- taskdepdata = self.sq_build_taskdepdata(task)
-
- taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
- taskhash = self.rqdata.get_task_hash(task)
- unihash = self.rqdata.get_task_unihash(task)
- if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
- if not mc in self.rq.fakeworker:
- self.rq.start_fakeworker(self, mc)
- self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
- self.rq.fakeworker[mc].process.stdin.flush()
- else:
- self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
- self.rq.worker[mc].process.stdin.flush()
-
- self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True)
- self.build_stamps2.append(self.build_stamps[task])
- self.sq_running.add(task)
- self.sq_live.add(task)
- self.sq_stats.taskActive()
- if self.can_start_task():
- return True
-
- if self.sq_stats.active > 0:
- self.rq.read_workers()
- return self.rq.active_fds()
-
- #for tid in self.sqdata.sq_revdeps:
- # if tid not in self.sq_running:
- # buildable = tid in self.sq_buildable
- # revdeps = self.sqdata.sq_revdeps[tid]
- # bb.warn("Found we didn't run %s %s %s" % (tid, buildable, str(revdeps)))
-
- self.rq.scenequeue_covered = self.scenequeue_covered
- self.rq.scenequeue_notcovered = self.scenequeue_notcovered
-
- logger.debug(1, 'We can skip tasks %s', "\n".join(sorted(self.rq.scenequeue_covered)))
-
- self.rq.state = runQueueRunInit
-
- completeevent = sceneQueueComplete(self.sq_stats, self.rq)
- bb.event.fire(completeevent, self.cfgData)
-
- return True
-
- def sq_build_taskdepdata(self, task):
- def getsetscenedeps(tid):
- deps = set()
- (mc, fn, taskname, _) = split_tid_mcfn(tid)
- realtid = tid + "_setscene"
- idepends = self.rqdata.taskData[mc].taskentries[realtid].idepends
- for (depname, idependtask) in idepends:
- if depname not in self.rqdata.taskData[mc].build_targets:
- continue
-
- depfn = self.rqdata.taskData[mc].build_targets[depname][0]
- if depfn is None:
- continue
- deptid = depfn + ":" + idependtask.replace("_setscene", "")
- deps.add(deptid)
- return deps
-
- taskdepdata = {}
- next = getsetscenedeps(task)
- next.add(task)
- while next:
- additional = []
- for revdep in next:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(revdep)
- pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
- deps = getsetscenedeps(revdep)
- provides = self.rqdata.dataCaches[mc].fn_provides[taskfn]
- taskhash = self.rqdata.runtaskentries[revdep].hash
- unihash = self.rqdata.runtaskentries[revdep].unihash
- taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, unihash]
- for revdep2 in deps:
- if revdep2 not in taskdepdata:
- additional.append(revdep2)
- next = additional
-
- #bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n"))
- return taskdepdata
-
class TaskFailure(Exception):
"""
Exception raised when a task in a runqueue fails
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 10/26] runqueue: Fix counter/task updating glitch
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (7 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 09/26] runqueue: Merge scenequeue and real task queue code together Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 11/26] runqueue: Remove RunQueueExecuteScenequeue and RunQueueExecuteTasks Richard Purdie
` (16 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Some tasks were not being marked as covered/notcovered since internal
calls were being made without using the external call points.
Fix the accounting issues by using the correct external call points.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 9ec2f7d2d4..54dce83737 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2044,7 +2044,7 @@ class RunQueueExecute:
for dep in self.sqdata.sq_deps[task]:
if fail and task in self.sqdata.sq_harddeps and dep in self.sqdata.sq_harddeps[task]:
logger.debug(2, "%s was unavailable and is a hard dependency of %s so skipping" % (task, dep))
- self.scenequeue_updatecounters(dep, fail)
+ self.sq_task_failoutright(dep)
continue
if task not in self.sqdata.sq_revdeps2[dep]:
# May already have been removed by the fail case above
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 11/26] runqueue: Remove RunQueueExecuteScenequeue and RunQueueExecuteTasks
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (8 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 10/26] runqueue: Fix counter/task updating glitch Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 12/26] runqueue: Simplify _execute_runqueue logic Richard Purdie
` (15 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Replace the remains of the Tasks and Scenequeue Tasks classes with simple
function calls. Also drop the dummy version of the execution class to
simplify further changes as its not needed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 99 ++++++++++++++++++++--------------------------
1 file changed, 43 insertions(+), 56 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 54dce83737..3456fd976b 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1407,7 +1407,6 @@ class RunQueue:
retval = True
if self.state is runQueuePrepare:
- self.rqexe = RunQueueExecuteDummy(self)
# NOTE: if you add, remove or significantly refactor the stages of this
# process then you should recalculate the weightings here. This is quite
# easy to do - just change the next line temporarily to pass debug=True as
@@ -1449,7 +1448,9 @@ class RunQueue:
self.rqdata.init_progress_reporter.next_stage()
self.start_worker()
self.rqdata.init_progress_reporter.next_stage()
- self.rqexe = RunQueueExecuteScenequeue(self)
+ if not self.rqexe:
+ self.rqexe = RunQueueExecute(self)
+ start_scenequeue_tasks(self.rqexe)
if self.state is runQueueSceneRun:
retval = self.rqexe.sq_execute()
@@ -1461,7 +1462,9 @@ class RunQueue:
# Just in case we didn't setscene
self.rqdata.init_progress_reporter.finish()
logger.info("Executing RunQueue Tasks")
- self.rqexe = RunQueueExecuteTasks(self)
+ if not self.rqexe:
+ self.rqexe = RunQueueExecute(self)
+ start_runqueue_tasks(self.rqexe)
self.state = runQueueRunning
if self.state is runQueueRunning:
@@ -1478,11 +1481,12 @@ class RunQueue:
if build_done and self.rqexe:
self.teardown_workers()
- if self.rqexe.stats.failed:
- logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed)
- else:
- # Let's avoid the word "failed" if nothing actually did
- logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and all succeeded.", self.rqexe.stats.completed, self.rqexe.stats.skipped)
+ if self.rqexe:
+ if self.rqexe.stats.failed:
+ logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed)
+ else:
+ # Let's avoid the word "failed" if nothing actually did
+ logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and all succeeded.", self.rqexe.stats.completed, self.rqexe.stats.skipped)
if self.state is runQueueFailed:
raise bb.runqueue.TaskFailure(self.rqexe.failed_tids)
@@ -2472,88 +2476,71 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
logger.debug(2, 'No package found, so skipping setscene task %s', tid)
sqdata.outrightfail.append(tid)
-
-class RunQueueExecuteDummy(RunQueueExecute):
- def __init__(self, rq):
- self.rq = rq
- self.stats = RunQueueStats(0)
-
- def finish(self):
- self.rq.state = runQueueComplete
- return
-
-class RunQueueExecuteTasks(RunQueueExecute):
- def __init__(self, rq):
- RunQueueExecute.__init__(self, rq)
-
- self.stampcache = {}
+def start_runqueue_tasks(rqexec):
# Mark initial buildable tasks
- for tid in self.rqdata.runtaskentries:
- if len(self.rqdata.runtaskentries[tid].depends) == 0:
- self.runq_buildable.add(tid)
- if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
- self.rq.scenequeue_covered.add(tid)
+ for tid in rqexec.rqdata.runtaskentries:
+ if len(rqexec.rqdata.runtaskentries[tid].depends) == 0:
+ rqexec.runq_buildable.add(tid)
+ if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.rq.scenequeue_covered):
+ rqexec.rq.scenequeue_covered.add(tid)
found = True
while found:
found = False
- for tid in self.rqdata.runtaskentries:
- if tid in self.rq.scenequeue_covered:
+ for tid in rqexec.rqdata.runtaskentries:
+ if tid in rqexec.rq.scenequeue_covered:
continue
- logger.debug(1, 'Considering %s: %s' % (tid, str(self.rqdata.runtaskentries[tid].revdeps)))
+ logger.debug(1, 'Considering %s: %s' % (tid, str(rqexec.rqdata.runtaskentries[tid].revdeps)))
- if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
- if tid in self.rq.scenequeue_notcovered:
+ if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.rq.scenequeue_covered):
+ if tid in rqexec.rq.scenequeue_notcovered:
continue
found = True
- self.rq.scenequeue_covered.add(tid)
+ rqexec.rq.scenequeue_covered.add(tid)
- logger.debug(1, 'Skip list %s', sorted(self.rq.scenequeue_covered))
+ logger.debug(1, 'Skip list %s', sorted(rqexec.rq.scenequeue_covered))
for task in self.rq.scenequeue_notcovered:
logger.debug(1, 'Not skipping task %s', task)
- for mc in self.rqdata.dataCaches:
+ for mc in rqexec.rqdata.dataCaches:
target_pairs = []
- for tid in self.rqdata.target_tids:
+ for tid in rqexec.rqdata.target_tids:
(tidmc, fn, taskname, _) = split_tid_mcfn(tid)
if tidmc == mc:
target_pairs.append((fn, taskname))
- event.fire(bb.event.StampUpdate(target_pairs, self.rqdata.dataCaches[mc].stamp), self.cfgData)
+ event.fire(bb.event.StampUpdate(target_pairs, rqexec.rqdata.dataCaches[mc].stamp), rqexec.cfgData)
- schedulers = self.get_schedulers()
+ schedulers = rqexec.get_schedulers()
for scheduler in schedulers:
- if self.scheduler == scheduler.name:
- self.sched = scheduler(self, self.rqdata)
+ if rqexec.scheduler == scheduler.name:
+ rqexec.sched = scheduler(rqexec, rqexec.rqdata)
logger.debug(1, "Using runqueue scheduler '%s'", scheduler.name)
break
else:
bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" %
- (self.scheduler, ", ".join(obj.name for obj in schedulers)))
-
-class RunQueueExecuteScenequeue(RunQueueExecute):
- def __init__(self, rq):
- RunQueueExecute.__init__(self, rq)
+ (rqexec.scheduler, ", ".join(obj.name for obj in schedulers)))
- self.scenequeue_covered = set()
- self.scenequeue_notcovered = set()
- self.scenequeue_notneeded = set()
+def start_scenequeue_tasks(rqexec):
+ rqexec.scenequeue_covered = set()
+ rqexec.scenequeue_notcovered = set()
+ rqexec.scenequeue_notneeded = set()
# If we don't have any setscene functions, skip this step
- if len(self.rqdata.runq_setscene_tids) == 0:
- rq.scenequeue_covered = set()
- rq.scenequeue_notcovered = set()
- rq.state = runQueueRunInit
+ if len(rqexec.rqdata.runq_setscene_tids) == 0:
+ rqexec.rq.scenequeue_covered = set()
+ rqexec.rq.scenequeue_notcovered = set()
+ rqexec.rq.state = runQueueRunInit
return
- self.sqdata = SQData()
- build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self)
+ rqexec.sqdata = SQData()
+ build_scenequeue_data(rqexec.sqdata, rqexec.rqdata, rqexec.rq, rqexec.cooker, rqexec.stampcache, rqexec)
logger.info('Executing SetScene Tasks')
- self.rq.state = runQueueSceneRun
+ rqexec.rq.state = runQueueSceneRun
class TaskFailure(Exception):
"""
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 12/26] runqueue: Simplify _execute_runqueue logic
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (9 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 11/26] runqueue: Remove RunQueueExecuteScenequeue and RunQueueExecuteTasks Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 13/26] runqueue: Fold remains of the scenequeue setup into RunQueueExecute Richard Purdie
` (14 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Cleanup to the _execute_runqueue logic to reduce indentation, drop the
dummy executor class concept and prepare for further changes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 46 ++++++++++++++++++++++++----------------------
1 file changed, 24 insertions(+), 22 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 3456fd976b..a19fb61fe7 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1420,15 +1420,16 @@ class RunQueue:
self.state = runQueueComplete
else:
self.state = runQueueSceneInit
- self.rqdata.init_progress_reporter.next_stage()
-
- # we are ready to run, emit dependency info to any UI or class which
- # needs it
- depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
- self.rqdata.init_progress_reporter.next_stage()
- bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
if self.state is runQueueSceneInit:
+ self.rqdata.init_progress_reporter.next_stage()
+
+ # we are ready to run, emit dependency info to any UI or class which
+ # needs it
+ depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
+ self.rqdata.init_progress_reporter.next_stage()
+ bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
+
if not self.dm_event_handler_registered:
res = bb.event.register(self.dm_event_handler_name,
lambda x: self.dm.check(self) if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp] else False,
@@ -1444,13 +1445,13 @@ class RunQueue:
if 'printdiff' in dump:
self.write_diffscenetasks(invalidtasks)
self.state = runQueueComplete
- else:
- self.rqdata.init_progress_reporter.next_stage()
- self.start_worker()
- self.rqdata.init_progress_reporter.next_stage()
- if not self.rqexe:
- self.rqexe = RunQueueExecute(self)
- start_scenequeue_tasks(self.rqexe)
+
+ if self.state is runQueueSceneInit:
+ self.rqdata.init_progress_reporter.next_stage()
+ self.start_worker()
+ self.rqdata.init_progress_reporter.next_stage()
+ self.rqexe = RunQueueExecute(self)
+ start_scenequeue_tasks(self.rqexe)
if self.state is runQueueSceneRun:
retval = self.rqexe.sq_execute()
@@ -1458,14 +1459,15 @@ class RunQueue:
if self.state is runQueueRunInit:
if self.cooker.configuration.setsceneonly:
self.state = runQueueComplete
- else:
- # Just in case we didn't setscene
- self.rqdata.init_progress_reporter.finish()
- logger.info("Executing RunQueue Tasks")
- if not self.rqexe:
- self.rqexe = RunQueueExecute(self)
- start_runqueue_tasks(self.rqexe)
- self.state = runQueueRunning
+
+ if self.state is runQueueRunInit:
+ # Just in case we didn't setscene
+ self.rqdata.init_progress_reporter.finish()
+ logger.info("Executing RunQueue Tasks")
+ if not self.rqexe:
+ self.rqexe = RunQueueExecute(self)
+ start_runqueue_tasks(self.rqexe)
+ self.state = runQueueRunning
if self.state is runQueueRunning:
retval = self.rqexe.execute()
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 13/26] runqueue: Fold remains of the scenequeue setup into RunQueueExecute
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (10 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 12/26] runqueue: Simplify _execute_runqueue logic Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 14/26] event/runqueue: Drop StampUpdate event, its pointless/unused Richard Purdie
` (13 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Also move the scheduler init over, apart for the builtable tasks part which need
to remain called later.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 89 ++++++++++++++++++++++------------------------
1 file changed, 42 insertions(+), 47 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index a19fb61fe7..46146466fc 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -196,6 +196,11 @@ class RunQueueScheduler(object):
if self.rq.can_start_task():
return self.next_buildable_task()
+ def initbuildable(self):
+ for tid in self.rqdata.runtaskentries:
+ if tid in self.rq.runq_buildable:
+ self.buildable.append(tid)
+
def newbuildable(self, task):
self.buildable.append(task)
@@ -1451,7 +1456,14 @@ class RunQueue:
self.start_worker()
self.rqdata.init_progress_reporter.next_stage()
self.rqexe = RunQueueExecute(self)
- start_scenequeue_tasks(self.rqexe)
+
+ # If we don't have any setscene functions, skip execution
+ if len(self.rqdata.runq_setscene_tids) == 0:
+ self.rqdata.init_progress_reporter.finish()
+ self.state = runQueueRunInit
+ else:
+ logger.info('Executing SetScene Tasks')
+ self.state = runQueueSceneRun
if self.state is runQueueSceneRun:
retval = self.rqexe.sq_execute()
@@ -1461,11 +1473,7 @@ class RunQueue:
self.state = runQueueComplete
if self.state is runQueueRunInit:
- # Just in case we didn't setscene
- self.rqdata.init_progress_reporter.finish()
logger.info("Executing RunQueue Tasks")
- if not self.rqexe:
- self.rqexe = RunQueueExecute(self)
start_runqueue_tasks(self.rqexe)
self.state = runQueueRunning
@@ -1690,7 +1698,7 @@ def process_setscene_whitelist(rq, rqdata, stampcache, sched, rqex):
def check_norun_task(tid, showerror=False):
(mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
# Ignore covered tasks
- if tid in rq.scenequeue_covered:
+ if tid in rqex.scenequeue_covered:
return False
# Ignore stamped tasks
if rq.check_stamp_task(tid, taskname, cache=stampcache):
@@ -1765,6 +1773,24 @@ class RunQueueExecute:
if self.number_tasks <= 0:
bb.fatal("Invalid BB_NUMBER_THREADS %s" % self.number_tasks)
+ self.scenequeue_covered = set()
+ self.scenequeue_notcovered = set()
+ self.scenequeue_notneeded = set()
+
+ if len(self.rqdata.runq_setscene_tids) > 0:
+ self.sqdata = SQData()
+ build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self)
+
+ schedulers = self.get_schedulers()
+ for scheduler in schedulers:
+ if self.scheduler == scheduler.name:
+ self.sched = scheduler(self, self.rqdata)
+ logger.debug(1, "Using runqueue scheduler '%s'", scheduler.name)
+ break
+ else:
+ bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" %
+ (self.scheduler, ", ".join(obj.name for obj in schedulers)))
+
def runqueue_process_waitpid(self, task, status):
# self.build_stamps[pid] may not exist when use shared work directory.
@@ -1937,7 +1963,7 @@ class RunQueueExecute:
if task is not None:
(mc, fn, taskname, taskfn) = split_tid_mcfn(task)
- if task in self.rq.scenequeue_covered:
+ if task in self.scenequeue_covered:
logger.debug(2, "Setscene covered task %s", task)
self.task_skip(task, "covered")
return True
@@ -2194,10 +2220,7 @@ class RunQueueExecute:
# revdeps = self.sqdata.sq_revdeps[tid]
# bb.warn("Found we didn't run %s %s %s" % (tid, buildable, str(revdeps)))
- self.rq.scenequeue_covered = self.scenequeue_covered
- self.rq.scenequeue_notcovered = self.scenequeue_notcovered
-
- logger.debug(1, 'We can skip tasks %s', "\n".join(sorted(self.rq.scenequeue_covered)))
+ logger.debug(1, 'We can skip tasks %s', "\n".join(sorted(self.scenequeue_covered)))
self.rq.state = runQueueRunInit
@@ -2479,29 +2502,28 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
sqdata.outrightfail.append(tid)
def start_runqueue_tasks(rqexec):
-
# Mark initial buildable tasks
for tid in rqexec.rqdata.runtaskentries:
if len(rqexec.rqdata.runtaskentries[tid].depends) == 0:
rqexec.runq_buildable.add(tid)
- if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.rq.scenequeue_covered):
- rqexec.rq.scenequeue_covered.add(tid)
+ if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.scenequeue_covered):
+ rqexec.scenequeue_covered.add(tid)
found = True
while found:
found = False
for tid in rqexec.rqdata.runtaskentries:
- if tid in rqexec.rq.scenequeue_covered:
+ if tid in rqexec.scenequeue_covered:
continue
logger.debug(1, 'Considering %s: %s' % (tid, str(rqexec.rqdata.runtaskentries[tid].revdeps)))
- if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.rq.scenequeue_covered):
- if tid in rqexec.rq.scenequeue_notcovered:
+ if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.scenequeue_covered):
+ if tid in rqexec.scenequeue_notcovered:
continue
found = True
- rqexec.rq.scenequeue_covered.add(tid)
+ rqexec.scenequeue_covered.add(tid)
- logger.debug(1, 'Skip list %s', sorted(rqexec.rq.scenequeue_covered))
+ logger.debug(1, 'Skip list %s', sorted(rqexec.scenequeue_covered))
for task in self.rq.scenequeue_notcovered:
logger.debug(1, 'Not skipping task %s', task)
@@ -2514,35 +2536,8 @@ def start_runqueue_tasks(rqexec):
target_pairs.append((fn, taskname))
event.fire(bb.event.StampUpdate(target_pairs, rqexec.rqdata.dataCaches[mc].stamp), rqexec.cfgData)
+ rqexec.sched.initbuildable()
- schedulers = rqexec.get_schedulers()
- for scheduler in schedulers:
- if rqexec.scheduler == scheduler.name:
- rqexec.sched = scheduler(rqexec, rqexec.rqdata)
- logger.debug(1, "Using runqueue scheduler '%s'", scheduler.name)
- break
- else:
- bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" %
- (rqexec.scheduler, ", ".join(obj.name for obj in schedulers)))
-
-def start_scenequeue_tasks(rqexec):
- rqexec.scenequeue_covered = set()
- rqexec.scenequeue_notcovered = set()
- rqexec.scenequeue_notneeded = set()
-
- # If we don't have any setscene functions, skip this step
- if len(rqexec.rqdata.runq_setscene_tids) == 0:
- rqexec.rq.scenequeue_covered = set()
- rqexec.rq.scenequeue_notcovered = set()
- rqexec.rq.state = runQueueRunInit
- return
-
- rqexec.sqdata = SQData()
- build_scenequeue_data(rqexec.sqdata, rqexec.rqdata, rqexec.rq, rqexec.cooker, rqexec.stampcache, rqexec)
-
- logger.info('Executing SetScene Tasks')
-
- rqexec.rq.state = runQueueSceneRun
class TaskFailure(Exception):
"""
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 14/26] event/runqueue: Drop StampUpdate event, its pointless/unused
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (11 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 13/26] runqueue: Fold remains of the scenequeue setup into RunQueueExecute Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 15/26] runqueue: Add covered_tasks (or 'collated_deps') to scenequeue data Richard Purdie
` (12 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Whilst this class has existed for years, it doesn't have any
users and has a questionable interface. Drop it to allow for further
simplification and changes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/event.py | 17 -----------------
lib/bb/runqueue.py | 9 ---------
lib/bb/tests/event.py | 8 --------
lib/bb/ui/knotty.py | 1 -
4 files changed, 35 deletions(-)
diff --git a/lib/bb/event.py b/lib/bb/event.py
index 248fc1a6f8..d44621edf4 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -404,23 +404,6 @@ class RecipeTaskPreProcess(RecipeEvent):
class RecipeParsed(RecipeEvent):
""" Recipe Parsing Complete """
-class StampUpdate(Event):
- """Trigger for any adjustment of the stamp files to happen"""
-
- def __init__(self, targets, stampfns):
- self._targets = targets
- self._stampfns = stampfns
- Event.__init__(self)
-
- def getStampPrefix(self):
- return self._stampfns
-
- def getTargets(self):
- return self._targets
-
- stampPrefix = property(getStampPrefix)
- targets = property(getTargets)
-
class BuildBase(Event):
"""Base class for bitbake build events"""
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 46146466fc..30b13d54a8 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2528,17 +2528,8 @@ def start_runqueue_tasks(rqexec):
for task in self.rq.scenequeue_notcovered:
logger.debug(1, 'Not skipping task %s', task)
- for mc in rqexec.rqdata.dataCaches:
- target_pairs = []
- for tid in rqexec.rqdata.target_tids:
- (tidmc, fn, taskname, _) = split_tid_mcfn(tid)
- if tidmc == mc:
- target_pairs.append((fn, taskname))
-
- event.fire(bb.event.StampUpdate(target_pairs, rqexec.rqdata.dataCaches[mc].stamp), rqexec.cfgData)
rqexec.sched.initbuildable()
-
class TaskFailure(Exception):
"""
Exception raised when a task in a runqueue fails
diff --git a/lib/bb/tests/event.py b/lib/bb/tests/event.py
index b6e40c6ae9..9229b63d47 100644
--- a/lib/bb/tests/event.py
+++ b/lib/bb/tests/event.py
@@ -561,14 +561,6 @@ class EventClassesTest(unittest.TestCase):
self.assertEqual(event.fn(1), callback(1))
self.assertEqual(event.pid, EventClassesTest._worker_pid)
- def test_StampUpdate(self):
- targets = ["foo", "bar"]
- stampfns = [lambda:"foobar"]
- event = bb.event.StampUpdate(targets, stampfns)
- self.assertEqual(event.targets, targets)
- self.assertEqual(event.stampPrefix, stampfns)
- self.assertEqual(event.pid, EventClassesTest._worker_pid)
-
def test_BuildBase(self):
""" Test base class for bitbake build events """
name = "foo"
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 88f638fb38..1c72aa2947 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -660,7 +660,6 @@ def main(server, eventHandler, params, tf = TerminalFilter):
# ignore
if isinstance(event, (bb.event.BuildBase,
bb.event.MetadataEvent,
- bb.event.StampUpdate,
bb.event.ConfigParsed,
bb.event.MultiConfigParsed,
bb.event.RecipeParsed,
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 15/26] runqueue: Add covered_tasks (or 'collated_deps') to scenequeue data
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (12 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 14/26] event/runqueue: Drop StampUpdate event, its pointless/unused Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 16/26] runqueue: Simplify scenequeue unskippable calculation Richard Purdie
` (11 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Its useful to have a list of all the tasks a given setscene task covers
and we can easily generate this data whilst doing other data processing.
This is used in later changes to runqueue rather than trying to compute it
on the fly which is difficult.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 30b13d54a8..1d7706a219 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2278,6 +2278,7 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
sq_revdeps = {}
sq_revdeps_new = {}
sq_revdeps_squash = {}
+ sq_collated_deps = {}
# We need to construct a dependency graph for the setscene functions. Intermediate
# dependencies between the setscene tasks only complicate the code. This code
@@ -2299,6 +2300,7 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
# Secondly process the chains between setscene tasks.
for tid in rqdata.runq_setscene_tids:
+ sq_collated_deps[tid] = set()
#bb.warn("Added endpoint 2 %s" % (tid))
for dep in rqdata.runtaskentries[tid].depends:
if tid in sq_revdeps[dep]:
@@ -2318,6 +2320,9 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
tasks |= task
if sq_revdeps_new[point]:
tasks |= sq_revdeps_new[point]
+ if point not in rqdata.runq_setscene_tids:
+ for t in tasks:
+ sq_collated_deps[t].add(point)
sq_revdeps_new[point] = set()
if point in rqdata.runq_setscene_tids:
sq_revdeps_new[point] = tasks
@@ -2430,6 +2435,7 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
sqdata.sq_deps = {}
sqdata.sq_revdeps = sq_revdeps_squash
sqdata.sq_revdeps2 = copy.deepcopy(sqdata.sq_revdeps)
+ sqdata.sq_covered_tasks = sq_collated_deps
for tid in sqdata.sq_revdeps:
sqdata.sq_deps[tid] = set()
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 16/26] runqueue: Simplify scenequeue unskippable calculation
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (13 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 15/26] runqueue: Add covered_tasks (or 'collated_deps') to scenequeue data Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 17/26] runqueue: Tweak comments and debug code Richard Purdie
` (10 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
The existing code to compute the 'unskippable' setscene task list is overcomlicated,
so replace it with something functionally equivalent but simpler and more efficient.
We don't need to process all chains, just the 'top' ones to the first setscene tasks.
This also makes the code more readable.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 45 +++++++++++++--------------------------------
1 file changed, 13 insertions(+), 32 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 1d7706a219..1f41340170 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2271,7 +2271,7 @@ class SQData(object):
def __init__(self):
self.sq_harddeps = {}
self.stamps = {}
- self.unskippable = []
+ self.unskippable = set()
def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
@@ -2344,38 +2344,19 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
# Build a list of setscene tasks which are "unskippable"
# These are direct endpoints referenced by the build
- endpoints2 = {}
- sq_revdeps2 = {}
- sq_revdeps_new2 = {}
- def process_endpoints2(endpoints):
- newendpoints = {}
- for point, task in endpoints.items():
- tasks = set([point])
- if task:
- tasks |= task
- if sq_revdeps_new2[point]:
- tasks |= sq_revdeps_new2[point]
- sq_revdeps_new2[point] = set()
- if point in rqdata.runq_setscene_tids:
- sq_revdeps_new2[point] = tasks
- for dep in rqdata.runtaskentries[point].depends:
- if point in sq_revdeps2[dep]:
- sq_revdeps2[dep].remove(point)
- if tasks:
- sq_revdeps_new2[dep] |= tasks
- if (len(sq_revdeps2[dep]) == 0 or len(sq_revdeps_new2[dep]) != 0) and dep not in rqdata.runq_setscene_tids:
- newendpoints[dep] = tasks
- if len(newendpoints) != 0:
- process_endpoints2(newendpoints)
+ # Take the build endpoints (no revdeps) and find the sstate tasks they depend upon
+ new = True
for tid in rqdata.runtaskentries:
- sq_revdeps2[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps)
- sq_revdeps_new2[tid] = set()
- if (len(sq_revdeps2[tid]) == 0) and tid not in rqdata.runq_setscene_tids:
- endpoints2[tid] = set()
- process_endpoints2(endpoints2)
- for tid in rqdata.runq_setscene_tids:
- if sq_revdeps_new2[tid]:
- sqdata.unskippable.append(tid)
+ if len(rqdata.runtaskentries[tid].revdeps) == 0:
+ sqdata.unskippable.add(tid)
+ while new:
+ new = False
+ for tid in sqdata.unskippable.copy():
+ if tid in rqdata.runq_setscene_tids:
+ continue
+ sqdata.unskippable.remove(tid)
+ sqdata.unskippable |= rqdata.runtaskentries[tid].depends
+ new = True
rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries))
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 17/26] runqueue: Tweak comments and debug code
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (14 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 16/26] runqueue: Simplify scenequeue unskippable calculation Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 18/26] runqueue: Code simplification Richard Purdie
` (9 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Add some extra comments to build_scenequeue_data() and fix the debug code
so it actually works.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 1f41340170..05bc9f9488 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2360,6 +2360,7 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries))
+ # Sanity check all dependencies could be changed to setscene task references
for taskcounter, tid in enumerate(rqdata.runtaskentries):
if tid in rqdata.runq_setscene_tids:
deps = set()
@@ -2409,15 +2410,17 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
rqdata.init_progress_reporter.next_stage()
#for tid in sq_revdeps_squash:
+ # data = ""
# for dep in sq_revdeps_squash[tid]:
# data = data + "\n %s" % dep
- # bb.warn("Task %s_setscene: is %s " % (tid, data
+ # bb.warn("Task %s_setscene: is %s " % (tid, data))
sqdata.sq_deps = {}
sqdata.sq_revdeps = sq_revdeps_squash
sqdata.sq_revdeps2 = copy.deepcopy(sqdata.sq_revdeps)
sqdata.sq_covered_tasks = sq_collated_deps
+ # Build reverse version of revdeps to populate deps structure
for tid in sqdata.sq_revdeps:
sqdata.sq_deps[tid] = set()
for tid in sqdata.sq_revdeps:
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 18/26] runqueue: Code simplification
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (15 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 17/26] runqueue: Tweak comments and debug code Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 19/26] runqueue: Remove pointless variable Richard Purdie
` (8 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Simplfy some looping code which no longer has any purpose.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 05bc9f9488..13e90e709a 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2363,10 +2363,7 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
# Sanity check all dependencies could be changed to setscene task references
for taskcounter, tid in enumerate(rqdata.runtaskentries):
if tid in rqdata.runq_setscene_tids:
- deps = set()
- for dep in sq_revdeps_new[tid]:
- deps.add(dep)
- sq_revdeps_squash[tid] = deps
+ sq_revdeps_squash[tid] = set(sq_revdeps_new[tid])
elif len(sq_revdeps_new[tid]) != 0:
bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
rqdata.init_progress_reporter.update(taskcounter)
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 19/26] runqueue: Remove pointless variable
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (16 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 18/26] runqueue: Code simplification Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 20/26] runqueue: Further scheduler buildable tasks cleanup Richard Purdie
` (7 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Its now clear a variable is pointless, remove it and tweak the logic
so the data structure of the existing variable matches what we need.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 19 ++++++++++---------
1 file changed, 10 insertions(+), 9 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 13e90e709a..97c825a546 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2276,7 +2276,6 @@ class SQData(object):
def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
sq_revdeps = {}
- sq_revdeps_new = {}
sq_revdeps_squash = {}
sq_collated_deps = {}
@@ -2291,7 +2290,7 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
endpoints = {}
for tid in rqdata.runtaskentries:
sq_revdeps[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps)
- sq_revdeps_new[tid] = set()
+ sq_revdeps_squash[tid] = set()
if (len(sq_revdeps[tid]) == 0) and tid not in rqdata.runq_setscene_tids:
#bb.warn("Added endpoint %s" % (tid))
endpoints[tid] = set()
@@ -2318,21 +2317,21 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
tasks = set()
if task:
tasks |= task
- if sq_revdeps_new[point]:
- tasks |= sq_revdeps_new[point]
+ if sq_revdeps_squash[point]:
+ tasks |= sq_revdeps_squash[point]
if point not in rqdata.runq_setscene_tids:
for t in tasks:
sq_collated_deps[t].add(point)
- sq_revdeps_new[point] = set()
+ sq_revdeps_squash[point] = set()
if point in rqdata.runq_setscene_tids:
- sq_revdeps_new[point] = tasks
+ sq_revdeps_squash[point] = tasks
tasks = set()
continue
for dep in rqdata.runtaskentries[point].depends:
if point in sq_revdeps[dep]:
sq_revdeps[dep].remove(point)
if tasks:
- sq_revdeps_new[dep] |= tasks
+ sq_revdeps_squash[dep] |= tasks
if len(sq_revdeps[dep]) == 0 and dep not in rqdata.runq_setscene_tids:
newendpoints[dep] = task
if len(newendpoints) != 0:
@@ -2363,9 +2362,11 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
# Sanity check all dependencies could be changed to setscene task references
for taskcounter, tid in enumerate(rqdata.runtaskentries):
if tid in rqdata.runq_setscene_tids:
- sq_revdeps_squash[tid] = set(sq_revdeps_new[tid])
- elif len(sq_revdeps_new[tid]) != 0:
+ pass
+ elif len(sq_revdeps_squash[tid]) != 0:
bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
+ else:
+ del sq_revdeps_squash[tid]
rqdata.init_progress_reporter.update(taskcounter)
rqdata.init_progress_reporter.next_stage()
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 20/26] runqueue: Further scheduler buildable tasks cleanup
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (17 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 19/26] runqueue: Remove pointless variable Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 21/26] runqueue: Clarify scenequeue_covered vs. tasks_covered Richard Purdie
` (6 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
The code for setting up buildable tasks can be simplified.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 97c825a546..580323f285 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -196,11 +196,6 @@ class RunQueueScheduler(object):
if self.rq.can_start_task():
return self.next_buildable_task()
- def initbuildable(self):
- for tid in self.rqdata.runtaskentries:
- if tid in self.rq.runq_buildable:
- self.buildable.append(tid)
-
def newbuildable(self, task):
self.buildable.append(task)
@@ -2493,7 +2488,7 @@ def start_runqueue_tasks(rqexec):
# Mark initial buildable tasks
for tid in rqexec.rqdata.runtaskentries:
if len(rqexec.rqdata.runtaskentries[tid].depends) == 0:
- rqexec.runq_buildable.add(tid)
+ rqexec.setbuildable(tid)
if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.scenequeue_covered):
rqexec.scenequeue_covered.add(tid)
@@ -2516,8 +2511,6 @@ def start_runqueue_tasks(rqexec):
for task in self.rq.scenequeue_notcovered:
logger.debug(1, 'Not skipping task %s', task)
- rqexec.sched.initbuildable()
-
class TaskFailure(Exception):
"""
Exception raised when a task in a runqueue fails
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 21/26] runqueue: Clarify scenequeue_covered vs. tasks_covered
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (18 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 20/26] runqueue: Further scheduler buildable tasks cleanup Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 22/26] runqueue: Merge the queues and execute setscene and normal tasks in parallel Richard Purdie
` (5 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
It wasn't clear whether the variable contained just setscene covered
tasks or all covered tasks. We need both sets of data so lets just have
two clearly named variables.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 580323f285..53cf4c5c82 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1693,7 +1693,7 @@ def process_setscene_whitelist(rq, rqdata, stampcache, sched, rqex):
def check_norun_task(tid, showerror=False):
(mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
# Ignore covered tasks
- if tid in rqex.scenequeue_covered:
+ if tid in rqex.tasks_covered:
return False
# Ignore stamped tasks
if rq.check_stamp_task(tid, taskname, cache=stampcache):
@@ -1768,7 +1768,10 @@ class RunQueueExecute:
if self.number_tasks <= 0:
bb.fatal("Invalid BB_NUMBER_THREADS %s" % self.number_tasks)
+ # List of setscene tasks which we've covered
self.scenequeue_covered = set()
+ # List of tasks which are covered (including setscene ones)
+ self.tasks_covered = set()
self.scenequeue_notcovered = set()
self.scenequeue_notneeded = set()
@@ -1958,7 +1961,7 @@ class RunQueueExecute:
if task is not None:
(mc, fn, taskname, taskfn) = split_tid_mcfn(task)
- if task in self.scenequeue_covered:
+ if task in self.tasks_covered:
logger.debug(2, "Setscene covered task %s", task)
self.task_skip(task, "covered")
return True
@@ -2089,6 +2092,7 @@ class RunQueueExecute:
logger.debug(1, 'Found task %s which could be accelerated', task)
self.scenequeue_covered.add(task)
+ self.tasks_covered.add(task)
self.scenequeue_updatecounters(task)
def sq_check_taskfail(self, task):
@@ -2489,24 +2493,24 @@ def start_runqueue_tasks(rqexec):
for tid in rqexec.rqdata.runtaskentries:
if len(rqexec.rqdata.runtaskentries[tid].depends) == 0:
rqexec.setbuildable(tid)
- if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.scenequeue_covered):
- rqexec.scenequeue_covered.add(tid)
+ if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.tasks_covered):
+ rqexec.tasks_covered.add(tid)
found = True
while found:
found = False
for tid in rqexec.rqdata.runtaskentries:
- if tid in rqexec.scenequeue_covered:
+ if tid in rqexec.tasks_covered:
continue
logger.debug(1, 'Considering %s: %s' % (tid, str(rqexec.rqdata.runtaskentries[tid].revdeps)))
- if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.scenequeue_covered):
+ if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.tasks_covered):
if tid in rqexec.scenequeue_notcovered:
continue
found = True
- rqexec.scenequeue_covered.add(tid)
+ rqexec.tasks_covered.add(tid)
- logger.debug(1, 'Skip list %s', sorted(rqexec.scenequeue_covered))
+ logger.debug(1, 'Skip list %s', sorted(rqexec.tasks_covered))
for task in self.rq.scenequeue_notcovered:
logger.debug(1, 'Not skipping task %s', task)
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 22/26] runqueue: Merge the queues and execute setscene and normal tasks in parallel
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (19 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 21/26] runqueue: Clarify scenequeue_covered vs. tasks_covered Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 23/26] runqueue: Alter setscenewhitelist handling Richard Purdie
` (4 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
This is the serious functionality change in this runqueue patch series of
changes.
Rather than two phases of execution, the scenequeue setscene phase, followed
by normal task exeuction, this change allows them to execute in parallel
together.
To do this we need to handle marking of tasks as covered/uncovered in a piecemeal
fashion on a task by task basis rather than in a single function.
The code will block normal task exeuction until any setcene task which could
cover that task is executed and its status is known. There is a slight
optimisation which could be possible here at the risk of races but that
doesn't seem worthwhile.
The state engine isn't entirely cleaned up in this commit (see FIXME) and
the setscenewhitelist functionality is broken by it (see following patches)
however its good enough to test with normal workflows.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 176 +++++++++++++++++++++++++++++++--------------
1 file changed, 123 insertions(+), 53 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 53cf4c5c82..e4994f6c52 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -142,7 +142,7 @@ class RunQueueScheduler(object):
Return the id of the first task we find that is buildable
"""
self.buildable = [x for x in self.buildable if x not in self.rq.runq_running]
- buildable = self.buildable
+ buildable = [x for x in self.buildable if (x in self.rq.tasks_covered or x in self.rq.tasks_notcovered)]
if not buildable:
return None
@@ -1454,25 +1454,18 @@ class RunQueue:
# If we don't have any setscene functions, skip execution
if len(self.rqdata.runq_setscene_tids) == 0:
- self.rqdata.init_progress_reporter.finish()
- self.state = runQueueRunInit
- else:
- logger.info('Executing SetScene Tasks')
- self.state = runQueueSceneRun
-
- if self.state is runQueueSceneRun:
- retval = self.rqexe.sq_execute()
-
- if self.state is runQueueRunInit:
- if self.cooker.configuration.setsceneonly:
- self.state = runQueueComplete
-
- if self.state is runQueueRunInit:
- logger.info("Executing RunQueue Tasks")
- start_runqueue_tasks(self.rqexe)
+ logger.info('No setscene tasks')
+ for tid in self.rqdata.runtaskentries:
+ if len(self.rqdata.runtaskentries[tid].depends) == 0:
+ self.rqexe.setbuildable(tid)
+ self.rqexe.tasks_notcovered.add(tid)
+ self.rqexe.sqdone = True
+ logger.info('Executing Tasks')
self.state = runQueueRunning
if self.state is runQueueRunning:
+ retval = self.rqexe.sq_execute()
+ # FIXME revtal
retval = self.rqexe.execute()
if self.state is runQueueCleanUp:
@@ -1757,6 +1750,8 @@ class RunQueueExecute:
self.stampcache = {}
+ self.sqdone = False
+
self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
self.sq_stats = RunQueueStats(len(self.rqdata.runq_setscene_tids))
@@ -1772,12 +1767,12 @@ class RunQueueExecute:
self.scenequeue_covered = set()
# List of tasks which are covered (including setscene ones)
self.tasks_covered = set()
+ self.tasks_scenequeue_done = set()
self.scenequeue_notcovered = set()
+ self.tasks_notcovered = set()
self.scenequeue_notneeded = set()
- if len(self.rqdata.runq_setscene_tids) > 0:
- self.sqdata = SQData()
- build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self)
+ self.coveredtopocess = set()
schedulers = self.get_schedulers()
for scheduler in schedulers:
@@ -1789,6 +1784,10 @@ class RunQueueExecute:
bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" %
(self.scheduler, ", ".join(obj.name for obj in schedulers)))
+ if len(self.rqdata.runq_setscene_tids) > 0:
+ self.sqdata = SQData()
+ build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self)
+
def runqueue_process_waitpid(self, task, status):
# self.build_stamps[pid] may not exist when use shared work directory.
@@ -1951,6 +1950,9 @@ class RunQueueExecute:
if process_setscenewhitelist(self.rq, self.rqdata, self.stampcache, self.sched, self):
return True
+ if self.cooker.configuration.setsceneonly:
+ return True
+
self.rq.read_workers()
if self.stats.total == 0:
@@ -2014,7 +2016,7 @@ class RunQueueExecute:
if self.can_start_task():
return True
- if self.stats.active > 0:
+ if self.stats.active > 0 or self.sq_stats.active > 0:
self.rq.read_workers()
return self.rq.active_fds()
@@ -2026,9 +2028,9 @@ class RunQueueExecute:
for task in self.rqdata.runtaskentries:
if task not in self.runq_buildable:
logger.error("Task %s never buildable!", task)
- if task not in self.runq_running:
+ elif task not in self.runq_running:
logger.error("Task %s never ran!", task)
- if task not in self.runq_complete:
+ elif task not in self.runq_complete:
logger.error("Task %s never completed!", task)
self.rq.state = runQueueComplete
@@ -2070,6 +2072,84 @@ class RunQueueExecute:
#bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n"))
return taskdepdata
+ def scenequeue_process_notcovered(self, task):
+ logger.debug(1, 'Not skipping setscene task %s', task)
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
+ taskname = taskname + '_setscene'
+ bb.build.del_stamp(taskname, self.rqdata.dataCaches[mc], taskfn)
+ if len(self.rqdata.runtaskentries[task].depends) == 0:
+ self.setbuildable(task)
+ notcovered = set([task])
+ while notcovered:
+ new = set()
+ for t in notcovered:
+ for deptask in self.rqdata.runtaskentries[t].depends:
+ if deptask in notcovered or deptask in new or deptask in self.rqdata.runq_setscene_tids or deptask in self.tasks_notcovered:
+ continue
+ logger.debug(1, 'Task %s depends on non-setscene task %s so not skipping' % (t, deptask))
+ new.add(deptask)
+ self.tasks_notcovered.add(deptask)
+ if len(self.rqdata.runtaskentries[deptask].depends) == 0:
+ self.setbuildable(deptask)
+ notcovered = new
+
+ def scenequeue_process_unskippable(self, task):
+ # Look up the dependency chain for non-setscene things which depend on this task
+ # and mark as 'done'/notcovered
+ ready = set([task])
+ while ready:
+ new = set()
+ for t in ready:
+ for deptask in self.rqdata.runtaskentries[t].revdeps:
+ if deptask in ready or deptask in new or deptask in self.tasks_scenequeue_done or deptask in self.rqdata.runq_setscene_tids:
+ continue
+ if self.rqdata.runtaskentries[deptask].depends.issubset(self.tasks_scenequeue_done):
+ new.add(deptask)
+ self.tasks_scenequeue_done.add(deptask)
+ self.tasks_notcovered.add(deptask)
+ #logger.warning("Up: " + str(deptask))
+ ready = new
+
+
+ def scenequeue_donetask(self, task):
+
+ next = set([task])
+ while next:
+ new = set()
+ for t in next:
+ self.tasks_scenequeue_done.add(t)
+ # Look down the dependency chain for non-setscene things which this task depends on
+ # and mark as 'done'
+ for dep in self.rqdata.runtaskentries[t].depends:
+ if dep in self.rqdata.runq_setscene_tids or dep in self.tasks_scenequeue_done:
+ continue
+ if self.rqdata.runtaskentries[dep].revdeps.issubset(self.tasks_scenequeue_done):
+ new.add(dep)
+ #logger.warning(" Down: " + dep)
+ next = new
+
+ if task in self.sqdata.unskippable:
+ self.scenequeue_process_unskippable(task)
+
+ if task in self.scenequeue_notcovered:
+ self.scenequeue_process_notcovered(task)
+ elif task in self.scenequeue_covered:
+ logger.debug(1, 'Queued setscene task %s', task)
+ self.coveredtopocess.add(task)
+
+ for task in self.coveredtopocess.copy():
+ if self.sqdata.sq_covered_tasks[task].issubset(self.tasks_scenequeue_done):
+ logger.debug(1, 'Processing setscene task %s', task)
+ covered = self.sqdata.sq_covered_tasks[task]
+ covered.add(task)
+ # Remove notcovered tasks
+ covered.difference_update(self.tasks_notcovered)
+ self.tasks_covered.update(covered)
+ self.coveredtopocess.remove(task)
+ for tid in covered:
+ if len(self.rqdata.runtaskentries[tid].depends) == 0:
+ self.setbuildable(tid)
+
def scenequeue_updatecounters(self, task, fail = False):
for dep in self.sqdata.sq_deps[task]:
if fail and task in self.sqdata.sq_harddeps and dep in self.sqdata.sq_harddeps[task]:
@@ -2083,6 +2163,9 @@ class RunQueueExecute:
if len(self.sqdata.sq_revdeps2[dep]) == 0:
self.sq_buildable.add(dep)
+ self.scenequeue_donetask(task)
+
+
def sq_task_completeoutright(self, task):
"""
Mark a task as completed
@@ -2113,6 +2196,7 @@ class RunQueueExecute:
self.sq_stats.taskFailed()
bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats, result, self), self.cfgData)
self.scenequeue_notcovered.add(task)
+ self.tasks_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
self.sq_check_taskfail(task)
@@ -2122,6 +2206,7 @@ class RunQueueExecute:
self.sq_stats.taskSkipped()
self.sq_stats.taskCompleted()
self.scenequeue_notcovered.add(task)
+ self.tasks_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
def sq_task_skip(self, task):
@@ -2136,6 +2221,9 @@ class RunQueueExecute:
Run the tasks in a queue prepared by prepare_runqueue
"""
+ if self.sqdone:
+ return True
+
self.rq.read_workers()
task = None
@@ -2209,7 +2297,7 @@ class RunQueueExecute:
if self.can_start_task():
return True
- if self.sq_stats.active > 0:
+ if self.stats.active > 0 or self.sq_stats.active > 0:
self.rq.read_workers()
return self.rq.active_fds()
@@ -2221,11 +2309,14 @@ class RunQueueExecute:
logger.debug(1, 'We can skip tasks %s', "\n".join(sorted(self.scenequeue_covered)))
- self.rq.state = runQueueRunInit
-
completeevent = sceneQueueComplete(self.sq_stats, self.rq)
bb.event.fire(completeevent, self.cfgData)
+ if self.cooker.configuration.setsceneonly:
+ self.rq.state = runQueueComplete
+
+ self.sqdone = True
+
return True
def sq_build_taskdepdata(self, task):
@@ -2353,6 +2444,12 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
if tid in rqdata.runq_setscene_tids:
continue
sqdata.unskippable.remove(tid)
+ if len(rqdata.runtaskentries[tid].depends) == 0:
+ # These are tasks which have no setscene tasks in their chain, need to mark as directly buildable
+ sqrq.tasks_notcovered.add(tid)
+ sqrq.tasks_scenequeue_done.add(tid)
+ sqrq.setbuildable(tid)
+ sqrq.scenequeue_process_unskippable(tid)
sqdata.unskippable |= rqdata.runtaskentries[tid].depends
new = True
@@ -2488,33 +2585,6 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
logger.debug(2, 'No package found, so skipping setscene task %s', tid)
sqdata.outrightfail.append(tid)
-def start_runqueue_tasks(rqexec):
- # Mark initial buildable tasks
- for tid in rqexec.rqdata.runtaskentries:
- if len(rqexec.rqdata.runtaskentries[tid].depends) == 0:
- rqexec.setbuildable(tid)
- if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.tasks_covered):
- rqexec.tasks_covered.add(tid)
-
- found = True
- while found:
- found = False
- for tid in rqexec.rqdata.runtaskentries:
- if tid in rqexec.tasks_covered:
- continue
- logger.debug(1, 'Considering %s: %s' % (tid, str(rqexec.rqdata.runtaskentries[tid].revdeps)))
-
- if len(rqexec.rqdata.runtaskentries[tid].revdeps) > 0 and rqexec.rqdata.runtaskentries[tid].revdeps.issubset(rqexec.tasks_covered):
- if tid in rqexec.scenequeue_notcovered:
- continue
- found = True
- rqexec.tasks_covered.add(tid)
-
- logger.debug(1, 'Skip list %s', sorted(rqexec.tasks_covered))
-
- for task in self.rq.scenequeue_notcovered:
- logger.debug(1, 'Not skipping task %s', task)
-
class TaskFailure(Exception):
"""
Exception raised when a task in a runqueue fails
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 23/26] runqueue: Alter setscenewhitelist handling
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (20 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 22/26] runqueue: Merge the queues and execute setscene and normal tasks in parallel Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 24/26] runqueue: Complete the merge of scenequeue and normal task execution Richard Purdie
` (3 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
Since there is now parallel execution of setscene and normal tasks, the way
setscenewhitelist handling worked can't function the way it did. Paul and I
never liked its error output anyway.
This code tries a different approach, checking the task at execution time
but printing the uncovered task list.
This code may need improvement after real world usage but can
work with the new task flows.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 78 +++++++++++++++++-----------------------------
1 file changed, 29 insertions(+), 49 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index e4994f6c52..e7ba610723 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1681,49 +1681,6 @@ class RunQueue:
output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb)
bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, closest matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output))
-def process_setscene_whitelist(rq, rqdata, stampcache, sched, rqex):
- # Check tasks that are going to run against the whitelist
- def check_norun_task(tid, showerror=False):
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- # Ignore covered tasks
- if tid in rqex.tasks_covered:
- return False
- # Ignore stamped tasks
- if rq.check_stamp_task(tid, taskname, cache=stampcache):
- return False
- # Ignore noexec tasks
- taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
- if 'noexec' in taskdep and taskname in taskdep['noexec']:
- return False
-
- pn = rqdata.dataCaches[mc].pkg_fn[taskfn]
- if not check_setscene_enforce_whitelist(pn, taskname, rqdata.setscenewhitelist):
- if showerror:
- if tid in rqdata.runq_setscene_tids:
- logger.error('Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname))
- else:
- logger.error('Task %s.%s attempted to execute unexpectedly' % (pn, taskname))
- return True
- return False
- # Look to see if any tasks that we think shouldn't run are going to
- unexpected = False
- for tid in rqdata.runtaskentries:
- if check_norun_task(tid):
- unexpected = True
- break
- if unexpected:
- # Run through the tasks in the rough order they'd have executed and print errors
- # (since the order can be useful - usually missing sstate for the last few tasks
- # is the cause of the problem)
- task = sched.next()
- while task is not None:
- check_norun_task(task, showerror=True)
- rqex.task_skip(task, 'Setscene enforcement check')
- task = sched.next()
-
- rq.state = runQueueCleanUp
- return True
-
class RunQueueExecute:
@@ -1944,12 +1901,6 @@ class RunQueueExecute:
Run the tasks in a queue prepared by rqdata.prepare()
"""
- if self.rqdata.setscenewhitelist is not None and not self.rqdata.setscenewhitelist_checked:
- self.rqdata.setscenewhitelist_checked = True
-
- if process_setscenewhitelist(self.rq, self.rqdata, self.stampcache, self.sched, self):
- return True
-
if self.cooker.configuration.setsceneonly:
return True
@@ -1963,6 +1914,11 @@ class RunQueueExecute:
if task is not None:
(mc, fn, taskname, taskfn) = split_tid_mcfn(task)
+ if self.rqdata.setscenewhitelist is not None:
+ if self.check_setscenewhitelist(task):
+ self.task_fail(task, "setscene whitelist")
+ return True
+
if task in self.tasks_covered:
logger.debug(2, "Setscene covered task %s", task)
self.task_skip(task, "covered")
@@ -2357,6 +2313,30 @@ class RunQueueExecute:
#bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n"))
return taskdepdata
+ def check_setscenewhitelist(self, tid):
+ # Check task that is going to run against the whitelist
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ # Ignore covered tasks
+ if tid in self.tasks_covered:
+ return False
+ # Ignore stamped tasks
+ if self.rq.check_stamp_task(tid, taskname, cache=self.stampcache):
+ return False
+ # Ignore noexec tasks
+ taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
+ if 'noexec' in taskdep and taskname in taskdep['noexec']:
+ return False
+
+ pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
+ if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
+ if tid in self.rqdata.runq_setscene_tids:
+ msg = 'Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname)
+ else:
+ msg = 'Task %s.%s attempted to execute unexpectedly' % (pn, taskname)
+ logger.error(msg + '\nThis is usually due to missing setscene tasks. Those missing in this build were: %s' % pprint.pformat(self.scenequeue_notcovered))
+ return True
+ return False
+
class SQData(object):
def __init__(self):
self.sq_harddeps = {}
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 24/26] runqueue: Complete the merge of scenequeue and normal task execution
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (21 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 23/26] runqueue: Alter setscenewhitelist handling Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 25/26] siggen: Fix default handler Richard Purdie
` (2 subsequent siblings)
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
This combines the scqenequeue and normal task execution into one function
and simplifies the state engine accordingly.
This is the final set of cleanup to fully merge things without adding the
extra noise to the previous commits.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/runqueue.py | 226 ++++++++++++++++++++++-----------------------
1 file changed, 109 insertions(+), 117 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index e7ba610723..3eaaf51b35 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -102,8 +102,6 @@ class RunQueueStats:
# runQueue state machine
runQueuePrepare = 2
runQueueSceneInit = 3
-runQueueSceneRun = 4
-runQueueRunInit = 5
runQueueRunning = 6
runQueueFailed = 7
runQueueCleanUp = 8
@@ -1202,7 +1200,7 @@ class RunQueue:
# Invoked at regular time intervals via the bitbake heartbeat event
# while the build is running. We generate a unique name for the handler
# here, just in case that there ever is more than one RunQueue instance,
- # start the handler when reaching runQueueSceneRun, and stop it when
+ # start the handler when reaching runQueueSceneInit, and stop it when
# done with the build.
self.dm = monitordisk.diskMonitor(cfgData)
self.dm_event_handler_name = '_bb_diskmonitor_' + str(id(self))
@@ -1432,7 +1430,7 @@ class RunQueue:
if not self.dm_event_handler_registered:
res = bb.event.register(self.dm_event_handler_name,
- lambda x: self.dm.check(self) if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp] else False,
+ lambda x: self.dm.check(self) if self.state in [runQueueRunning, runQueueCleanUp] else False,
('bb.event.HeartbeatEvent',))
self.dm_event_handler_registered = True
@@ -1464,8 +1462,6 @@ class RunQueue:
self.state = runQueueRunning
if self.state is runQueueRunning:
- retval = self.rqexe.sq_execute()
- # FIXME revtal
retval = self.rqexe.execute()
if self.state is runQueueCleanUp:
@@ -1898,19 +1894,118 @@ class RunQueueExecute:
def execute(self):
"""
- Run the tasks in a queue prepared by rqdata.prepare()
+ Run the tasks in a queue prepared by prepare_runqueue
"""
- if self.cooker.configuration.setsceneonly:
- return True
-
self.rq.read_workers()
- if self.stats.total == 0:
- # nothing to do
- self.rq.state = runQueueCleanUp
+ task = None
+ if not self.sqdone and self.can_start_task():
+ # Find the next setscene to run
+ for nexttask in self.rqdata.runq_setscene_tids:
+ if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
+ if nexttask in self.sqdata.unskippable:
+ logger.debug(2, "Setscene task %s is unskippable" % nexttask)
+ if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]):
+ fn = fn_from_tid(nexttask)
+ foundtarget = False
- task = self.sched.next()
+ if nexttask in self.rqdata.target_tids:
+ foundtarget = True
+ if not foundtarget:
+ logger.debug(2, "Skipping setscene for task %s" % nexttask)
+ self.sq_task_skip(nexttask)
+ self.scenequeue_notneeded.add(nexttask)
+ return True
+ if nexttask in self.sqdata.outrightfail:
+ self.sq_task_failoutright(nexttask)
+ return True
+ task = nexttask
+ break
+ if task is not None:
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
+ taskname = taskname + "_setscene"
+ if self.rq.check_stamp_task(task, taskname_from_tid(task), recurse = True, cache=self.stampcache):
+ logger.debug(2, 'Stamp for underlying task %s is current, so skipping setscene variant', task)
+ self.sq_task_failoutright(task)
+ return True
+
+ if self.cooker.configuration.force:
+ if task in self.rqdata.target_tids:
+ self.sq_task_failoutright(task)
+ return True
+
+ if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
+ logger.debug(2, 'Setscene stamp current task %s, so skip it and its dependencies', task)
+ self.sq_task_skip(task)
+ return True
+
+ if self.cooker.configuration.skipsetscene:
+ logger.debug(2, 'No setscene tasks should be executed. Skipping %s', task)
+ self.sq_task_failoutright(task)
+ return True
+
+ startevent = sceneQueueTaskStarted(task, self.sq_stats, self.rq)
+ bb.event.fire(startevent, self.cfgData)
+
+ taskdepdata = self.sq_build_taskdepdata(task)
+
+ taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
+ taskhash = self.rqdata.get_task_hash(task)
+ unihash = self.rqdata.get_task_unihash(task)
+ if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
+ if not mc in self.rq.fakeworker:
+ self.rq.start_fakeworker(self, mc)
+ self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
+ self.rq.fakeworker[mc].process.stdin.flush()
+ else:
+ self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
+ self.rq.worker[mc].process.stdin.flush()
+
+ self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True)
+ self.build_stamps2.append(self.build_stamps[task])
+ self.sq_running.add(task)
+ self.sq_live.add(task)
+ self.sq_stats.taskActive()
+ if self.can_start_task():
+ return True
+
+ if not self.sq_live and not self.sqdone:
+ logger.info("Setscene tasks completed")
+ logger.debug(1, 'We could skip tasks %s', "\n".join(sorted(self.scenequeue_covered)))
+
+ completeevent = sceneQueueComplete(self.sq_stats, self.rq)
+ bb.event.fire(completeevent, self.cfgData)
+
+ err = False
+ for x in self.rqdata.runtaskentries:
+ if x not in self.tasks_covered and x not in self.tasks_notcovered:
+ logger.error("Task %s was never moved from the setscene queue" % x)
+ err = True
+ if x not in self.tasks_scenequeue_done:
+ logger.error("Task %s was never processed by the setscene code" % x)
+ err = True
+ if len(self.rqdata.runtaskentries[x].depends) == 0 and x not in self.runq_buildable:
+ logger.error("Task %s was never marked as buildable by the setscene code" % x)
+ err = True
+ if err:
+ self.rq.state = runQueueFailed
+ return True
+
+ if self.cooker.configuration.setsceneonly:
+ self.rq.state = runQueueComplete
+ return True
+ self.sqdone = True
+
+ if self.stats.total == 0:
+ # nothing to do
+ self.rq.state = runQueueComplete
+ return True
+
+ if self.cooker.configuration.setsceneonly:
+ task = None
+ else:
+ task = self.sched.next()
if task is not None:
(mc, fn, taskname, taskfn) = split_tid_mcfn(task)
@@ -2172,109 +2267,6 @@ class RunQueueExecute:
self.sq_stats.taskSkipped()
self.sq_stats.taskCompleted()
- def sq_execute(self):
- """
- Run the tasks in a queue prepared by prepare_runqueue
- """
-
- if self.sqdone:
- return True
-
- self.rq.read_workers()
-
- task = None
- if self.can_start_task():
- # Find the next setscene to run
- for nexttask in self.rqdata.runq_setscene_tids:
- if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
- if nexttask in self.sqdata.unskippable:
- logger.debug(2, "Setscene task %s is unskippable" % nexttask)
- if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]):
- fn = fn_from_tid(nexttask)
- foundtarget = False
-
- if nexttask in self.rqdata.target_tids:
- foundtarget = True
- if not foundtarget:
- logger.debug(2, "Skipping setscene for task %s" % nexttask)
- self.sq_task_skip(nexttask)
- self.scenequeue_notneeded.add(nexttask)
- return True
- if nexttask in self.sqdata.outrightfail:
- self.sq_task_failoutright(nexttask)
- return True
- task = nexttask
- break
- if task is not None:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
- taskname = taskname + "_setscene"
- if self.rq.check_stamp_task(task, taskname_from_tid(task), recurse = True, cache=self.stampcache):
- logger.debug(2, 'Stamp for underlying task %s is current, so skipping setscene variant', task)
- self.sq_task_failoutright(task)
- return True
-
- if self.cooker.configuration.force:
- if task in self.rqdata.target_tids:
- self.sq_task_failoutright(task)
- return True
-
- if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
- logger.debug(2, 'Setscene stamp current task %s, so skip it and its dependencies', task)
- self.sq_task_skip(task)
- return True
-
- if self.cooker.configuration.skipsetscene:
- logger.debug(2, 'No setscene tasks should be executed. Skipping %s', task)
- self.sq_task_failoutright(task)
- return True
-
- startevent = sceneQueueTaskStarted(task, self.sq_stats, self.rq)
- bb.event.fire(startevent, self.cfgData)
-
- taskdepdata = self.sq_build_taskdepdata(task)
-
- taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
- taskhash = self.rqdata.get_task_hash(task)
- unihash = self.rqdata.get_task_unihash(task)
- if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
- if not mc in self.rq.fakeworker:
- self.rq.start_fakeworker(self, mc)
- self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
- self.rq.fakeworker[mc].process.stdin.flush()
- else:
- self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
- self.rq.worker[mc].process.stdin.flush()
-
- self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True)
- self.build_stamps2.append(self.build_stamps[task])
- self.sq_running.add(task)
- self.sq_live.add(task)
- self.sq_stats.taskActive()
- if self.can_start_task():
- return True
-
- if self.stats.active > 0 or self.sq_stats.active > 0:
- self.rq.read_workers()
- return self.rq.active_fds()
-
- #for tid in self.sqdata.sq_revdeps:
- # if tid not in self.sq_running:
- # buildable = tid in self.sq_buildable
- # revdeps = self.sqdata.sq_revdeps[tid]
- # bb.warn("Found we didn't run %s %s %s" % (tid, buildable, str(revdeps)))
-
- logger.debug(1, 'We can skip tasks %s', "\n".join(sorted(self.scenequeue_covered)))
-
- completeevent = sceneQueueComplete(self.sq_stats, self.rq)
- bb.event.fire(completeevent, self.cfgData)
-
- if self.cooker.configuration.setsceneonly:
- self.rq.state = runQueueComplete
-
- self.sqdone = True
-
- return True
-
def sq_build_taskdepdata(self, task):
def getsetscenedeps(tid):
deps = set()
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 25/26] siggen: Fix default handler
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (22 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 24/26] runqueue: Complete the merge of scenequeue and normal task execution Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-10 23:54 ` [PATCH 26/26] tests: Add initial scenario based test for runqueue Richard Purdie
2019-07-11 0:05 ` [PATCH 00/26] runqueue changes Richard Purdie
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
After the unihash changes the default signature handler didn't work. Tweak it
to adapt to those changes (allowing the runqueue tests to work).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/siggen.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index fe580e487f..3b017219ed 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -49,7 +49,8 @@ class SignatureGenerator(object):
return self.taskhash[task]
def get_taskhash(self, fn, task, deps, dataCache):
- return "0"
+ self.taskhash[fn + "." + task] = "0"
+ return self.taskhash[fn + "." + task]
def writeout_file_checksum_cache(self):
"""Write/update the file checksum cache onto disk"""
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 26/26] tests: Add initial scenario based test for runqueue
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (23 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 25/26] siggen: Fix default handler Richard Purdie
@ 2019-07-10 23:54 ` Richard Purdie
2019-07-11 0:05 ` [PATCH 00/26] runqueue changes Richard Purdie
25 siblings, 0 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-10 23:54 UTC (permalink / raw)
To: bitbake-devel
We need some tests for runqueue, its been something which has been hard to test
for a long time. Add some dummy metadata to allow this, mirroring the OE
structure in spirit.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
bin/bitbake-selftest | 1 +
.../tests/runqueue-tests/classes/base.bbclass | 219 ++++++++++++++++++
.../runqueue-tests/classes/image.bbclass | 5 +
.../runqueue-tests/classes/native.bbclass | 2 +
lib/bb/tests/runqueue-tests/conf/bitbake.conf | 10 +
lib/bb/tests/runqueue-tests/recipes/a1.bb | 0
lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 +
lib/bb/tests/runqueue-tests/recipes/c1.bb | 0
lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 +
lib/bb/tests/runqueue.py | 191 +++++++++++++++
10 files changed, 432 insertions(+)
create mode 100644 lib/bb/tests/runqueue-tests/classes/base.bbclass
create mode 100644 lib/bb/tests/runqueue-tests/classes/image.bbclass
create mode 100644 lib/bb/tests/runqueue-tests/classes/native.bbclass
create mode 100644 lib/bb/tests/runqueue-tests/conf/bitbake.conf
create mode 100644 lib/bb/tests/runqueue-tests/recipes/a1.bb
create mode 100644 lib/bb/tests/runqueue-tests/recipes/b1.bb
create mode 100644 lib/bb/tests/runqueue-tests/recipes/c1.bb
create mode 100644 lib/bb/tests/runqueue-tests/recipes/d1.bb
create mode 100644 lib/bb/tests/runqueue.py
diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
index 20553e9814..041a2719f0 100755
--- a/bin/bitbake-selftest
+++ b/bin/bitbake-selftest
@@ -25,6 +25,7 @@ tests = ["bb.tests.codeparser",
"bb.tests.fetch",
"bb.tests.parse",
"bb.tests.persist_data",
+ "bb.tests.runqueue",
"bb.tests.utils",
"hashserv.tests",
"layerindexlib.tests.layerindexobj",
diff --git a/lib/bb/tests/runqueue-tests/classes/base.bbclass b/lib/bb/tests/runqueue-tests/classes/base.bbclass
new file mode 100644
index 0000000000..e81df7ac42
--- /dev/null
+++ b/lib/bb/tests/runqueue-tests/classes/base.bbclass
@@ -0,0 +1,219 @@
+SLOWTASKS ??= ""
+SSTATEVALID ??= ""
+
+def stamptask(d):
+ import time
+
+ thistask = d.expand("${PN}:${BB_CURRENTTASK}")
+ if thistask in d.getVar("SLOWTASKS").split():
+ bb.note("Slowing task %s" % thistask)
+ time.sleep(0.5)
+
+ with open(d.expand("${TOPDIR}/task.log"), "a+") as f:
+ f.write(thistask + "\n")
+
+python do_fetch() {
+ stamptask(d)
+}
+python do_unpack() {
+ stamptask(d)
+}
+python do_patch() {
+ stamptask(d)
+}
+python do_populate_lic() {
+ stamptask(d)
+}
+python do_prepare_recipe_sysroot() {
+ stamptask(d)
+}
+python do_configure() {
+ stamptask(d)
+}
+python do_compile() {
+ stamptask(d)
+}
+python do_install() {
+ stamptask(d)
+}
+python do_populate_sysroot() {
+ stamptask(d)
+}
+python do_package() {
+ stamptask(d)
+}
+python do_package_write_ipk() {
+ stamptask(d)
+}
+python do_package_write_rpm() {
+ stamptask(d)
+}
+python do_packagedata() {
+ stamptask(d)
+}
+python do_package_qa() {
+ stamptask(d)
+}
+python do_build() {
+ stamptask(d)
+}
+do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot"
+do_package[deptask] += "do_packagedata"
+do_build[recrdeptask] += "do_deploy"
+do_build[recrdeptask] += "do_package_write_ipk"
+do_build[recrdeptask] += "do_package_write_rpm"
+do_package_qa[rdeptask] = "do_packagedata"
+do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
+
+DEBIANRDEP = "do_packagedata"
+oo_package_write_ipk[rdeptask] = "${DEBIANRDEP}"
+do_package_write_rpm[rdeptask] = "${DEBIANRDEP}"
+
+addtask fetch
+addtask unpack after do_fetch
+addtask patch after do_unpack
+addtask prepare_recipe_sysroot after do_patch
+addtask configure after do_prepare_recipe_sysroot
+addtask compile after do_configure
+addtask install after do_compile
+addtask populate_sysroot after do_install
+addtask package after do_install
+addtask package_write_ipk after do_packagedata package
+addtask package_write_rpm after do_packagedata package
+addtask packagedata after do_package
+addtask package_qa after do_package
+addtask build after do_package_qa do_package_write_rpm do_package_write_ipk do_populate_sysroot
+
+python do_package_setscene() {
+ stamptask(d)
+}
+python do_package_qa_setscene() {
+ stamptask(d)
+}
+python do_package_write_ipk_setscene() {
+ stamptask(d)
+}
+python do_package_write_rpm_setscene() {
+ stamptask(d)
+}
+python do_packagedata_setscene() {
+ stamptask(d)
+}
+python do_populate_lic_setscene() {
+ stamptask(d)
+}
+python do_populate_sysroot_setscene() {
+ stamptask(d)
+}
+
+addtask package_setscene
+addtask package_qa_setscene
+addtask package_write_ipk_setscene
+addtask package_write_rpm_setscene
+addtask packagedata_setscene
+addtask populate_lic_setscene
+addtask populate_sysroot_setscene
+
+BB_SETSCENE_DEPVALID = "setscene_depvalid"
+
+def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
+ # taskdependees is a dict of tasks which depend on task, each being a 3 item list of [PN, TASKNAME, FILENAME]
+ # task is included in taskdependees too
+ # Return - False - We need this dependency
+ # - True - We can skip this dependency
+ import re
+
+ def logit(msg, log):
+ if log is not None:
+ log.append(msg)
+ else:
+ bb.debug(2, msg)
+
+ logit("Considering setscene task: %s" % (str(taskdependees[task])), log)
+
+ def isNativeCross(x):
+ return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
+
+ # We only need to trigger populate_lic through direct dependencies
+ if taskdependees[task][1] == "do_populate_lic":
+ return True
+
+ # We only need to trigger packagedata through direct dependencies
+ # but need to preserve packagedata on packagedata links
+ if taskdependees[task][1] == "do_packagedata":
+ for dep in taskdependees:
+ if taskdependees[dep][1] == "do_packagedata":
+ return False
+ return True
+
+ for dep in taskdependees:
+ logit(" considering dependency: %s" % (str(taskdependees[dep])), log)
+ if task == dep:
+ continue
+ if dep in notneeded:
+ continue
+ # do_package_write_* and do_package doesn't need do_package
+ if taskdependees[task][1] == "do_package" and taskdependees[dep][1] in ['do_package', 'do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
+ continue
+ # do_package_write_* need do_populate_sysroot as they're mainly postinstall dependencies
+ if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package_write_ipk', 'do_package_write_rpm']:
+ return False
+ # do_package/packagedata/package_qa don't need do_populate_sysroot
+ if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_packagedata', 'do_package_qa']:
+ continue
+ # Native/Cross packages don't exist and are noexec anyway
+ if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1] in ['do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package', 'do_package_qa']:
+ continue
+
+ # This is due to the [depends] in useradd.bbclass complicating matters
+ # The logic *is* reversed here due to the way hard setscene dependencies are injected
+ if (taskdependees[task][1] == 'do_package' or taskdependees[task][1] == 'do_populate_sysroot') and taskdependees[dep][0].endswith(('shadow-native', 'shadow-sysroot', 'base-passwd', 'pseudo-native')) and taskdependees[dep][1] == 'do_populate_sysroot':
+ continue
+
+ # Consider sysroot depending on sysroot tasks
+ if taskdependees[task][1] == 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot':
+ # Native/Cross populate_sysroot need their dependencies
+ if isNativeCross(taskdependees[task][0]) and isNativeCross(taskdependees[dep][0]):
+ return False
+ # Target populate_sysroot depended on by cross tools need to be installed
+ if isNativeCross(taskdependees[dep][0]):
+ return False
+ # Native/cross tools depended upon by target sysroot are not needed
+ # Add an exception for shadow-native as required by useradd.bbclass
+ if isNativeCross(taskdependees[task][0]) and taskdependees[task][0] != 'shadow-native':
+ continue
+ # Target populate_sysroot need their dependencies
+ return False
+
+
+ if taskdependees[dep][1] == "do_populate_lic":
+ continue
+
+ # Safe fallthrough default
+ logit(" Default setscene dependency fall through due to dependency: %s" % (str(taskdependees[dep])), log)
+ return False
+ return True
+
+BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
+
+def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=False, *, sq_unihash=None):
+
+ ret = []
+ missed = []
+
+ valid = d.getVar("SSTATEVALID").split()
+
+ for task in range(len(sq_fn)):
+ n = os.path.basename(sq_fn[task]).rsplit(".", 1)[0] + ":" + sq_task[task]
+ if n in valid:
+ bb.note("SState: Found valid sstate for %s" % n)
+ ret.append(task)
+ else:
+ missed.append(task)
+ bb.note("SState: Found no valid sstate for %s" % n)
+
+ if hasattr(bb.parse.siggen, "checkhashes"):
+ bb.parse.siggen.checkhashes(missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d)
+
+ return ret
+
diff --git a/lib/bb/tests/runqueue-tests/classes/image.bbclass b/lib/bb/tests/runqueue-tests/classes/image.bbclass
new file mode 100644
index 0000000000..da9ff11064
--- /dev/null
+++ b/lib/bb/tests/runqueue-tests/classes/image.bbclass
@@ -0,0 +1,5 @@
+do_rootfs[recrdeptask] += "do_package_write_deb do_package_qa"
+do_rootfs[recrdeptask] += "do_package_write_ipk do_package_qa"
+do_rootfs[recrdeptask] += "do_package_write_rpm do_package_qa
+do_rootfs[recrdeptask] += "do_packagedata"
+do_rootfs[recrdeptask] += "do_populate_lic"
diff --git a/lib/bb/tests/runqueue-tests/classes/native.bbclass b/lib/bb/tests/runqueue-tests/classes/native.bbclass
new file mode 100644
index 0000000000..7eaaee54ad
--- /dev/null
+++ b/lib/bb/tests/runqueue-tests/classes/native.bbclass
@@ -0,0 +1,2 @@
+RECIPERDEPTASK = "do_populate_sysroot"
+do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}"
diff --git a/lib/bb/tests/runqueue-tests/conf/bitbake.conf b/lib/bb/tests/runqueue-tests/conf/bitbake.conf
new file mode 100644
index 0000000000..cccd677966
--- /dev/null
+++ b/lib/bb/tests/runqueue-tests/conf/bitbake.conf
@@ -0,0 +1,10 @@
+CACHE = "${TOPDIR}/cache"
+THISDIR = "${@os.path.dirname(d.getVar('FILE'))}"
+COREBASE := "${@os.path.normpath(os.path.dirname(d.getVar('FILE')+'/../../'))}"
+BBFILES = "${COREBASE}/recipes/*.bb"
+PROVIDES = "${PN}"
+PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0]}"
+export PATH
+STAMP = "${TOPDIR}/stamps/${PN}"
+T = "${TOPDIR}/workdir/${PN}/temp"
+BB_NUMBER_THREADS = "4"
diff --git a/lib/bb/tests/runqueue-tests/recipes/a1.bb b/lib/bb/tests/runqueue-tests/recipes/a1.bb
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/lib/bb/tests/runqueue-tests/recipes/b1.bb b/lib/bb/tests/runqueue-tests/recipes/b1.bb
new file mode 100644
index 0000000000..c0b288e5bc
--- /dev/null
+++ b/lib/bb/tests/runqueue-tests/recipes/b1.bb
@@ -0,0 +1 @@
+DEPENDS = "a1"
\ No newline at end of file
diff --git a/lib/bb/tests/runqueue-tests/recipes/c1.bb b/lib/bb/tests/runqueue-tests/recipes/c1.bb
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/lib/bb/tests/runqueue-tests/recipes/d1.bb b/lib/bb/tests/runqueue-tests/recipes/d1.bb
new file mode 100644
index 0000000000..5ba197515b
--- /dev/null
+++ b/lib/bb/tests/runqueue-tests/recipes/d1.bb
@@ -0,0 +1,3 @@
+DEPENDS = "a1"
+
+do_package_setscene[depends] = "a1:do_populate_sysroot_setscene"
diff --git a/lib/bb/tests/runqueue.py b/lib/bb/tests/runqueue.py
new file mode 100644
index 0000000000..b1a23bc5d4
--- /dev/null
+++ b/lib/bb/tests/runqueue.py
@@ -0,0 +1,191 @@
+#
+# BitBake Tests for runqueue task processing
+#
+# Copyright (C) 2019 Richard Purdie
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import unittest
+import bb
+import os
+import tempfile
+import subprocess
+
+#
+# TODO:
+# Add tests on task ordering (X happens before Y after Z)
+#
+
+class RunQueueTests(unittest.TestCase):
+
+ alltasks = ['package', 'fetch', 'unpack', 'patch', 'prepare_recipe_sysroot', 'configure',
+ 'compile', 'install', 'packagedata', 'package_qa', 'package_write_rpm', 'package_write_ipk',
+ 'populate_sysroot', 'build']
+ a1_sstatevalid = "a1:do_package a1:do_package_qa a1:do_packagedata a1:do_package_write_ipk a1:do_package_write_rpm a1:do_populate_lic a1:do_populate_sysroot"
+ b1_sstatevalid = "b1:do_package b1:do_package_qa b1:do_packagedata b1:do_package_write_ipk b1:do_package_write_rpm b1:do_populate_lic b1:do_populate_sysroot"
+
+ def run_bitbakecmd(self, cmd, builddir, sstatevalid="", slowtasks="", extraenv=None):
+ env = os.environ.copy()
+ env["BBPATH"] = os.path.realpath(os.path.join(os.path.dirname(__file__), "runqueue-tests"))
+ env["BB_ENV_EXTRAWHITE"] = "SSTATEVALID SLOWTASKS"
+ env["SSTATEVALID"] = sstatevalid
+ env["SLOWTASKS"] = slowtasks
+ if extraenv:
+ for k in extraenv:
+ env[k] = extraenv[k]
+ env["BB_ENV_EXTRAWHITE"] = env["BB_ENV_EXTRAWHITE"] + " " + k
+ try:
+ output = subprocess.check_output(cmd, env=env, stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
+ except subprocess.CalledProcessError as e:
+ self.fail("Command %s failed with %s" % (cmd, e.output))
+ tasks = []
+ with open(builddir + "/task.log", "r") as f:
+ tasks = [line.rstrip() for line in f]
+ return tasks
+
+ def test_no_setscenevalid(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:' + x for x in self.alltasks]
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_single_setscenevalid(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = "a1:do_package"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_setscene', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot', 'a1:build']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_intermediate_setscenevalid(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = "a1:do_package a1:do_populate_sysroot"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_setscene', 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot_setscene', 'a1:build']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_all_setscenevalid(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = self.a1_sstatevalid
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
+ 'a1:package_qa_setscene', 'a1:build', 'a1:populate_sysroot_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_no_settasks(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1", "-c", "patch"]
+ sstatevalid = self.a1_sstatevalid
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:fetch', 'a1:unpack', 'a1:patch']
+ self.assertEqual(set(tasks), set(expected))
+
+ # Test targets with intermediate setscene tasks alongside a target with no intermediate setscene tasks
+ def test_mixed_direct_tasks_setscene_tasks(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "c1:do_patch", "a1"]
+ sstatevalid = self.a1_sstatevalid
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['c1:fetch', 'c1:unpack', 'c1:patch', 'a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
+ 'a1:package_qa_setscene', 'a1:build', 'a1:populate_sysroot_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ # This test slows down the execution of do_package_setscene until after other real tasks have
+ # started running which tests for a bug where tasks were being lost from the buildable list of real
+ # tasks if they weren't in tasks_covered or tasks_notcovered
+ def test_slow_setscene(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = "a1:do_package"
+ slowtasks = "a1:package_setscene"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, slowtasks)
+ expected = ['a1:package_setscene', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot', 'a1:build']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_setscenewhitelist(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ extraenv = {
+ "BB_SETSCENE_ENFORCE" : "1",
+ "BB_SETSCENE_ENFORCE_WHITELIST" : "a1:do_package_write_rpm a1:do_build"
+ }
+ sstatevalid = "a1:do_package a1:do_package_qa a1:do_packagedata a1:do_package_write_ipk a1:do_populate_lic a1:do_populate_sysroot"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv)
+ expected = ['a1:packagedata_setscene', 'a1:package_qa_setscene', 'a1:package_write_ipk_setscene',
+ 'a1:populate_sysroot_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ # Tests for problems with dependencies between setscene tasks
+ def test_no_setscenevalid_harddeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "d1"]
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot', 'd1:package', 'd1:fetch', 'd1:unpack', 'd1:patch', 'd1:prepare_recipe_sysroot', 'd1:configure',
+ 'd1:compile', 'd1:install', 'd1:packagedata', 'd1:package_qa', 'd1:package_write_rpm', 'd1:package_write_ipk',
+ 'd1:populate_sysroot', 'd1:build']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_no_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:' + x for x in self.alltasks] + ['b1:' + x for x in self.alltasks]
+ expected.remove('a1:build')
+ expected.remove('a1:package_qa')
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_single_a1_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = "a1:do_package"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_setscene', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot'] + ['b1:' + x for x in self.alltasks]
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_single_b1_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = "b1:do_package"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot', 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
+ expected.remove('b1:package')
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_intermediate_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = "a1:do_package a1:do_populate_sysroot b1:do_package"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_setscene', 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot_setscene', 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
+ expected.remove('b1:package')
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_all_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = self.a1_sstatevalid + " " + self.b1_sstatevalid
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
+ 'b1:build', 'a1:populate_sysroot_setscene', 'b1:package_write_ipk_setscene', 'b1:package_write_rpm_setscene',
+ 'b1:packagedata_setscene', 'b1:package_qa_setscene', 'b1:populate_sysroot_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
--
2.20.1
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 00/26] runqueue changes
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
` (24 preceding siblings ...)
2019-07-10 23:54 ` [PATCH 26/26] tests: Add initial scenario based test for runqueue Richard Purdie
@ 2019-07-11 0:05 ` Richard Purdie
[not found] ` <63270c2a-f78a-65e1-16c0-8ce271d4dd8e@mikrodidakt.se>
2019-07-11 16:21 ` Richard Purdie
25 siblings, 2 replies; 31+ messages in thread
From: Richard Purdie @ 2019-07-11 0:05 UTC (permalink / raw)
To: bitbake-devel
This patchset reworks runqueue so that instead of having two distinct
phases, specifically "setscene", followed by "normal tasks" we have one
single phase.
In this single phase, both sets of tasks can run in parallel using the
same workers. Thankfully the code already used common classes and
functions so making this transition was relatively straight forward.
Initially all tasks are in the scenequeue, as it works through them,
they are marked as "buildable" which means normal tasks can execute.
The big change here are in the code which incrementally marks tasks as
available rather than being able to process tasks "en mass".
The most common failures from these changes are potentially:
a) tasks not migrating from the scenequeue
b) tasks not becomming buildable
I've made some tests on the autobuilder and believe I've tested many of
the common usecases. There is debug code to abort the builds with the
above errors when something unexpected happens.
I've also written an initial set of runqueue unittests. To do that I
had to create a dummy set of metadata so we had some dependency trees
to test with.
This patchset did remove an API we no longer need/use after RSS was
introduced and gets rid of what seems like a pointless form of event.
These changes set up the foundations so that we can work on two other
significant problems:
a) reuse of sstate artefacts between multiconfig build tasks
b) adapting the task queue 'on the fly' if the output of a task matches
a previous output and hence we have sstate hash equivalency
Solving these would be great for OE!
I will follow up with more details on the future plans for the above
and this code soon.
Cheers,
Richard
^ permalink raw reply [flat|nested] 31+ messages in thread[parent not found: <63270c2a-f78a-65e1-16c0-8ce271d4dd8e@mikrodidakt.se>]
* Re: [PATCH 00/26] runqueue changes
[not found] ` <63270c2a-f78a-65e1-16c0-8ce271d4dd8e@mikrodidakt.se>
@ 2019-07-11 11:33 ` richard.purdie
0 siblings, 0 replies; 31+ messages in thread
From: richard.purdie @ 2019-07-11 11:33 UTC (permalink / raw)
To: Jacob Kroon, bitbake-devel
On Thu, 2019-07-11 at 10:57 +0000, Jacob Kroon wrote:
> Exciting to see steps towards the sstate hash equivalence thing!
>
> I tested master-next, 6ff6485fa9195055ab6057527c6388f6c013cb8d, and
> got:
>
> Sstate summary: Wanted 1152 Found 579 Missed 573 Current 0 (50%
> match,
> 0% complete)
> NOTE: Executing Tasks
> Traceback (most recent call last):
> File "<snip>/bitbake/lib/bb/ui/knotty.py", line 480, in main
> termfilter.updateFooter()
> File "<snip>/bitbake/lib/bb/ui/knotty.py", line 291, in
> updateFooter
> content = self.main_progress.update(progress)
> File "<snip>/bitbake/lib/progressbar/progressbar.py", line 256, in
> update
> raise ValueError('Value out of range')
> ValueError: Value out of range
>
> Restarted the build, and now its running...
Thanks, I think this is the UI not understanding both sets of tasks run
in parallel. There is a patch added to -next and on the list which
hopefully should stop the above happening (I couldn't reproduce it
though).
Cheers,
Richard
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 00/26] runqueue changes
2019-07-11 0:05 ` [PATCH 00/26] runqueue changes Richard Purdie
[not found] ` <63270c2a-f78a-65e1-16c0-8ce271d4dd8e@mikrodidakt.se>
@ 2019-07-11 16:21 ` Richard Purdie
2019-07-12 0:57 ` Khem Raj
1 sibling, 1 reply; 31+ messages in thread
From: Richard Purdie @ 2019-07-11 16:21 UTC (permalink / raw)
To: bitbake-devel
On Thu, 2019-07-11 at 01:05 +0100, Richard Purdie wrote:
> These changes set up the foundations so that we can work on two other
> significant problems:
>
> a) reuse of sstate artefacts between multiconfig build tasks
I have spent a long time thinking about this so now the codebase was
ready it turned out not to be too hard to make it work. Patches aren't
tested apart from with the unittest but I've posted them and we can
take them from here.
> b) adapting the task queue 'on the fly' if the output of a task
> matches a previous output and hence we have sstate hash equivalency
>
> Solving these would be great for OE!
>
> I will follow up with more details on the future plans for the above
> and this code soon.
Adapting to new hashes "on the fly" is definitely going to be the
harder challenge.
This will involve the following steps:
a) Knowing there are new unitask hashes available and propagating that
information
b) Figuring out which tasks should now have their sstate retested
c) Test for sstate existence (if it doesn't exist, no point in going
further)
d) Ensure any normal tasks covered by the sstate task are not running,
wait if any are
e) Remove all those normal tasks from access by the normal task running
code
f) Enable those tasks in the scenequeue code
We have no existing code which moves things "backward" in the state
engine from normal task to scenequeue so this will be new ground, but
should be feasible.
I'm going to wait for the current patches to settle before I look into
this any further though.
Cheers,
Richard
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH 00/26] runqueue changes
2019-07-11 16:21 ` Richard Purdie
@ 2019-07-12 0:57 ` Khem Raj
2019-07-12 8:13 ` richard.purdie
0 siblings, 1 reply; 31+ messages in thread
From: Khem Raj @ 2019-07-12 0:57 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
Hi Richard
Good changes I believe. Can you push them to a temp branch on bitbake repo ?
On Thu, Jul 11, 2019 at 9:21 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2019-07-11 at 01:05 +0100, Richard Purdie wrote:
> > These changes set up the foundations so that we can work on two other
> > significant problems:
> >
> > a) reuse of sstate artefacts between multiconfig build tasks
>
> I have spent a long time thinking about this so now the codebase was
> ready it turned out not to be too hard to make it work. Patches aren't
> tested apart from with the unittest but I've posted them and we can
> take them from here.
>
> > b) adapting the task queue 'on the fly' if the output of a task
> > matches a previous output and hence we have sstate hash equivalency
> >
> > Solving these would be great for OE!
> >
> > I will follow up with more details on the future plans for the above
> > and this code soon.
>
> Adapting to new hashes "on the fly" is definitely going to be the
> harder challenge.
>
> This will involve the following steps:
>
> a) Knowing there are new unitask hashes available and propagating that
> information
>
> b) Figuring out which tasks should now have their sstate retested
>
> c) Test for sstate existence (if it doesn't exist, no point in going
> further)
>
> d) Ensure any normal tasks covered by the sstate task are not running,
> wait if any are
>
> e) Remove all those normal tasks from access by the normal task running
> code
>
> f) Enable those tasks in the scenequeue code
>
> We have no existing code which moves things "backward" in the state
> engine from normal task to scenequeue so this will be new ground, but
> should be feasible.
>
> I'm going to wait for the current patches to settle before I look into
> this any further though.
>
> Cheers,
>
> Richard
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
^ permalink raw reply [flat|nested] 31+ messages in thread