[PATCH] sstate: Remove stale objects before the main build

* [PATCH] sstate: Remove stale objects before the main build
@ 2021-03-19 17:26 Richard Purdie
  2021-03-19 17:39 ` [OE-core] " Khem Raj
  0 siblings, 1 reply; 3+ messages in thread
From: Richard Purdie @ 2021-03-19 17:26 UTC (permalink / raw)
  To: openembedded-core

The split of util-linux-uuid out from util-linux caused some interesting
sstate file overlap errors on existing build directories. This is a
challenge to handle since util-linux depends on util-linux-uuid and has
overlapping files in package data and deploy/packages directories.
The util-linux build happens later and is what would clean up those files
but it happens too late for uuid.

Fixing this is hard as we don't know the taskhashes until the task
graph is calculated. Once that is ready, we can compare the hashes
with the existing hashes and know which sstate tasks are "stale".

This patch adds a handler which iterates the sstate manifests looking
for matching stamp paths and then removes the manifests along with the
associated stamp files.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/sstate.bbclass | 45 +++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index f5791681625..673292c450d 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -1223,3 +1223,48 @@ python sstate_eventhandler2() {
     if preservestamps:
         os.remove(preservestampfile)
 }
+
+addhandler sstate_eventhandler_stalesstate
+sstate_eventhandler_stalesstate[eventmask] = "bb.event.StaleSetSceneTasks"
+python sstate_eventhandler_stalesstate() {
+    d = e.data
+    tasks = e.tasks
+
+    bb.utils.mkdirhier(d.expand("${SSTATE_MANIFESTS}"))
+
+    for a in sorted(list(set(d.getVar("SSTATE_ARCHS").split()))):
+        toremove = []
+        i = d.expand("${SSTATE_MANIFESTS}/index-" + a)
+        if not os.path.exists(i):
+            continue
+        with open(i, "r") as f:
+            lines = f.readlines()
+            for l in lines:
+                try:
+                    (stamp, manifest, workdir) = l.split()
+                    for tid in tasks:
+                        for s in tasks[tid]:
+                            if s.startswith(stamp):
+                                taskname = bb.runqueue.taskname_from_tid(tid)[3:]
+                                manname = manifest + "." + taskname
+                                if os.path.exists(manname):
+                                    bb.debug(2, "Sstate for %s is stale, removing related manifest %s" % (tid, manname))
+                                    toremove.append((manname, tid, tasks[tid]))
+                                    break
+                except ValueError:
+                    bb.fatal("Invalid line '%s' in sstate manifest '%s'" % (l, i))
+
+        if toremove:
+            msg = "Removing %d stale sstate objects for arch %s" % (len(toremove), a)
+            bb.event.fire(bb.event.ProcessStarted(msg, len(toremove)), d)
+
+            removed = 0
+            for (manname, tid, stamps) in toremove:
+                sstate_clean_manifest(manname, d)
+                for stamp in stamps:
+                    bb.utils.remove(stamp)
+                removed = removed + 1
+                bb.event.fire(bb.event.ProcessProgress(msg, removed), d)
+
+            bb.event.fire(bb.event.ProcessFinished(msg), d)
+}
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 3+ messages in thread