[OE-core][RFC][PATCH] buildstats-summary-native: add bbclass for tracking native sstate reuse

* [OE-core][RFC][PATCH] buildstats-summary-native: add bbclass for tracking native sstate reuse
@ 2020-10-21 19:30 Trevor Gamblin
  2020-10-21 21:12 ` Richard Purdie
  0 siblings, 1 reply; 3+ messages in thread
From: Trevor Gamblin @ 2020-10-21 19:30 UTC (permalink / raw)
  To: openembedded-core

buildstats-summary doesn't distinguish between native and non-native
sstate reuse, where native sstate reuse in particular may be useful to
track when sharing sstate caches across multiple target architectures.
We can tweak buildstats-summary and add a buildstats-summary-native
bbclass that is more explicit in reporting sstate reuse, as well as
making the output more readable for parsing by other scripts.

Note that (as far as my tests have gone, anyway) do_package* tasks
haven't had any native reuse at all, so that result should always
be zero unless something has gone horribly wrong.

An example, where core-image-full-cmdline has been built after the
sstate-cache has been filled by a core-image-minimal build:

NOTE: Build completion summary:
NOTE:   do_populate_sysroot: native from source 5 of 139 (96.4% sstate reuse), target from source 35 of 161 (78.3% sstate reuse), 300 total
NOTE:   do_deploy_source_date_epoch: native from source 5 of 139 (96.4% sstate reuse), target from source 36 of 162 (77.8% sstate reuse), 301 total
NOTE:   do_package_qa: native from source 0 of 0 (0% sstate reuse), target from source 37 of 159 (76.7% sstate reuse), 159 total
NOTE:   do_package: native from source 0 of 0 (0% sstate reuse), target from source 37 of 37 (0.0% sstate reuse), 37 total
NOTE:   do_packagedata: native from source 0 of 0 (0% sstate reuse), target from source 37 of 159 (76.7% sstate reuse), 159 total
NOTE:   do_package_write_rpm: native from source 0 of 0 (0% sstate reuse), target from source 37 of 159 (76.7% sstate reuse), 159 total
NOTE:   do_populate_lic: native from source 5 of 139 (96.4% sstate reuse), target from source 37 of 165 (77.6% sstate reuse), 304 total

Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
---
 .../classes/buildstats-summary-native.bbclass | 55 +++++++++++++++++++
 1 file changed, 55 insertions(+)
 create mode 100644 meta/classes/buildstats-summary-native.bbclass

diff --git a/meta/classes/buildstats-summary-native.bbclass b/meta/classes/buildstats-summary-native.bbclass
new file mode 100644
index 0000000000..f8e502ca7d
--- /dev/null
+++ b/meta/classes/buildstats-summary-native.bbclass
@@ -0,0 +1,55 @@
+# Summarize sstate usage at the end of the build
+python buildstats_summary () {
+    import collections
+    import os.path
+
+    bsdir = e.data.expand("${BUILDSTATS_BASE}/${BUILDNAME}")
+    if not os.path.exists(bsdir):
+        return
+
+    sstatetasks = (e.data.getVar('SSTATETASKS') or '').split()
+    built = collections.defaultdict(lambda: [set(), set(), set(), set()])
+    for pf in os.listdir(bsdir):
+        taskdir = os.path.join(bsdir, pf)
+        bb.note(taskdir)
+        if not os.path.isdir(taskdir):
+            continue
+
+        tasks = os.listdir(taskdir)
+        for t in sstatetasks:
+            target_source, native_source, native_sstate, target_sstate = built[t]
+            if t in tasks and "-native" in pf:
+                native_source.add(pf)
+            elif t in tasks:
+                target_source.add(pf)
+            elif t + '_setscene' in tasks and "-native" in pf:
+                native_sstate.add(pf)
+            elif t + '_setscene' in tasks:
+                target_sstate.add(pf)
+
+    header_printed = False
+    for t in sstatetasks:
+        target_source, native_source, native_sstate, target_sstate = built[t]
+        if target_source | native_source | native_sstate | target_sstate:
+            if not header_printed:
+                header_printed = True
+                bb.note("Build completion summary:")
+
+            target_sstate_count = len(target_sstate)
+            native_sstate_count = len(native_sstate)
+            target_source_count = len(target_source)
+            native_source_count = len(native_source)
+            total_source_count = target_source_count + native_source_count
+            total_sstate_count = target_sstate_count + native_sstate_count
+            total_native_count = native_source_count + native_sstate_count
+            total_target_count = target_source_count + target_sstate_count
+            total_count = total_source_count + total_sstate_count
+
+            #do_package_qa, do_packagedata, and do_package_write_rpm won't have native sstate, so declare them as 0% reuse 
+            if native_sstate_count == 0:
+                bb.note("  {0}: native from source {1} of {2} (0% sstate reuse), target from source {3} of {4} ({5:.1f}% sstate reuse), {6} total".format(t, native_source_count, total_native_count, target_source_count, total_target_count, round(100 * target_sstate_count / total_target_count, 1), total_count))
+            else:
+                bb.note("  {0}: native from source {1} of {2} ({3:.1f}% sstate reuse), target from source {4} of {5} ({6:.1f}% sstate reuse), {7} total".format(t, native_source_count, total_native_count, round(100 * native_sstate_count / total_native_count, 1), target_source_count, total_target_count, round(100 * target_sstate_count / total_target_count, 1), total_count))
+}
+addhandler buildstats_summary
+buildstats_summary[eventmask] = "bb.event.BuildCompleted"
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 3+ messages in thread