* RFC: Backwards compatibility checking sstate-cache
@ 2017-06-30 9:48 Michael Ho
2017-06-30 15:01 ` Darcy Watkins
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Michael Ho @ 2017-06-30 9:48 UTC (permalink / raw)
To: yocto
[-- Attachment #1: Type: text/plain, Size: 3506 bytes --]
Hi, at BMW Car IT we are working on an experimental feature to improve sstate
cache hits and we are looking for comments on the approach who might have some
insights to the problem and seeing if anyone is interested in the feature for
mainline.
The sstate-cache of a recipe is tied closely to its build dependencies, as the
signature generated for a task includes all parent task's signatures as part of
the signature information. This means that when changes occur in the parent
recipes, even if insignificant, all children recipes that have valid sstate
cache must invalidate their sstate cache objects.
What this patchset does is propose to add another optional variable to recipes,
CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature
generation it excludes any parent tasks that come from dependencies listed in
it. This is to break the signature change domino effect.
This patchset also proposes modifying RunQueue to then be able to run a
compatibility checker during task execution phase for recipes and tasks that use
CDEPENDS and allow for deciding to re-execute a task despite being covered by
sstate-cache.
The patchset is based on the jethro branch for the poky repository, as this is
the branch that we are using. If the general idea sounds good, we can consider
porting it to master.
Included is an patch that adds an example recipe and compatibility checker,
where compatibility is based on the file checksums of the parent recipes
packages. An example recipe, cdepends-test1, generates a compatibility report
containing the file checksums of all files that it packages and which file paths
they are at. Another recipe, cdepends-test2, can then strip this compatibility
report to the minimal files it needs to be consistent and can compare the latest
checksums it used to configure/compile/install with and if they have changed,
trigger a rebuild. If not, the previous version restored from sstate-cache is
used.
We are still experimenting with the usages of this, including the use of having
abi-compliance-checker to compare the ABI of shared libraries as a compatibility
checker during RunQueue and using the results to avoid rebuilding child recipes
when the .so files they depend on are still compatible. Example use cases of
this are allowing recipes which provide multiple shared libraries to change a
single .so file without rebuilding all its children that depend on the other
shared libraries but not the one that changed.
We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it didn't meet
the feature requirements of what this compatibility checker callback is doing,
although maybe when porting to master we could refactor to make better use of
the work already done there. The current implementation is a bit hacky but
comments would be appreciated in regards to if the concept is feasible and if
people are interested in making use of it and their use cases.
Kind regards,
Michael Ho
--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm
Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael.ho@bmw-carit.de
Web: http://www.bmw-carit.de
--------------------------------------------------------
BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--------------------------------------------------------
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-cache.py-add-support-for-CDEPENDS.patch --]
[-- Type: text/x-patch; name="0001-cache.py-add-support-for-CDEPENDS.patch", Size: 3546 bytes --]
From 0afcfc5bde251e96a434c345b4e0e4db895feeae Mon Sep 17 00:00:00 2001
From: Michael Ho <Michael.Ho@bmw.de>
Date: Fri, 30 Jun 2017 06:52:12 +0200
Subject: [PATCH 1/9] cache.py: add support for CDEPENDS
To: yocto@yoctoproject.org
Modifies the bitbake recipe cache handling to now parse also the CDEPENDS
variable, the cdepends flag that can be set for tasks, and also the machine
architecture for the recipe target (since compatibility will be tied to the
machine type also).
---
bitbake/lib/bb/cache.py | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index ab09b08..3596242 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -146,6 +146,9 @@ class CoreRecipeInfo(RecipeInfoCommon):
self.fakerootenv = self.getvar('FAKEROOTENV', metadata)
self.fakerootdirs = self.getvar('FAKEROOTDIRS', metadata)
self.fakerootnoenv = self.getvar('FAKEROOTNOENV', metadata)
+ self.cdepends = self.depvar('CDEPENDS', metadata)
+ self.task_cdepends = self.flaglist('cdepends', self.tasks, metadata, True)
+ self.target_sys = self.getvar('MULTIMACH_TARGET_SYS', metadata)
@classmethod
def init_cacheData(cls, cachedata):
@@ -172,6 +175,12 @@ class CoreRecipeInfo(RecipeInfoCommon):
cachedata.rproviders = defaultdict(list)
cachedata.packages_dynamic = defaultdict(list)
+ cachedata.all_cdepends = []
+ cachedata.compatible_deps = defaultdict(list)
+ cachedata.all_compatible_deps = defaultdict(list)
+ cachedata.compatible_task_deps = {}
+ cachedata.target_sys = {}
+
cachedata.rundeps = defaultdict(lambda: defaultdict(list))
cachedata.runrecs = defaultdict(lambda: defaultdict(list))
cachedata.possible_world = []
@@ -197,6 +206,9 @@ class CoreRecipeInfo(RecipeInfoCommon):
cachedata.stamp_extrainfo[fn] = self.stamp_extrainfo
cachedata.file_checksums[fn] = self.file_checksums
+ cachedata.compatible_task_deps[fn] = self.task_cdepends
+ cachedata.target_sys[fn] = self.target_sys
+
provides = [self.pn]
for provide in self.provides:
if provide not in provides:
@@ -214,6 +226,23 @@ class CoreRecipeInfo(RecipeInfoCommon):
if dep not in cachedata.all_depends:
cachedata.all_depends.append(dep)
+ for cdep in self.cdepends:
+ if cdep not in cachedata.deps[fn]:
+ cachedata.deps[fn].append(cdep)
+ if cdep not in cachedata.compatible_deps[fn]:
+ cachedata.compatible_deps[fn].append(cdep)
+ bb.debug(1, "cdepends: Adding %s to compatible_deps for %s", cdep, fn)
+ if cdep not in cachedata.all_compatible_deps[fn]:
+ cachedata.all_compatible_deps[fn].append(cdep)
+ bb.debug(1, "cdepends: Adding %s to all_compatible_deps for %s", cdep, fn)
+ if cdep not in cachedata.all_depends:
+ cachedata.all_depends.append(cdep)
+ cachedata.all_cdepends.append(cdep)
+
+ for cdep in cachedata.all_compatible_deps[fn]:
+ if cdep not in cachedata.compatible_deps[fn]:
+ bb.warn("cdepends: %s removed from CDEPENDS for %s because it is also in DEPENDS" % (cdep, fn))
+
rprovides = self.rprovides
for package in self.packages:
cachedata.packages[package].append(fn)
--
2.7.4
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-siggen.py-add-support-for-CDEPENDS.patch --]
[-- Type: text/x-patch; name="0002-siggen.py-add-support-for-CDEPENDS.patch", Size: 2129 bytes --]
From e2c8ba70a2b723e1fc67215b1ff0ccb105341633 Mon Sep 17 00:00:00 2001
From: Michael Ho <Michael.Ho@bmw.de>
Date: Fri, 30 Jun 2017 06:52:52 +0200
Subject: [PATCH 2/9] siggen.py: add support for CDEPENDS
To: yocto@yoctoproject.org
This change modifies the basic signature generator to not add the task
dependencies sstate hash of a dependers task when that task comes from
a provider that is listed in the CDEPENDS variable.
---
bitbake/lib/bb/siggen.py | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 0352e45..e4b8374 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -86,6 +86,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST", True) or "").split())
self.taskwhitelist = None
+ self.cdepends_disabled = data.getVar("BB_CDEPENDS_DISABLED", True)
self.init_rundepcheck(data)
def init_rundepcheck(self, data):
@@ -180,10 +181,20 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.runtaskdeps[k] = []
self.file_checksum_values[k] = {}
recipename = dataCache.pkg_fn[fn]
+
+ def is_cdepends_task(fn, dep, dataCache):
+ for cdep in dataCache.compatible_deps[fn]:
+ for cdep_provider_fn in dataCache.providers[cdep]:
+ if dep.startswith(cdep_provider_fn):
+ return True
+ return False
+
for dep in sorted(deps, key=clean_basepath):
depname = dataCache.pkg_fn[self.pkgnameextract.search(dep).group('fn')]
if not self.rundep_check(fn, recipename, task, dep, depname, dataCache):
continue
+ if not self.cdepends_disabled and is_cdepends_task(fn, dep, dataCache):
+ continue
if dep not in self.taskhash:
bb.fatal("%s is not in taskhash, caller isn't calling in dependency order?", dep)
data = data + self.taskhash[dep]
--
2.7.4
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-runqueue.py-add-support-for-CDEPENDS.patch --]
[-- Type: text/x-patch; name="0003-runqueue.py-add-support-for-CDEPENDS.patch", Size: 17922 bytes --]
From 229bbc3e3feb0b60a70cb98bc1648ffe09b106ce Mon Sep 17 00:00:00 2001
From: Michael Ho <Michael.Ho@bmw.de>
Date: Fri, 30 Jun 2017 06:53:31 +0200
Subject: [PATCH 3/9] runqueue.py: add support for CDEPENDS
To: yocto@yoctoproject.org
This change modifies the runqueue execution to inject a check where
tasks with a cdepends flag run an extra section of code that invokes a
custom sstate hashing check and compatibility checker (defined by the
recipe).
The result of this extra section of code is then used to decide
whether to skip the sstate/stamp coverage checks during the runqueue
execution.
---
bitbake/lib/bb/runqueue.py | 197 ++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 186 insertions(+), 11 deletions(-)
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 878028a..3962ee1 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -35,6 +35,7 @@ import bb
from bb import msg, data, event
from bb import monitordisk
import subprocess
+import imp
try:
import cPickle as pickle
@@ -237,6 +238,7 @@ class RunQueueData:
self.runq_depends = []
self.runq_revdeps = []
self.runq_hash = []
+ self.runq_arch = []
def runq_depends_names(self, ids):
import re
@@ -555,6 +557,7 @@ class RunQueueData:
self.runq_depends.append(depends)
self.runq_revdeps.append(set())
self.runq_hash.append("")
+ self.runq_arch.append("")
runq_build.append(0)
@@ -668,6 +671,7 @@ class RunQueueData:
del runq_build[listid-delcount]
del self.runq_revdeps[listid-delcount]
del self.runq_hash[listid-delcount]
+ del self.runq_arch[listid-delcount]
delcount = delcount + 1
maps.append(-1)
@@ -806,6 +810,16 @@ class RunQueueData:
if hasattr(bb.parse.siggen, "tasks_resolved"):
bb.parse.siggen.tasks_resolved(virtmap, virtpnmap, self.dataCache)
+ # Iterate over the fn list and cache the variable MULTIMACH_TARGET_SYS
+ cached_archs = []
+ for i in range(max(self.runq_fnid)+1):
+ cached_archs.append("")
+ dealtwith = set()
+ for i in self.runq_fnid:
+ if i not in dealtwith:
+ dealtwith.add(i)
+ cached_archs[i] = self.dataCache.target_sys[self.taskData.fn_index[i]]
+
# Iterate over the task list and call into the siggen code
dealtwith = set()
todeal = set(range(len(self.runq_fnid)))
@@ -818,6 +832,9 @@ class RunQueueData:
for dep in self.runq_depends[task]:
procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
+ self.runq_arch[task] = cached_archs[self.runq_fnid[task]]
+ if self.runq_task[task] == "do_compatibility_report":
+ bb.debug(1, "cdepends: hash for %s:%s is %s" % (self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], self.runq_hash[task]))
return len(self.runq_fnid)
@@ -1281,6 +1298,8 @@ class RunQueueExecute:
self.runq_running = []
self.runq_complete = []
+ self.tainted = []
+
self.build_stamps = {}
self.build_stamps2 = []
self.failed_fnids = []
@@ -1503,6 +1522,18 @@ class RunQueueExecuteTasks(RunQueueExecute):
"""
self.runq_complete[task] = 1
for revdep in self.rqdata.runq_revdeps[task]:
+ # Taint reverse dependency if task is tainted from a cdepends check
+ # in its build tree and if it is not chained by CDEPENDS (we don't
+ # want cdepends tainting to cross the compatibility checking
+ # boundaries)
+ if task in self.tainted:
+ taskfn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
+ taskpn = self.rqdata.dataCache.pkg_fn[taskfn]
+ revdepfn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[revdep]]
+ revdeppn = self.rqdata.dataCache.pkg_fn[revdepfn]
+ # Try to be smart so that tainting stops between CDEPENDS items
+ if revdepfn == taskfn or not taskpn in self.rqdata.dataCache.compatible_deps[revdep]:
+ self.tainted.append(revdep)
if self.runq_running[revdep] == 1:
continue
if self.runq_buildable[revdep] == 1:
@@ -1559,17 +1590,159 @@ class RunQueueExecuteTasks(RunQueueExecute):
fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
taskname = self.rqdata.runq_task[task]
- if task in self.rq.scenequeue_covered:
- logger.debug(2, "Setscene covered task %s (%s)", task,
- self.rqdata.get_user_idstring(task))
- self.task_skip(task, "covered")
- return True
+ # Run compatibility check if task recipe has cdepends flag set and if stamp or setscene covered. Skip if
+ # CDEPENDS is disabled by config however.
+ # TODO: Skip if the cache compatibility task is not covered (maybe we crashed halfway through compile of the
+ # recipe, thus we want to continue with what we have in our workdir and not try to compatibility check -
+ # although we need to check the integrity of the workdir by comparing the sstate hash it starts with and
+ # ends with before we can add this check).
+ compatibility_check_ran = False
+ if (taskname in self.rqdata.dataCache.compatible_task_deps[fn]):
+ if (not self.cfgData.getVar("BB_CDEPENDS_DISABLED", True)) and (task in self.rq.scenequeue_covered or self.rq.check_stamp_task(task, taskname, cache=self.stampcache)):
+ # Parse recipe to extract cdepends variables and run compatibility check for each cdepends
+ tmp_cachedata = bb.cache.Cache.loadDataFull(fn, self.cooker.collection.get_file_appends(fn), self.cooker.data)
+
+ # Loop over all tasks that have the cdepends flag set
+ for cdepend in self.rqdata.dataCache.compatible_task_deps[fn][taskname].split():
+ compatibility_result = "incompatible"
+
+ cdepend_tool = tmp_cachedata.getVar(("CDEPENDS_TOOL_%s" % (cdepend,)), True) or ""
+ cdepend_searchdir = tmp_cachedata.getVar("CDEPENDS_SEARCHDIR", True) or ""
+ cdepend_workdir = tmp_cachedata.getVar("CDEPENDS_WORKDIR", True) or ""
+
+ logger.debug(1, "cdepends: cdepend=%s,cdepend_tool=%s,cdepend_searchdir=%s,cdepend_workdir=%s",
+ cdepend, cdepend_tool, cdepend_searchdir, cdepend_workdir)
+
+ # Determine the provider of the cdepends and get the architecture/hashes
+
+ # XXX: Potential room to improve by fixing this to properly look through indices. However it
+ # may not be possible to do so. We start with PN's and we need to find the actual provider
+ # and the task do_cache_compatibility_reports to calculate the hashes to build against
+ # (using all providers hashes if multiple providers are available)
+ cdepend_providers = self.rqdata.dataCache.providers[cdepend]
+ cdepend_preferred_provider = tmp_cachedata.getVar("PREFERRED_PROVIDER_%s" % (cdepend,)) or cdepend
+ cdepend_hashes = ""
+ cdepend_hashes_provider = []
+ cdepend_arch = ""
+ cdepend_arch_provider = "self"
+
+ if cdepend_preferred_provider in cdepend_providers:
+ for x in range(len(self.rqdata.runq_task)):
+ x_name = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[x]] + ":" + self.rqdata.runq_task[x]
+ if x_name == cdepend_preferred_provider + ":" + "do_compatibility_report":
+ cdepend_hashes = self.rqdata.runq_hash[x]
+ cdepend_hashes_provider.append(cdepend_preferred_provider)
+ cdepend_arch = self.rqdata.dataCache.target_sys[self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[x]]]
+ cdepend_arch_provider = cdepend_preferred_provider
+ break
+ else:
+ # if multiple providers available and no preferred provider, then merge all (sorted) hashes for safety
+ found_hashes = dict()
+ for cdepend_provider in cdepend_providers:
+ for x in range(len(self.rqdata.runq_task)):
+ x_name = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[x]] + ":" + self.rqdata.runq_task[x]
+ if x_name == cdepend_provider + ":" + "do_compatibility_report":
+ found_hashes[x_name] = self.rqdata.runq_hash[x]
+ cdepend_hashes_provider.append(cdepend_provider)
+ cdepend_arch = self.rqdata.dataCache.target_sys[self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[x]]]
+ cdepend_arch_provider = cdepend_provider
+ sorted_hashes = sorted(found_hashes.values())
+ for cdepend_provider_hash in sorted_hashes:
+ cdepend_hashes += cdepend_provider_hash
+
+ # if cdepends insane skip is set, ignore the sstate hashing checks
+ # (since we won't be able to fetch the required sstate hashes for
+ # dependencies without do_compatibility_report)
+ if not tmp_cachedata.getVar("CDEPENDS_INSANE_SKIP_%s" % (cdepend,), True):
+ if not cdepend_hashes:
+ bb.fatal("cdepends: Cannot find provider task for %s:do_compatibility_report (providers: %s)" % (cdepend, cdepend_providers))
+
+ logger.debug(1, "cdepends: Provider for %s: %s,runq_hash=%s,runq_arch=%s", cdepend, ','.join(cdepend_hashes_provider), cdepend_hashes, cdepend_arch)
+
+ # First just check if the sstate hashes are okay to try to avoid the actual compatibility check
+ cached_sstate_hashes_path = os.path.join(cdepend_workdir, "sstate-hashes")
+ logger.debug(2, "cdepends: Trying to grab sstate hashes from %s.", cached_sstate_hashes_path)
+
+ current_runq_hash = cdepend_hashes
+ cached_runq_hash = ""
+ try:
+ with open(cached_sstate_hashes_path, "r") as hashes_log:
+ hashes_log_lines = hashes_log.readlines()
+ for line in hashes_log_lines:
+ line = line.strip()
+ try:
+ target_name, cached_target_hash = line.split(" ", 1)
+ if target_name == cdepend:
+ cached_runq_hash = cached_target_hash
+ except ValueError:
+ continue
+ except (OSError, IOError) as e:
+ bb.warn("cdepends: Could not open cached sstate hashes for %s:%s: %s" % (fn, taskname, str(e)))
+
+ if not cached_runq_hash == "":
+ logger.debug(2, "cdepends: Comparing sstate hashes: cached: \"%s\" to latest: \"%s\".", cached_runq_hash, current_runq_hash)
+ if current_runq_hash == cached_runq_hash:
+ logger.debug(1, "cdepends: \"%s\" sstate hash \"%s\" still valid for %s:%s.", cdepend, cached_runq_hash, fn, taskname)
+ continue
+
+ # if the sstate hash does not hold, then run the cdepend check
+ logger.debug(1, "cdepends: \"%s\" sstate hash has changed from \"%s\" to \"%s\" for %s:%s.", cdepend, cached_runq_hash, current_runq_hash, fn, taskname)
+ else:
+ logger.debug(1, "cdepends: \"%s\" sstate hash was not found in compatibility cache for %s:%s", cdepend, fn, taskname)
+
+ # Run actual compatibility checkers from here onwards
+ latest_reports_dir = os.path.join(cdepend_searchdir, cdepend_arch, cdepend)
+ cached_reports_dir = os.path.join(cdepend_workdir, cdepend)
+
+ cdepend_module_name = ("cdepends_%s" % (cdepend_tool,))
+ f = None
+ try:
+ logger.debug(2, "cdepends: Searching for python module \"%s\".", cdepend_module_name)
+ f, filename, description = imp.find_module(cdepend_module_name)
+ cdepend_module = imp.load_module(cdepend_module_name, f, filename, description)
+ logger.debug(2, "cdepends: Running compatibility check for %s:%s using python module \"%s\". Passing arguments \"%s\" and \"%s\"",
+ fn, taskname, cdepend_module_name, cached_reports_dir, latest_reports_dir)
+ compatibility_result = cdepend_module.compatibility_check(cached_reports_dir, latest_reports_dir, tmp_cachedata)
+ bb.note("cdepends: Compatibility check for %s:%s using tool (%s) returned \"%s\" for (%s)." % (fn, taskname, cdepend_tool, compatibility_result, cdepend))
+ except ImportError as e:
+ bb.warn("cdepends: Could not find and load/run cdepends module %s" % (cdepend_module_name,))
+ finally:
+ if f is not None:
+ f.close()
+
+ # Set a flag to show we actually ran a compatibility check to help improve our notifications to the UI
+ compatibility_check_ran = True
+
+ if compatibility_result != "compatible":
+ logger.debug(1, "cdepends: Tainting build chain of %s:%s because of %s.", fn, taskname, cdepend)
+ self.tainted.append(task)
+ logger.debug(1, "cdepends: Tainted task %s due to compatibility changes", task)
+
+ if compatibility_result != "incompatible":
+ bb.fatal("cdepends: Fatal compatibility error for %s." % (task,))
+ else:
+ bb.note("cdepends: Skipping cdepends checks for %s (%s)" % (task, self.rqdata.get_user_idstring(task)))
- if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
- logger.debug(2, "Stamp current task %s (%s)", task,
- self.rqdata.get_user_idstring(task))
- self.task_skip(task, "existing")
- return True
+ # Don't believe setscene or stamp if tainted flag set for task
+ if task not in self.tainted:
+ if task in self.rq.scenequeue_covered:
+ logger.debug(2, "Setscene covered task %s (%s)", task,
+ self.rqdata.get_user_idstring(task))
+ self.task_skip(task, "covered")
+ if compatibility_check_ran:
+ bb.note("cdepends: Avoided rebuilding %s" % (fn,))
+ return True
+
+ if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
+ logger.debug(2, "Stamp current task %s (%s)", task,
+ self.rqdata.get_user_idstring(task))
+ self.task_skip(task, "existing")
+ if compatibility_check_ran:
+ bb.note("cdepends: Avoided rebuilding %s" % (fn,))
+ return True
+
+ if compatibility_check_ran and task in self.tainted:
+ bb.note("cdepends: Rebuilding %s" % (fn,))
taskdep = self.rqdata.dataCache.task_deps[fn]
if 'noexec' in taskdep and taskname in taskdep['noexec']:
@@ -1642,7 +1815,9 @@ class RunQueueExecuteTasks(RunQueueExecute):
taskname = self.rqdata.runq_task[revdep]
deps = self.rqdata.runq_depends[revdep]
provides = self.rqdata.dataCache.fn_provides[fn]
- taskdepdata[revdep] = [pn, taskname, fn, deps, provides]
+ taskhash = self.rqdata.runq_hash[revdep]
+ taskarch = self.rqdata.runq_arch[revdep]
+ taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, taskarch]
for revdep2 in deps:
if revdep2 not in taskdepdata:
additional.append(revdep2)
--
2.7.4
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-taskdata.py-add-support-for-CDEPENDS.patch --]
[-- Type: text/x-patch; name="0004-taskdata.py-add-support-for-CDEPENDS.patch", Size: 1532 bytes --]
From fe6f61e7d29c64d8ffa4943283784339ff2d7411 Mon Sep 17 00:00:00 2001
From: Michael Ho <Michael.Ho@bmw.de>
Date: Fri, 30 Jun 2017 06:54:27 +0200
Subject: [PATCH 4/9] taskdata.py: add support for CDEPENDS
To: yocto@yoctoproject.org
This change adds a new variable to the TaskData called cdepids and
contains the task ids of tasks that are CDEPENDS of the task being
called on.
---
bitbake/lib/bb/taskdata.py | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
index 4d12b33..5b9ea8c 100644
--- a/bitbake/lib/bb/taskdata.py
+++ b/bitbake/lib/bb/taskdata.py
@@ -74,6 +74,8 @@ class TaskData:
self.skiplist = skiplist
+ self.cdepids = {}
+
def getbuild_id(self, name):
"""
Return an ID number for the build target name.
@@ -212,6 +214,14 @@ class TaskData:
self.depids[fnid] = dependids.keys()
logger.debug(2, "Added dependencies %s for %s", str(dataCache.deps[fn]), fn)
+ # Work out build compatible dependencies
+ if not fnid in self.cdepids:
+ cdependids = {}
+ for cdepend in dataCache.compatible_deps[fn]:
+ cdependids[self.getbuild_id(cdepend)] = None
+ self.cdepids[fnid] = cdependids.keys()
+ logger.debug(2, "Added compatible dependencies %s for %s", str(dataCache.compatible_deps[fn]), fn)
+
# Work out runtime dependencies
if not fnid in self.rdepids:
rdependids = {}
--
2.7.4
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-rm_work.bbclass-add-exception-for-do_cache_compatibi.patch --]
[-- Type: text/x-patch; name="0005-rm_work.bbclass-add-exception-for-do_cache_compatibi.patch", Size: 1113 bytes --]
From d0149bcd7fba3d432eebd6218d6199878bdd91f8 Mon Sep 17 00:00:00 2001
From: Michael Ho <Michael.Ho@bmw.de>
Date: Fri, 30 Jun 2017 06:54:39 +0200
Subject: [PATCH 5/9] rm_work.bbclass: add exception for
do_cache_compatibility_reports
To: yocto@yoctoproject.org
This change is needed to handle the situation of rm_work clearing
compatibility reports that are actually shared to depender recipes.
This is similar to the workarounds in place for do_package.
---
meta/classes/rm_work.bbclass | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass
index 5e9efc1..ce802cd 100644
--- a/meta/classes/rm_work.bbclass
+++ b/meta/classes/rm_work.bbclass
@@ -76,6 +76,11 @@ do_rm_work () {
i=dummy
break
;;
+ *do_cache_compatibility_reports|*do_cache_compatibility_reports.*|*do_cache_compatibility_reports_setscene.*)
+ rm -f $i;
+ i=dummy
+ break
+ ;;
*_setscene*)
i=dummy
break
--
2.7.4
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-native.bbclass-add-support-for-CDEPENDS.patch --]
[-- Type: text/x-patch; name="0006-native.bbclass-add-support-for-CDEPENDS.patch", Size: 1271 bytes --]
From 77070002f7970ccab89b38eb5eb91bc768120052 Mon Sep 17 00:00:00 2001
From: Michael Ho <Michael.Ho@bmw.de>
Date: Fri, 30 Jun 2017 06:55:24 +0200
Subject: [PATCH 6/9] native.bbclass: add support for CDEPENDS
To: yocto@yoctoproject.org
Remap CDEPENDS in the same way DEPENDS is remapped when extending a
recipe.
---
meta/classes/native.bbclass | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/meta/classes/native.bbclass b/meta/classes/native.bbclass
index bcbcd61..1357f1f 100644
--- a/meta/classes/native.bbclass
+++ b/meta/classes/native.bbclass
@@ -163,7 +163,18 @@ python native_virtclass_handler () {
nprovides.append(prov)
e.data.setVar("PROVIDES", ' '.join(nprovides))
-
+ map_dependencies("CDEPENDS", e.data)
+ for cdep in bb.utils.explode_deps(e.data.getVar("CDEPENDS") or ""):
+ cdep_tool = d.getVar("CDEPENDS_TOOL_%s" % (cdep,), True) or ""
+ if cdep == pn:
+ continue
+ elif "-cross-" in cdep:
+ cdep.replace("-cross", "-native")
+ elif not cdep.endswith("-native"):
+ cdep += "-native"
+ else:
+ pass
+ e.data.setVar("CDEPENDS_TOOL_%s" % (cdep,), cdep_tool)
}
addhandler native_virtclass_handler
--
2.7.4
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #8: 0007-nativesdk.bbclass-add-support-for-CDEPENDS.patch --]
[-- Type: text/x-patch; name="0007-nativesdk.bbclass-add-support-for-CDEPENDS.patch", Size: 1697 bytes --]
From 2ce44b4ac4ebae7e90e07e95911e8c5be0ff15c6 Mon Sep 17 00:00:00 2001
From: Michael Ho <Michael.Ho@bmw.de>
Date: Fri, 30 Jun 2017 06:55:45 +0200
Subject: [PATCH 7/9] nativesdk.bbclass: add support for CDEPENDS
To: yocto@yoctoproject.org
Remap CDEPENDS in the same way DEPENDS is remapped when extending a
recipe. Remaps the CDEPENDS_TOOL_* variable also.
---
meta/classes/nativesdk.bbclass | 3 +++
meta/lib/oe/classextend.py | 5 +++++
2 files changed, 8 insertions(+)
diff --git a/meta/classes/nativesdk.bbclass b/meta/classes/nativesdk.bbclass
index f74da62..6a98732 100644
--- a/meta/classes/nativesdk.bbclass
+++ b/meta/classes/nativesdk.bbclass
@@ -86,6 +86,9 @@ python () {
clsextend.map_packagevars()
clsextend.map_variable("PROVIDES")
clsextend.map_regexp_variable("PACKAGES_DYNAMIC")
+
+ clsextend.rename_cdepends_tools()
+ clsextend.map_depends_variable("CDEPENDS")
}
addhandler nativesdk_virtclass_handler
diff --git a/meta/lib/oe/classextend.py b/meta/lib/oe/classextend.py
index 5107ecd..a19b213 100644
--- a/meta/lib/oe/classextend.py
+++ b/meta/lib/oe/classextend.py
@@ -108,6 +108,11 @@ class ClassExtender(object):
for subs in variables:
self.d.renameVar("%s_%s" % (subs, pkg_mapping[0]), "%s_%s" % (subs, pkg_mapping[1]))
+ def rename_cdepends_tools(self):
+ for cdep in (self.d.getVar("CDEPENDS", True) or "").split():
+ renamed_cdep = self.map_depends(cdep)
+ self.d.renameVar("CDEPENDS_TOOL_%s" % (cdep,), "CDEPENDS_TOOL_%s" % (renamed_cdep,))
+
class NativesdkClassExtender(ClassExtender):
def map_depends(self, dep):
if dep.startswith(self.extname):
--
2.7.4
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: 0008-classes-add-bbclasses-compatible-depends-and-compati.patch --]
[-- Type: text/x-patch; name="0008-classes-add-bbclasses-compatible-depends-and-compati.patch", Size: 10250 bytes --]
From 7d9d7e67099ef8ed4c6aa32453bf5cbd8ae53957 Mon Sep 17 00:00:00 2001
From: Michael Ho <Michael.Ho@bmw.de>
Date: Fri, 30 Jun 2017 06:56:36 +0200
Subject: [PATCH 8/9] classes: add bbclasses compatible-depends and
compatible-depends-report
To: yocto@yoctoproject.org
Adds bbclasses that are used in conjunction with triggering the
CDEPENDS mechanisms in Bitbake.
The compatible-depends bbclass adds the base functionality to cache
compatibility reports from the dependencies.
The compatible-depends-report bbclass adds the base functionality to
generate the compatibility reports for recipes that can be depended
on via CDEPENDS.
---
.../classes/compatible-depends-report.bbclass | 32 +++++
meta-yocto/classes/compatible-depends.bbclass | 144 +++++++++++++++++++++
2 files changed, 176 insertions(+)
create mode 100644 meta-yocto/classes/compatible-depends-report.bbclass
create mode 100644 meta-yocto/classes/compatible-depends.bbclass
diff --git a/meta-yocto/classes/compatible-depends-report.bbclass b/meta-yocto/classes/compatible-depends-report.bbclass
new file mode 100644
index 0000000..ffb9e4e
--- /dev/null
+++ b/meta-yocto/classes/compatible-depends-report.bbclass
@@ -0,0 +1,32 @@
+# Not to be used directly - The concrete bbclass for a tool should inherit this
+
+# Working directory and output directory for reports
+# This needs to stage somewhere that sstate can recover so that when rebuilding
+# sysroots from sstate this is also recovered.
+COMPATIBILITY_REPORT_WORKDIR = "${WORKDIR}/compatibility-info/"
+COMPATIBILITY_REPORT_DIR = "${TMPDIR}/compatibility-info/${MULTIMACH_TARGET_SYS}/${PN}/"
+
+# Dummy task for generating reports - the concrete formats are expected to
+# append to this task to add their actual code for generating reports
+do_compatibility_report[dirs] += "${COMPATIBILITY_REPORT_WORKDIR}"
+do_compatibility_report () {
+ :
+}
+addtask compatibility_report after do_package before do_build
+
+# Make do_compatibility_report task SSTATE compatible
+SSTATETASKS += "do_compatibility_report"
+
+do_compatibility_report[sstate-inputdirs] = "${COMPATIBILITY_REPORT_WORKDIR}"
+do_compatibility_report[sstate-outputdirs] = "${COMPATIBILITY_REPORT_DIR}"
+
+python do_compatibility_report_setscene () {
+ sstate_setscene(d)
+}
+addtask do_compatibility_report_setscene
+
+# Needed for recipes like native XXX: is this better than just adding do_install by default?
+python __anonymous () {
+ if not d.getVarFlag("do_package", "task", True):
+ d.appendVarFlag('do_compatibility_report', 'depends', ' do_install')
+}
diff --git a/meta-yocto/classes/compatible-depends.bbclass b/meta-yocto/classes/compatible-depends.bbclass
new file mode 100644
index 0000000..3e6823d
--- /dev/null
+++ b/meta-yocto/classes/compatible-depends.bbclass
@@ -0,0 +1,144 @@
+# Recipes that use CDEPENDS inherit this bbclass
+
+# Add default CDEPENDS because this variable must be declared
+CDEPENDS ?= ""
+
+# Add CDEPENDS to the sstate hashing signatures so that changes to this variable
+# trigger sstate invalidations and can enforce rebuilds
+do_unpack[vardeps] += "CDEPENDS"
+do_cache_compatibility_reports[vardeps] += "CDEPENDS"
+
+# Delay the compiling tasks of the recipe to wait for the report data of other
+# recipes it CDEPENDS on (if they have a do_compatibility_report task).
+do_unpack[deptask] += "do_compatibility_report"
+
+# Give do_unpack task a cdepends flag, so runQueueExecuteTask can determine if
+# it should run compatibility checks prior to running do_unpack
+do_unpack[cdepends] = "${CDEPENDS}"
+
+# Working directories to get reports from and store them while working on them
+# Extracted by Bitbake during compatibility checking
+# Potential bug here, what if a dependency doesn't fit into these arch targets, how do we find out his architecture?
+# To fix: we now pass architecture through BBTASKDEPS - we need to use this somehow
+CDEPENDS_SEARCHDIR = "${TMPDIR}/compatibility-info/"
+CDEPENDS_STAGEDIR = "${WORKDIR}/dependency-compatibility-info-staging/"
+CDEPENDS_WORKDIR = "${WORKDIR}/dependency-compatibility-info/"
+
+# Task to copy the compatibility reports of its CDEPENDS to the local working
+# directory. These are then cached by sstate and placed into the expected
+# directory for Bitbake to use for future compatibility checks.
+python copy_compatibility_reports () {
+ import errno
+ import os
+ import shutil
+ import subprocess
+
+ cdepends = d.getVar("CDEPENDS", True).strip() or ""
+ cdepends_searchdir = d.getVar("CDEPENDS_SEARCHDIR", True)
+ cdepends_stagedir = d.getVar("CDEPENDS_STAGEDIR", True)
+
+ taskdepdata = d.getVar("BB_TASKDEPDATA", True)
+
+ if cdepends_stagedir:
+ # Need to make the directory ourselves because prefuncs are too early..
+ bb.utils.mkdirhier(cdepends_stagedir)
+
+ for cdepend in cdepends.split():
+ cdepend = cdepend.strip()
+ cdepend_arch = d.getVar("MULTIMACH_TARGET_SYS", True) or ""
+ cdepend_provider = cdepend
+ for task in taskdepdata:
+ if cdepend in taskdepdata[task][4]:
+ if taskdepdata[task][1] == "do_compatibility_report":
+ cdepend_arch = taskdepdata[task][6]
+ cdepend_provider = taskdepdata[task][0]
+
+ shutil.rmtree(os.path.join(cdepends_stagedir, cdepend), ignore_errors=True)
+ try:
+ shutil.copytree(os.path.join(cdepends_searchdir, cdepend_arch, cdepend_provider), os.path.join(cdepends_stagedir, cdepend), symlinks=True)
+ bb.debug(3, "cdepends: %s.copy_compatibility_reports: copied from %s to %s" % (d.getVar("PN", True), os.path.join(cdepends_searchdir, cdepend_arch, cdepend_provider), os.path.join(cdepends_stagedir, cdepend)))
+ except OSError as e:
+ # Sometimes we CDEPENDS on things with no compatibility report
+ # (ie. binary deliveries) - so avoid throwing errors when
+ # CDEPENDS_INSANE_SKIP is set for this dependency
+ insane_skip = d.getVar("CDEPENDS_INSANE_SKIP_%s" % (cdepend,), True)
+ if e.errno == errno.ENOENT and insane_skip:
+ pass
+ else:
+ raise e
+}
+
+do_cache_compatibility_reports () {
+ :
+}
+do_cache_compatibility_reports[prefuncs] += "copy_compatibility_reports"
+addtask cache_compatibility_reports after do_install do_package before do_populate_sysroot do_build
+
+# Make do_cache_compatibility_reports task SSTATE compatible
+SSTATETASKS += "do_cache_compatibility_reports"
+
+do_cache_compatibility_reports[dirs] = "${CDEPENDS_STAGEDIR} ${CDEPENDS_WORKDIR}"
+do_cache_compatibility_reports[sstate-inputdirs] = "${CDEPENDS_STAGEDIR}"
+do_cache_compatibility_reports[sstate-outputdirs] = "${CDEPENDS_WORKDIR}"
+
+python do_cache_compatibility_reports_setscene () {
+ sstate_setscene(d)
+}
+addtask do_cache_compatibility_reports_setscene
+
+python cache_sstate_hashes () {
+ import os
+
+ cdepends = d.getVar("CDEPENDS", True).strip() or ""
+ cdepends_hashes = dict()
+ if cdepends:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: CDEPENDS: %s" % (d.getVar("PN", True), cdepends))
+ taskdepdata = d.getVar("BB_TASKDEPDATA", True)
+
+ for cdepend in cdepends.split():
+ cdepend = cdepend.strip()
+ cdepend_preferred_provider = d.getVar("PREFERRED_PROVIDER_%s" % (cdepend), True) or ""
+ cdepend_preferred_provider_hash = ""
+
+ cdepend_providers = ""
+ cdepend_hashes = ""
+ found_hashes = dict()
+ for task in taskdepdata:
+ if cdepend in taskdepdata[task][4]:
+ if taskdepdata[task][1] == "do_compatibility_report":
+ found_hashes[taskdepdata[task][0]] = taskdepdata[task][5]
+ if cdepend_preferred_provider == taskdepdata[task][0]:
+ cdepend_preferred_provider_hash = taskdepdata[task][5]
+
+ # Sort hashes
+ cdepend_providers = ','.join(found_hashes.keys())
+ sorted_hashes = sorted(found_hashes.values())
+ for cdepend_provider_hash in sorted_hashes:
+ cdepend_hashes += cdepend_provider_hash
+
+ if cdepend_preferred_provider_hash:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: CDEPENDS_HASH_%s=%s from single provider: %s" % (d.getVar("PN", True), cdepend, cdepend_preferred_provider_hash, cdepend_preferred_provider))
+ cdepends_hashes[cdepend] = cdepend_preferred_provider_hash
+ else:
+ if len(found_hashes.keys()) == 0:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: No sstate hashes found for provider of: %s" % (d.getVar("PN", True), cdepend))
+ cdepends_hashes[cdepend] = ""
+ elif len(found_hashes.keys()) == 1:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: CDEPENDS_HASH_%s=%s from single provider: %s" % (d.getVar("PN", True), cdepend, cdepend_hashes, cdepend_providers))
+ cdepends_hashes[cdepend] = cdepend_hashes
+ else:
+ bb.debug(1, "cdepends: %s:cache_sstate_hashes: CDEPENDS_HASH_%s=%s from group of providers: %s" % (d.getVar("PN", True), cdepend, cdepend_hashes, cdepend_providers))
+ cdepends_hashes[cdepend] = cdepend_hashes
+
+ # Write to sstate cached compatibility reports directory
+ try:
+ bb.utils.mkdirhier(d.getVar("CDEPENDS_STAGEDIR", True))
+ sstate_hash_manifest_file_path = os.path.join(d.getVar("CDEPENDS_STAGEDIR", True), "sstate-hashes")
+ with open(sstate_hash_manifest_file_path, "w") as sstate_hash_manifest_file:
+ for cdepend in cdepends_hashes:
+ sstate_hash_manifest_file.write("%s %s\n" % (cdepend, cdepends_hashes[cdepend]))
+ sstate_hash_manifest_file.close()
+ except (OSError, IOError) as e:
+ bb.warn("cdepends: Could not cache sstate hashes for %s: %s" % (d.getVar("PN", True), str(e)))
+}
+do_cache_compatibility_reports[postfuncs] += "cache_sstate_hashes"
--
2.7.4
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #10: 0009-recipes-cdepends-test-example-recipe-framework-for-u.patch --]
[-- Type: text/x-patch; name="0009-recipes-cdepends-test-example-recipe-framework-for-u.patch", Size: 4484 bytes --]
From ebc8216d7439fbf60c98a326382687f212dbbeb4 Mon Sep 17 00:00:00 2001
From: Michael Ho <Michael.Ho@bmw.de>
Date: Fri, 30 Jun 2017 07:07:27 +0200
Subject: [PATCH 9/9] recipes-cdepends-test: example recipe framework for using
CDEPENDS
To: yocto@yoctoproject.org
Adds required classes, python library, and recipes for testing the
CDEPENDS usage of a parent recipe generating text files and another
recipe that depends on the file checksums of a key file from the
parent.
Recipe cdepends-test1 should always rebuild and generate a new text
file. However, it contains a file called motd2.txt that is taken from
the recipe files path. This does not change as often.
Recipe cdepends-test2 depends on the motd2.txt file that is built in
the parent cdepends-test1. It only needs to recompile when this text
file is changed.
What should be found by running these examples is that after the
sstate cache is populated with a first run, the recipe cdepends-test2
should only rebuild when the file motd2.txt is modified in the recipe
cdepends-test1.
---
.../compatible-depends-report-pkgcontents.bbclass | 10 ++++++
meta-yocto/lib/cdepends_pkgcontents.py | 41 ++++++++++++++++++++++
meta-yocto/recipes-cdepends-test | 1 +
3 files changed, 52 insertions(+)
create mode 100644 meta-yocto/classes/compatible-depends-report-pkgcontents.bbclass
create mode 100644 meta-yocto/lib/cdepends_pkgcontents.py
create mode 160000 meta-yocto/recipes-cdepends-test
diff --git a/meta-yocto/classes/compatible-depends-report-pkgcontents.bbclass b/meta-yocto/classes/compatible-depends-report-pkgcontents.bbclass
new file mode 100644
index 0000000..a060a51
--- /dev/null
+++ b/meta-yocto/classes/compatible-depends-report-pkgcontents.bbclass
@@ -0,0 +1,10 @@
+inherit compatible-depends-report
+
+pkgcontents_compatibility_report () {
+ REPORT_TOOL="pkgcontents"
+ mkdir -p "${COMPATIBILITY_REPORT_WORKDIR}"/"${REPORT_TOOL}"
+ find "${WORKDIR}/packages-split/" -type f -exec sha1sum "{}" \; | tee "${COMPATIBILITY_REPORT_WORKDIR}"/"${REPORT_TOOL}"/manifest
+ sed -i -e "s?${WORKDIR}/packages-split/??" "${COMPATIBILITY_REPORT_WORKDIR}"/"${REPORT_TOOL}"/manifest
+}
+
+do_compatibility_report[postfuncs] += "pkgcontents_compatibility_report"
diff --git a/meta-yocto/lib/cdepends_pkgcontents.py b/meta-yocto/lib/cdepends_pkgcontents.py
new file mode 100644
index 0000000..6af4c84
--- /dev/null
+++ b/meta-yocto/lib/cdepends_pkgcontents.py
@@ -0,0 +1,41 @@
+import os.path
+
+import bb
+
+def compatibility_check(cached_report_dir, new_report_dir, bb_cachedata):
+ if not os.path.isdir(cached_report_dir):
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check cannot find %s" % (cached_report_dir,))
+ return "incompatible"
+
+ cached_manifest_path = os.path.join(cached_report_dir, "pkgcontents", "manifest")
+ if not os.path.isfile(cached_manifest_path):
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check cannot find %s" % (cached_manifest_path,))
+ return "incompatible"
+
+ new_manifest_path = os.path.join(new_report_dir, "pkgcontents", "manifest")
+ if not os.path.isfile(new_manifest_path):
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check cannot find %s" % (new_manifest_path,))
+ return "incompatible"
+
+ match = 1
+ try:
+ with open(cached_manifest_path) as f1, open(new_manifest_path) as f2:
+ for line1 in f1:
+ line_match = 0
+ for line2 in f2:
+ if line1 == line2:
+ line_match = 1
+ break
+ if line_match == 0:
+ match = 0
+ break
+ except (IOError, OSError) as e:
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check could not read manifest file/s: %s" % (str(e),))
+ return "incompatible"
+
+ if match == 0:
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check returning incompatible")
+ return "incompatible"
+ else:
+ bb.debug(1, "cdepends: cdepends_pkgcontents.compatibility_check returning compatible")
+ return "compatible"
diff --git a/meta-yocto/recipes-cdepends-test b/meta-yocto/recipes-cdepends-test
new file mode 160000
index 0000000..013ad65
--- /dev/null
+++ b/meta-yocto/recipes-cdepends-test
@@ -0,0 +1 @@
+Subproject commit 013ad65d7a884931eab619e9f613760ba209e1cf
--
2.7.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: RFC: Backwards compatibility checking sstate-cache
2017-06-30 9:48 RFC: Backwards compatibility checking sstate-cache Michael Ho
@ 2017-06-30 15:01 ` Darcy Watkins
2017-06-30 22:44 ` Michael Ho
2017-07-01 7:48 ` Martin Jansa
2017-09-21 16:11 ` Richard Purdie
2 siblings, 1 reply; 9+ messages in thread
From: Darcy Watkins @ 2017-06-30 15:01 UTC (permalink / raw)
To: Michael Ho; +Cc: yocto
[-- Attachment #1: Type: text/plain, Size: 5506 bytes --]
Hi,
It would also be nice to add some sort of hook script support where user supplied hook script performs a check and returns either a "yes it's OK" or a "no have to rebuild it" status.
And then tie that into the dependency logic.
I have unusual use case where I need to have a dependency on changes to the kernel version, but not necessarily something triggered after every kernel rebuild. The package I'm building will trigger OpenSSL and a whole whack of stuff each time it rebuilds.
It depends on the kernel version and the headers, but shouldn't need to depend on every patch or kconfig change that happens.
___
Regards,
Darcy
Darcy Watkins :: Senior Staff Engineer, Firmware
SIERRA WIRELESS
Direct +1 604 233 7989<tel:+1%20604%20233%207989> :: Fax +1 604 231 1109<tel:+1%20604%20231%201109> :: Main +1 604 231 1100<tel:+1%20604%20231%201100>
13811 Wireless Way :: Richmond, BC Canada V6V 3A4<x-apple-data-detectors://3/1>
[M4]
dwatkins@sierrawireless.com<mailto:dwatkins@sierrawireless.com> :: www.sierrawireless.com<http://www.sierrawireless.com/> :: www.inmotiontechnology.com<http://www.inmotiontechnology.com/>
On Jun 30, 2017, at 2:53 AM, Michael Ho <Michael.Ho@bmw-carit.de<mailto:Michael.Ho@bmw-carit.de>> wrote:
Hi, at BMW Car IT we are working on an experimental feature to improve sstate
cache hits and we are looking for comments on the approach who might have some
insights to the problem and seeing if anyone is interested in the feature for
mainline.
The sstate-cache of a recipe is tied closely to its build dependencies, as the
signature generated for a task includes all parent task's signatures as part of
the signature information. This means that when changes occur in the parent
recipes, even if insignificant, all children recipes that have valid sstate
cache must invalidate their sstate cache objects.
What this patchset does is propose to add another optional variable to recipes,
CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature
generation it excludes any parent tasks that come from dependencies listed in
it. This is to break the signature change domino effect.
This patchset also proposes modifying RunQueue to then be able to run a
compatibility checker during task execution phase for recipes and tasks that use
CDEPENDS and allow for deciding to re-execute a task despite being covered by
sstate-cache.
The patchset is based on the jethro branch for the poky repository, as this is
the branch that we are using. If the general idea sounds good, we can consider
porting it to master.
Included is an patch that adds an example recipe and compatibility checker,
where compatibility is based on the file checksums of the parent recipes
packages. An example recipe, cdepends-test1, generates a compatibility report
containing the file checksums of all files that it packages and which file paths
they are at. Another recipe, cdepends-test2, can then strip this compatibility
report to the minimal files it needs to be consistent and can compare the latest
checksums it used to configure/compile/install with and if they have changed,
trigger a rebuild. If not, the previous version restored from sstate-cache is
used.
We are still experimenting with the usages of this, including the use of having
abi-compliance-checker to compare the ABI of shared libraries as a compatibility
checker during RunQueue and using the results to avoid rebuilding child recipes
when the .so files they depend on are still compatible. Example use cases of
this are allowing recipes which provide multiple shared libraries to change a
single .so file without rebuilding all its children that depend on the other
shared libraries but not the one that changed.
We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it didn't meet
the feature requirements of what this compatibility checker callback is doing,
although maybe when porting to master we could refactor to make better use of
the work already done there. The current implementation is a bit hacky but
comments would be appreciated in regards to if the concept is feasible and if
people are interested in making use of it and their use cases.
Kind regards,
Michael Ho
--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm
Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael.ho@bmw-carit.de<mailto:michael.ho@bmw-carit.de>
Web: http://www.bmw-carit.de
--------------------------------------------------------
BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--------------------------------------------------------
<0001-cache.py-add-support-for-CDEPENDS.patch>
<0002-siggen.py-add-support-for-CDEPENDS.patch>
<0003-runqueue.py-add-support-for-CDEPENDS.patch>
<0004-taskdata.py-add-support-for-CDEPENDS.patch>
<0005-rm_work.bbclass-add-exception-for-do_cache_compatibi.patch>
<0006-native.bbclass-add-support-for-CDEPENDS.patch>
<0007-nativesdk.bbclass-add-support-for-CDEPENDS.patch>
<0008-classes-add-bbclasses-compatible-depends-and-compati.patch>
<0009-recipes-cdepends-test-example-recipe-framework-for-u.patch>
--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>
https://lists.yoctoproject.org/listinfo/yocto
[-- Attachment #2: Type: text/html, Size: 10525 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: Backwards compatibility checking sstate-cache
2017-06-30 15:01 ` Darcy Watkins
@ 2017-06-30 22:44 ` Michael Ho
0 siblings, 0 replies; 9+ messages in thread
From: Michael Ho @ 2017-06-30 22:44 UTC (permalink / raw)
To: Darcy Watkins; +Cc: yocto
Hi Darcy,
Thank you for the feedback. Actually, depending on what exactly you mean by user
supplied script, this patch set for the concept already supports that.
I'm not sure if my attachments got skewered by the mailing list or not but if
you can see my patch "0009-recipes-cdepends-test-example-recipe-framework-for-u.patch",
you can see an example implementation and what needs to be done to introduce a
custom compatibility checker.
This patch set provides the hooks for Bitbake and recipes to communicate about
the compatible dependency checker it needs to use. In essence, the recipes point
Bitbake to a python module that it can load from any meta-layer lib folder and
run a predefined function, passing the directories of the compatibility reports
(the latest from the freshly built recipe, and the older from the sstate cache
of the child recipe) which are also defined by the recipes. This python function
can then do whatever it wants with the compatibility data it has access to (i.e.
call a custom user script to look at the report information) and return
"incompatible" or "compatible" to control the RunQueue. Bitbake will rebuild if
incompatible is returned, and not rebuild if compatible is returned. It can also
return a different value which Bitbake will interpret as a compatibility error
(e.g. binary delivery recipe that will be incompatible so you want to crash the
build instead of detecting the error at runtime).
Therefore, the recipes themselves define all the compatibility report
information and how to do the comparison, allowing for "user supplied hook
scripts" which can be defined at any meta-layer level.
In the case you're hinting towards, you would need to implement the following
which can already hook into these patches:
1. a report generation function for the kernel recipe to output the data you
need to compare - i.e. a compatibility report (a directory of files) that
consists of the file checksums of the kernel headers stored in a text file
and the PV of the recipe stored in another text file - to do so the kernel
recipe merely needs to inherit compatible-depends-report and add a function
that outputs these report files and add this function to the prefuncs of the
task do_compatibility_report.
2. A python function to compare the different versions of the report format
suggested in (1).
3. Modifying the child recipes to add the kernel to CDEPENDS and point Bitbake
to the python function by setting the variable CDEPENDS_TOOL_xxx for the
kernel provider.
If you look at the patch 0009, you'll see the example is fairly simple but
similar to what I explained above. The python function added to the lib folder
is the actual compatibility comparison function and more can be easily
introduced in any meta-layer to match the compatibility checking situation. (If
interested, I can share another example compatibility checker that uses the PV
of a recipe to do rebuild decisions).
Happy to answer in more detail if you require, just let me know.
Thank you.
Kind regards,
Michael Ho
--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm
Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael.ho@bmw-carit.de
Web: http://www.bmw-carit.de
--------------------------------------------------------
BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--------------------------------------------------------
From: Darcy Watkins <dwatkins@sierrawireless.com>
Sent: 30 June 2017 17:01
To: Michael Ho
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] RFC: Backwards compatibility checking sstate-cache
Hi,
It would also be nice to add some sort of hook script support where user supplied hook script performs a check and returns either a "yes it's OK" or a "no have to rebuild it" status.
And then tie that into the dependency logic.
I have unusual use case where I need to have a dependency on changes to the kernel version, but not necessarily something triggered after every kernel rebuild. The package I'm building will trigger OpenSSL and a whole whack of stuff each time it rebuilds.
It depends on the kernel version and the headers, but shouldn't need to depend on every patch or kconfig change that happens.
___
Regards,
Darcy
Darcy Watkins :: Senior Staff Engineer, Firmware
SIERRA WIRELESS
Direct +1 604 233 7989 :: Fax +1 604 231 1109 :: Main +1 604 231 1100
13811 Wireless Way :: Richmond, BC Canada V6V 3A4
[M4]
dwatkins@sierrawireless.com :: www.sierrawireless.com :: www.inmotiontechnology.com
On Jun 30, 2017, at 2:53 AM, Michael Ho <Michael.Ho@bmw-carit.de> wrote:
Hi, at BMW Car IT we are working on an experimental feature to improve sstate
cache hits and we are looking for comments on the approach who might have some
insights to the problem and seeing if anyone is interested in the feature for
mainline.
The sstate-cache of a recipe is tied closely to its build dependencies, as the
signature generated for a task includes all parent task's signatures as part of
the signature information. This means that when changes occur in the parent
recipes, even if insignificant, all children recipes that have valid sstate
cache must invalidate their sstate cache objects.
What this patchset does is propose to add another optional variable to recipes,
CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature
generation it excludes any parent tasks that come from dependencies listed in
it. This is to break the signature change domino effect.
This patchset also proposes modifying RunQueue to then be able to run a
compatibility checker during task execution phase for recipes and tasks that use
CDEPENDS and allow for deciding to re-execute a task despite being covered by
sstate-cache.
The patchset is based on the jethro branch for the poky repository, as this is
the branch that we are using. If the general idea sounds good, we can consider
porting it to master.
Included is an patch that adds an example recipe and compatibility checker,
where compatibility is based on the file checksums of the parent recipes
packages. An example recipe, cdepends-test1, generates a compatibility report
containing the file checksums of all files that it packages and which file paths
they are at. Another recipe, cdepends-test2, can then strip this compatibility
report to the minimal files it needs to be consistent and can compare the latest
checksums it used to configure/compile/install with and if they have changed,
trigger a rebuild. If not, the previous version restored from sstate-cache is
used.
We are still experimenting with the usages of this, including the use of having
abi-compliance-checker to compare the ABI of shared libraries as a compatibility
checker during RunQueue and using the results to avoid rebuilding child recipes
when the .so files they depend on are still compatible. Example use cases of
this are allowing recipes which provide multiple shared libraries to change a
single .so file without rebuilding all its children that depend on the other
shared libraries but not the one that changed.
We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it didn't meet
the feature requirements of what this compatibility checker callback is doing,
although maybe when porting to master we could refactor to make better use of
the work already done there. The current implementation is a bit hacky but
comments would be appreciated in regards to if the concept is feasible and if
people are interested in making use of it and their use cases.
Kind regards,
Michael Ho
--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm
Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael.ho@bmw-carit.de
Web: http://www.bmw-carit.de
--------------------------------------------------------
BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--------------------------------------------------------
<0001-cache.py-add-support-for-CDEPENDS.patch>
<0002-siggen.py-add-support-for-CDEPENDS.patch>
<0003-runqueue.py-add-support-for-CDEPENDS.patch>
<0004-taskdata.py-add-support-for-CDEPENDS.patch>
<0005-rm_work.bbclass-add-exception-for-do_cache_compatibi.patch>
<0006-native.bbclass-add-support-for-CDEPENDS.patch>
<0007-nativesdk.bbclass-add-support-for-CDEPENDS.patch>
<0008-classes-add-bbclasses-compatible-depends-and-compati.patch>
<0009-recipes-cdepends-test-example-recipe-framework-for-u.patch>
--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: Backwards compatibility checking sstate-cache
2017-06-30 9:48 RFC: Backwards compatibility checking sstate-cache Michael Ho
2017-06-30 15:01 ` Darcy Watkins
@ 2017-07-01 7:48 ` Martin Jansa
2017-09-22 14:00 ` Mike Looijmans
2017-09-21 16:11 ` Richard Purdie
2 siblings, 1 reply; 9+ messages in thread
From: Martin Jansa @ 2017-07-01 7:48 UTC (permalink / raw)
To: Michael Ho; +Cc: yocto
[-- Attachment #1: Type: text/plain, Size: 4362 bytes --]
I haven't tried the patches, but I really like this idea (I was suggesting
something like that since 2011
http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/10327)
and I'm glad you weren't discouraged attempting to do this.
It also implements 3) b) idea from
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970 , you might be
interested read that ticket.
Thanks
On Fri, Jun 30, 2017 at 11:48 AM, Michael Ho <Michael.Ho@bmw-carit.de>
wrote:
> Hi, at BMW Car IT we are working on an experimental feature to improve
> sstate
> cache hits and we are looking for comments on the approach who might have
> some
> insights to the problem and seeing if anyone is interested in the feature
> for
> mainline.
>
> The sstate-cache of a recipe is tied closely to its build dependencies, as
> the
> signature generated for a task includes all parent task's signatures as
> part of
> the signature information. This means that when changes occur in the parent
> recipes, even if insignificant, all children recipes that have valid sstate
> cache must invalidate their sstate cache objects.
>
> What this patchset does is propose to add another optional variable to
> recipes,
> CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for
> signature
> generation it excludes any parent tasks that come from dependencies listed
> in
> it. This is to break the signature change domino effect.
>
> This patchset also proposes modifying RunQueue to then be able to run a
> compatibility checker during task execution phase for recipes and tasks
> that use
> CDEPENDS and allow for deciding to re-execute a task despite being covered
> by
> sstate-cache.
>
> The patchset is based on the jethro branch for the poky repository, as
> this is
> the branch that we are using. If the general idea sounds good, we can
> consider
> porting it to master.
>
> Included is an patch that adds an example recipe and compatibility checker,
> where compatibility is based on the file checksums of the parent recipes
> packages. An example recipe, cdepends-test1, generates a compatibility
> report
> containing the file checksums of all files that it packages and which file
> paths
> they are at. Another recipe, cdepends-test2, can then strip this
> compatibility
> report to the minimal files it needs to be consistent and can compare the
> latest
> checksums it used to configure/compile/install with and if they have
> changed,
> trigger a rebuild. If not, the previous version restored from sstate-cache
> is
> used.
>
> We are still experimenting with the usages of this, including the use of
> having
> abi-compliance-checker to compare the ABI of shared libraries as a
> compatibility
> checker during RunQueue and using the results to avoid rebuilding child
> recipes
> when the .so files they depend on are still compatible. Example use cases
> of
> this are allowing recipes which provide multiple shared libraries to
> change a
> single .so file without rebuilding all its children that depend on the
> other
> shared libraries but not the one that changed.
>
> We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it
> didn't meet
> the feature requirements of what this compatibility checker callback is
> doing,
> although maybe when porting to master we could refactor to make better use
> of
> the work already done there. The current implementation is a bit hacky but
> comments would be appreciated in regards to if the concept is feasible and
> if
> people are interested in making use of it and their use cases.
>
> Kind regards,
> Michael Ho
>
> --
> BMW Car IT GmbH
> Michael Ho
> Spezialist Entwicklung - Linux Software Integration
> Lise-Meitner-Str. 14
> 89081 Ulm
>
> Tel.: +49 731 3780 4071
> Mobil: +49 152 5498 0471
> Fax: +49-731-37804-001
> Mail: michael.ho@bmw-carit.de
> Web: http://www.bmw-carit.de
> --------------------------------------------------------
> BMW Car IT GmbH
> Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
> Sitz und Registergericht: München HRB 134810
> --------------------------------------------------------
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
>
[-- Attachment #2: Type: text/html, Size: 5418 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: Backwards compatibility checking sstate-cache
2017-06-30 9:48 RFC: Backwards compatibility checking sstate-cache Michael Ho
2017-06-30 15:01 ` Darcy Watkins
2017-07-01 7:48 ` Martin Jansa
@ 2017-09-21 16:11 ` Richard Purdie
2017-10-12 14:12 ` Michael Ho
2 siblings, 1 reply; 9+ messages in thread
From: Richard Purdie @ 2017-09-21 16:11 UTC (permalink / raw)
To: Michael Ho, yocto
On Fri, 2017-06-30 at 09:48 +0000, Michael Ho wrote:
> Hi, at BMW Car IT we are working on an experimental feature to
> improve sstate cache hits and we are looking for comments on the
> approach who might have some insights to the problem and seeing if
> anyone is interested in the feature for mainline.
Sorry I didn't see this before now but it was brought to my attention.
I have given this problem quite some thought off and on. I do worry
that the interface you propose is complex and requires changes
throughout the metadata and those changes impose "policy". One of our
strengths is that we're managed to keep "policy" out of the metadata.
In this context by policy, I mean when a recipe is equivalent and when
it is not.
I do have a counter-proposal of how we could solve this in a less
invasive way. This isn't implemented but wouldn't in theory be hard to
do.
The idea would be to have some kind of equivalence list. The first
built object's sstate checksum would become the definitive checksum and
the object added to the sstate cache.
If a new object is built, it would be compared with the one in sstate.
If deemed equivalent (by whatever policy), a mapping would be added to
the list. If not, there is no mapping and it becomes a new definitive
checksum.
All runqueue would then have to do is present its list of sstate
checksums to the list and convert them (in dependency order) into
canonical checksums.
This list could be a local equivalence file, or an equivalence server
which could resolve list of checksums.
Doing it this way totally isolates the comparison from the metadata
itself and in my view protects us from future changes in definition of
equivalence.
How does that sound?
Cheers,
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: Backwards compatibility checking sstate-cache
2017-07-01 7:48 ` Martin Jansa
@ 2017-09-22 14:00 ` Mike Looijmans
2017-09-22 22:51 ` Joshua Lock
0 siblings, 1 reply; 9+ messages in thread
From: Mike Looijmans @ 2017-09-22 14:00 UTC (permalink / raw)
To: Martin Jansa, Michael Ho; +Cc: yocto
I think this remark in the referenced link is the best summary of "what could
be improved":
"""the biggest weakness of the sstate signature bits, in my opinion, is that
it only tracks inputs, not outputs. If task A depends on B, and the metadata
input to B changes, then A will be rebuilt, even if the *output* of B didn't
change as a result of the change to its metadata."""
For example, if someone fixes a bug in gcc that only applies to MIPS, then
builds that only target an ARM machine shouldn't be affected. Much worse than
that, I have a dependency like:
gcc -> ... -> python -> bitstream tool -> FPGA image
so this means that a change in gcc causes Python to rebuild, which causes the
bitstream tool to be rebuilt and produce idential output, and that triggers a
roughly 31-hour build of lots of FPGA images. These are the ones we really
want to break. A binary output compare would help a lot, even if 80% of the
libraries fail to create reproducible output, I may still be spared those 31
hours...
I think tracking digital output is technically feasible, and won't require any
change to existing recipes.
Also think about "feed" setups. When I see my settop reporting "331 packages
need updating"... It'd be great if that could be avoided.
On 01-07-17 09:48, Martin Jansa wrote:
> I haven't tried the patches, but I really like this idea (I was suggesting
> something like that since 2011
> http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/10327) and
> I'm glad you weren't discouraged attempting to do this.
>
> It also implements 3) b) idea from
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970 , you might be
> interested read that ticket.
>
> Thanks
>
> On Fri, Jun 30, 2017 at 11:48 AM, Michael Ho <Michael.Ho@bmw-carit.de
> <mailto:Michael.Ho@bmw-carit.de>> wrote:
>
> Hi, at BMW Car IT we are working on an experimental feature to improve sstate
> cache hits and we are looking for comments on the approach who might have some
> insights to the problem and seeing if anyone is interested in the feature for
> mainline.
>
> The sstate-cache of a recipe is tied closely to its build dependencies, as the
> signature generated for a task includes all parent task's signatures as
> part of
> the signature information. This means that when changes occur in the parent
> recipes, even if insignificant, all children recipes that have valid sstate
> cache must invalidate their sstate cache objects.
>
> What this patchset does is propose to add another optional variable to
> recipes,
> CDEPENDS, which acts like DEPENDS for all RunQueue purposes but for signature
> generation it excludes any parent tasks that come from dependencies listed in
> it. This is to break the signature change domino effect.
>
> This patchset also proposes modifying RunQueue to then be able to run a
> compatibility checker during task execution phase for recipes and tasks
> that use
> CDEPENDS and allow for deciding to re-execute a task despite being covered by
> sstate-cache.
>
> The patchset is based on the jethro branch for the poky repository, as this is
> the branch that we are using. If the general idea sounds good, we can
> consider
> porting it to master.
>
> Included is an patch that adds an example recipe and compatibility checker,
> where compatibility is based on the file checksums of the parent recipes
> packages. An example recipe, cdepends-test1, generates a compatibility report
> containing the file checksums of all files that it packages and which file
> paths
> they are at. Another recipe, cdepends-test2, can then strip this compatibility
> report to the minimal files it needs to be consistent and can compare the
> latest
> checksums it used to configure/compile/install with and if they have changed,
> trigger a rebuild. If not, the previous version restored from sstate-cache is
> used.
>
> We are still experimenting with the usages of this, including the use of
> having
> abi-compliance-checker to compare the ABI of shared libraries as a
> compatibility
> checker during RunQueue and using the results to avoid rebuilding child
> recipes
> when the .so files they depend on are still compatible. Example use cases of
> this are allowing recipes which provide multiple shared libraries to change a
> single .so file without rebuilding all its children that depend on the other
> shared libraries but not the one that changed.
>
> We're aware of the SIGGEN_EXCLUDERECIPES_ABISAFE feature but feel it
> didn't meet
> the feature requirements of what this compatibility checker callback is doing,
> although maybe when porting to master we could refactor to make better use of
> the work already done there. The current implementation is a bit hacky but
> comments would be appreciated in regards to if the concept is feasible and if
> people are interested in making use of it and their use cases.
>
> Kind regards,
> Michael Ho
>
> --
> BMW Car IT GmbH
> Michael Ho
> Spezialist Entwicklung - Linux Software Integration
> Lise-Meitner-Str. 14
> 89081 Ulm
>
> Tel.: +49 731 3780 4071 <tel:%2B49%20731%203780%204071>
> Mobil: +49 152 5498 0471 <tel:%2B49%20152%205498%200471>
> Fax: +49-731-37804-001 <tel:%2B49-731-37804-001>
> Mail: michael.ho@bmw-carit.de <mailto:michael.ho@bmw-carit.de>
> Web: http://www.bmw-carit.de
> --------------------------------------------------------
> BMW Car IT GmbH
> Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
> Sitz und Registergericht: München HRB 134810
> --------------------------------------------------------
>
> --
>
Kind regards,
Mike Looijmans
System Expert
TOPIC Products
Materiaalweg 4, NL-5681 RJ Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com
Please consider the environment before printing this e-mail
Visit us at the Space Tech Expo Europe (Stand E-71)
_______________________________________________
> yocto mailing list
> yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> https://lists.yoctoproject.org/listinfo/yocto
> <https://lists.yoctoproject.org/listinfo/yocto>
>
>
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: Backwards compatibility checking sstate-cache
2017-09-22 14:00 ` Mike Looijmans
@ 2017-09-22 22:51 ` Joshua Lock
2017-09-25 11:47 ` Mike Looijmans
0 siblings, 1 reply; 9+ messages in thread
From: Joshua Lock @ 2017-09-22 22:51 UTC (permalink / raw)
To: Mike Looijmans, Martin Jansa, Michael Ho; +Cc: yocto
On 22/09/17 15:00, Mike Looijmans wrote:
> I think this remark in the referenced link is the best summary of "what
> could be improved":
>
> """the biggest weakness of the sstate signature bits, in my opinion, is
> that it only tracks inputs, not outputs. If task A depends on B, and the
> metadata input to B changes, then A will be rebuilt, even if the
> *output* of B didn't change as a result of the change to its metadata."""
>
> For example, if someone fixes a bug in gcc that only applies to MIPS,
> then builds that only target an ARM machine shouldn't be affected. Much
> worse than that, I have a dependency like:
> gcc -> ... -> python -> bitstream tool -> FPGA image
>
> so this means that a change in gcc causes Python to rebuild, which
> causes the bitstream tool to be rebuilt and produce idential output, and
> that triggers a roughly 31-hour build of lots of FPGA images. These are
> the ones we really want to break. A binary output compare would help a
> lot, even if 80% of the libraries fail to create reproducible output, I
> may still be spared those 31 hours...
>
> I think tracking digital output is technically feasible, and won't
> require any change to existing recipes.
>
> Also think about "feed" setups. When I see my settop reporting "331
> packages need updating"... It'd be great if that could be avoided.
>
That's what packagefeed-stability.bbclass is for. Work on binary
reproducibility will improve things here too.
Joshua
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: Backwards compatibility checking sstate-cache
2017-09-22 22:51 ` Joshua Lock
@ 2017-09-25 11:47 ` Mike Looijmans
0 siblings, 0 replies; 9+ messages in thread
From: Mike Looijmans @ 2017-09-25 11:47 UTC (permalink / raw)
To: Joshua Lock, Martin Jansa, Michael Ho; +Cc: yocto
On 23-09-17 00:51, Joshua Lock wrote:
> On 22/09/17 15:00, Mike Looijmans wrote:
>> I think this remark in the referenced link is the best summary of "what
>> could be improved":
>>
>> """the biggest weakness of the sstate signature bits, in my opinion, is that
>> it only tracks inputs, not outputs. If task A depends on B, and the metadata
>> input to B changes, then A will be rebuilt, even if the *output* of B didn't
>> change as a result of the change to its metadata."""
>>
>> For example, if someone fixes a bug in gcc that only applies to MIPS, then
>> builds that only target an ARM machine shouldn't be affected. Much worse
>> than that, I have a dependency like:
>> gcc -> ... -> python -> bitstream tool -> FPGA image
>>
>> so this means that a change in gcc causes Python to rebuild, which causes
>> the bitstream tool to be rebuilt and produce idential output, and that
>> triggers a roughly 31-hour build of lots of FPGA images. These are the ones
>> we really want to break. A binary output compare would help a lot, even if
>> 80% of the libraries fail to create reproducible output, I may still be
>> spared those 31 hours...
>>
>> I think tracking digital output is technically feasible, and won't require
>> any change to existing recipes.
>>
>> Also think about "feed" setups. When I see my settop reporting "331 packages
>> need updating"... It'd be great if that could be avoided.
>>
>
> That's what packagefeed-stability.bbclass is for. Work on binary
> reproducibility will improve things here too.
Interesting, thanks for the hint.
That might cut down a bit on the over 4TB/month "update my box" traffic.
Kind regards,
Mike Looijmans
System Expert
TOPIC Products
Materiaalweg 4, NL-5681 RJ Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com
Please consider the environment before printing this e-mail
Visit us at the Space Tech Expo Europe (Stand E-71)
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RFC: Backwards compatibility checking sstate-cache
2017-09-21 16:11 ` Richard Purdie
@ 2017-10-12 14:12 ` Michael Ho
0 siblings, 0 replies; 9+ messages in thread
From: Michael Ho @ 2017-10-12 14:12 UTC (permalink / raw)
To: Richard Purdie, yocto, mike.looijmans, twoerner, martin.jansa
Hi Richard (and all),
@Richard: Thanks for the input regarding the topic and apologies for not replying until now, I was a bit buried under workload and simultaneously completing my Masters studies. I also wanted to take some time to digest your comments amid working on this CDEPENDS concept, and I just want to say that I'm very open to finding another way to implement this concept of rebuild avoidance in Yocto.
I think the idea of a non-invasive approach can be a bit tricky, but your idea of a secondary sstate signature and mapping table sounds workable to me. (In fact, as I slowly developed out this concept, I think it's almost required). One point though with the removal of the metadata changes is that the view of "what is compatible" will be concrete and cannot differ among children. Sometimes you have only specific children of a recipe that can handle compatibility changes (and at different levels of compatibility) while others cannot handle any change at all, and that is why I developed with the metadata changes as a basis. But definitely, yes, the change to metadata makes things more complex and does in a sense impose policy, which I understand would be undesirable. Maybe we can figure out an approach that avoids metadata changes but keeps this ability to differ compatibility handling among children.
Currently I have a working proof of concept (I've developed out the earlier shared patches further) that I'd like to share soon and demonstrate. It still needs some further refinement but I think it's showing some interesting results already. I'll think further about the topic and see what I can come up with that matches your line of thoughts.
@Martin Jansa: Thank you for the comments and information, it's good to know that there's some interest from the community to look into this idea.
@Mike Looijmans: Yes, that's essentially the problem I want to resolve with this proof of concept, breaking long build chains at points where a clear precision cut could've been made depending on whether a rebuild is actually required or not (i.e. the bitstream tool not changing). In that case I would even use the tool version and assume even if the tool binary has changed, that the output result of tool would stay the same so it is compatible in essence (and avoid rebuilding 31 hours of stuff).
@Trevor Woerner: Thanks for the suggestion, I was planning to cross-post when I had a more stable proof of concept code to share.
Note: I'll be following my colleague, Mario Domenech Goulart (mario-goulart), to OEDEM 2017 where I hope to discuss the current concept code and show some potential use cases.
Kind regards,
Michael Ho
--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung - Linux Software Integration
Lise-Meitner-Str. 14
89081 Ulm
Tel.: +49 731 3780 4071
Mobil: +49 152 5498 0471
Fax: +49-731-37804-001
Mail: michael.ho@bmw-carit.de
Web: http://www.bmw-carit.de
--------------------------------------------------------
BMW Car IT GmbH
Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin
Sitz und Registergericht: München HRB 134810
--------------------------------------------------------
________________________________________
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Sent: 21 September 2017 18:11
To: Michael Ho; yocto@yoctoproject.org
Subject: Re: [yocto] RFC: Backwards compatibility checking sstate-cache
On Fri, 2017-06-30 at 09:48 +0000, Michael Ho wrote:
> Hi, at BMW Car IT we are working on an experimental feature to
> improve sstate cache hits and we are looking for comments on the
> approach who might have some insights to the problem and seeing if
> anyone is interested in the feature for mainline.
Sorry I didn't see this before now but it was brought to my attention.
I have given this problem quite some thought off and on. I do worry
that the interface you propose is complex and requires changes
throughout the metadata and those changes impose "policy". One of our
strengths is that we're managed to keep "policy" out of the metadata.
In this context by policy, I mean when a recipe is equivalent and when
it is not.
I do have a counter-proposal of how we could solve this in a less
invasive way. This isn't implemented but wouldn't in theory be hard to
do.
The idea would be to have some kind of equivalence list. The first
built object's sstate checksum would become the definitive checksum and
the object added to the sstate cache.
If a new object is built, it would be compared with the one in sstate.
If deemed equivalent (by whatever policy), a mapping would be added to
the list. If not, there is no mapping and it becomes a new definitive
checksum.
All runqueue would then have to do is present its list of sstate
checksums to the list and convert them (in dependency order) into
canonical checksums.
This list could be a local equivalence file, or an equivalence server
which could resolve list of checksums.
Doing it this way totally isolates the comparison from the metadata
itself and in my view protects us from future changes in definition of
equivalence.
How does that sound?
Cheers,
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2017-10-12 14:15 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-30 9:48 RFC: Backwards compatibility checking sstate-cache Michael Ho
2017-06-30 15:01 ` Darcy Watkins
2017-06-30 22:44 ` Michael Ho
2017-07-01 7:48 ` Martin Jansa
2017-09-22 14:00 ` Mike Looijmans
2017-09-22 22:51 ` Joshua Lock
2017-09-25 11:47 ` Mike Looijmans
2017-09-21 16:11 ` Richard Purdie
2017-10-12 14:12 ` Michael Ho
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.