* [PATCH 00/12] Tinfoil enhancements / fetch fixes
@ 2017-07-19 9:55 Paul Eggleton
2017-07-19 9:56 ` [PATCH 01/12] tinfoil: add internal mode to build_file() function Paul Eggleton
` (11 more replies)
0 siblings, 12 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:55 UTC (permalink / raw)
To: bitbake-devel
Some enhancements to tinfoil around improving OE's devtool / recipetool
fetch code, as well as general enhancements to the API.
The following changes since commit 6683338598ce97278f188fbcd780c3e3754e5b9a:
fetch2: fix indentation (2017-07-18 22:38:30 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib paule/tinfoil-fixes-bb3
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/tinfoil-fixes-bb3
Paul Eggleton (12):
tinfoil: add internal mode to build_file() function
tinfoil: add a parse_recipes() function
tinfoil: set a flag when recipes have been parsed
lib/bb/event: refactor printing events
lib/bb/ui/uihelper: indicate to caller of eventHandler() if events handled
knotty: make it possible to use termfilter without either console
tinfoil: add functionality for running full builds
tinfoil: enable access to additional cached items
tinfoil: add simple API for getting cached recipe information
tinfoil: add more doc comments
fetch2: allow hiding checksum warning
fetch2: fire an event when there are missing checksums
bin/bitbake-layers | 2 +-
lib/bb/command.py | 43 ++++-
lib/bb/cooker.py | 34 ++--
lib/bb/event.py | 32 ++++
lib/bb/fetch2/__init__.py | 16 ++
lib/bb/runqueue.py | 6 +
lib/bb/tinfoil.py | 389 +++++++++++++++++++++++++++++++++++++++++++++-
lib/bb/ui/knotty.py | 43 ++---
lib/bb/ui/ncurses.py | 2 +-
lib/bb/ui/taskexp.py | 20 +--
lib/bb/ui/toasterui.py | 25 +--
lib/bb/ui/uihelper.py | 3 +
12 files changed, 525 insertions(+), 90 deletions(-)
--
2.9.4
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 01/12] tinfoil: add internal mode to build_file() function
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 02/12] tinfoil: add a parse_recipes() function Paul Eggleton
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
In OE's devtool we want to repeatedly run build_file() without showing
unnecessary messages and triggering buildhistory for each call.
build_file() is just a wrapper around the buildFile command. Change
the final "hidewarning" parameter of the buildFile command to "internal"
and have this call a new buildFileInternal() function without triggering
any of the normal build events, silencing the normal info messages from
the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and
avoiding calling parseConfiguration() which we've already done at this
point.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/command.py | 9 ++++++---
lib/bb/cooker.py | 34 +++++++++++++++++++++++++---------
lib/bb/tinfoil.py | 9 ++++++---
3 files changed, 37 insertions(+), 15 deletions(-)
diff --git a/lib/bb/command.py b/lib/bb/command.py
index 36891b9..2c8223e 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -543,11 +543,14 @@ class CommandsAsync:
bfile = params[0]
task = params[1]
if len(params) > 2:
- hidewarning = params[2]
+ internal = params[2]
else:
- hidewarning = False
+ internal = False
- command.cooker.buildFile(bfile, task, hidewarning)
+ if internal:
+ command.cooker.buildFileInternal(bfile, task, fireevents=False, quietlog=True)
+ else:
+ command.cooker.buildFile(bfile, task)
buildFile.needcache = False
def buildTargets(self, command, params):
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index b1311bb..9041594 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1232,21 +1232,27 @@ class BBCooker:
raise NoSpecificMatch
return matches[0]
- def buildFile(self, buildfile, task, hidewarning=False):
+ def buildFile(self, buildfile, task):
"""
Build the file matching regexp buildfile
"""
bb.event.fire(bb.event.BuildInit(), self.data)
- if not hidewarning:
- # Too many people use -b because they think it's how you normally
- # specify a target to be built, so show a warning
- bb.warn("Buildfile specified, dependencies will not be handled. If this is not what you want, do not use -b / --buildfile.")
+ # Too many people use -b because they think it's how you normally
+ # specify a target to be built, so show a warning
+ bb.warn("Buildfile specified, dependencies will not be handled. If this is not what you want, do not use -b / --buildfile.")
# Parse the configuration here. We need to do it explicitly here since
# buildFile() doesn't use the cache
self.parseConfiguration()
+ self.buildFileInternal(buildfile, task)
+
+ def buildFileInternal(self, buildfile, task, fireevents=True, quietlog=False):
+ """
+ Build the file matching regexp buildfile
+ """
+
# If we are told to do the None task then query the default task
if (task == None):
task = self.configuration.cmd
@@ -1283,8 +1289,8 @@ class BBCooker:
# Remove external dependencies
self.recipecaches[mc].task_deps[fn]['depends'] = {}
self.recipecaches[mc].deps[fn] = []
- self.recipecaches[mc].rundeps[fn] = []
- self.recipecaches[mc].runrecs[fn] = []
+ self.recipecaches[mc].rundeps[fn] = defaultdict(list)
+ self.recipecaches[mc].runrecs[fn] = defaultdict(list)
# Invalidate task for target if force mode active
if self.configuration.force:
@@ -1296,8 +1302,13 @@ class BBCooker:
taskdata[mc] = bb.taskdata.TaskData(self.configuration.abort)
taskdata[mc].add_provider(self.databuilder.mcdata[mc], self.recipecaches[mc], item)
+ if quietlog:
+ rqloglevel = bb.runqueue.logger.getEffectiveLevel()
+ bb.runqueue.logger.setLevel(logging.WARNING)
+
buildname = self.databuilder.mcdata[mc].getVar("BUILDNAME")
- bb.event.fire(bb.event.BuildStarted(buildname, [item]), self.databuilder.mcdata[mc])
+ if fireevents:
+ bb.event.fire(bb.event.BuildStarted(buildname, [item]), self.databuilder.mcdata[mc])
# Execute the runqueue
runlist = [[mc, item, task, fn]]
@@ -1324,11 +1335,16 @@ class BBCooker:
retval = False
except SystemExit as exc:
self.command.finishAsyncCommand(str(exc))
+ if quietlog:
+ bb.runqueue.logger.setLevel(rqloglevel)
return False
if not retval:
- bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, item, failures, interrupted), self.databuilder.mcdata[mc])
+ if fireevents:
+ bb.event.fire(bb.event.BuildCompleted(len(rq.rqdata.runtaskentries), buildname, item, failures, interrupted), self.databuilder.mcdata[mc])
self.command.finishAsyncCommand(msg)
+ if quietlog:
+ bb.runqueue.logger.setLevel(rqloglevel)
return False
if retval is True:
return True
diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index fb0da62..b45acea 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -415,12 +415,15 @@ class Tinfoil:
else:
return None
- def build_file(self, buildfile, task):
+ def build_file(self, buildfile, task, internal=True):
"""
Runs the specified task for just a single recipe (i.e. no dependencies).
- This is equivalent to bitbake -b, except no warning will be printed.
+ This is equivalent to bitbake -b, except with the default internal=True
+ no warning about dependencies will be produced, normal info messages
+ from the runqueue will be silenced and BuildInit, BuildStarted and
+ BuildCompleted events will not be fired.
"""
- return self.run_command('buildFile', buildfile, task, True)
+ return self.run_command('buildFile', buildfile, task, internal)
def shutdown(self):
if self.server_connection:
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 02/12] tinfoil: add a parse_recipes() function
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
2017-07-19 9:56 ` [PATCH 01/12] tinfoil: add internal mode to build_file() function Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 03/12] tinfoil: set a flag when recipes have been parsed Paul Eggleton
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
Python style recommends underscore based naming rather than camelCase,
and thus the former has been used for most of tinfoil's functions. Add
an underscored version of parseRecipes() for consistency and change the
one place we call it to use the new version.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
bin/bitbake-layers | 2 +-
lib/bb/tinfoil.py | 8 +++++++-
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/bin/bitbake-layers b/bin/bitbake-layers
index 04e6bec..d184011 100755
--- a/bin/bitbake-layers
+++ b/bin/bitbake-layers
@@ -90,7 +90,7 @@ def main():
if getattr(args, 'parserecipes', False):
tinfoil.config_data.disableTracking()
- tinfoil.parseRecipes()
+ tinfoil.parse_recipes()
tinfoil.config_data.enableTracking()
return args.func(args)
diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index b45acea..e027930 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -289,7 +289,13 @@ class Tinfoil:
def parseRecipes(self):
"""
- Force a parse of all recipes. Normally you should specify
+ Legacy function - use parse_recipes() instead.
+ """
+ self.parse_recipes()
+
+ def parse_recipes(self):
+ """
+ Load information on all recipes. Normally you should specify
config_only=False when calling prepare() instead of using this
function; this function is designed for situations where you need
to initialise Tinfoil and use it with config_only=True first and
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 03/12] tinfoil: set a flag when recipes have been parsed
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
2017-07-19 9:56 ` [PATCH 01/12] tinfoil: add internal mode to build_file() function Paul Eggleton
2017-07-19 9:56 ` [PATCH 02/12] tinfoil: add a parse_recipes() function Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 04/12] lib/bb/event: refactor printing events Paul Eggleton
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
Make it easy to determine if recipes are parsed (and thus information
about available recipes is in memory).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/tinfoil.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index e027930..cba157c 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -217,6 +217,7 @@ class Tinfoil:
self.tracking = tracking
self.ui_module = None
self.server_connection = None
+ self.recipes_parsed = False
if setup_logging:
# This is the *client-side* logger, nothing to do with
# logging messages from the server
@@ -270,6 +271,7 @@ class Tinfoil:
self.run_command('parseConfiguration')
else:
self.run_actions(config_params)
+ self.recipes_parsed = True
self.config_data = bb.data.init()
connector = TinfoilDataStoreConnector(self, None)
@@ -303,6 +305,7 @@ class Tinfoil:
"""
config_params = TinfoilConfigParameters(config_only=False)
self.run_actions(config_params)
+ self.recipes_parsed = True
def run_command(self, command, *params):
"""
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 04/12] lib/bb/event: refactor printing events
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
` (2 preceding siblings ...)
2017-07-19 9:56 ` [PATCH 03/12] tinfoil: set a flag when recipes have been parsed Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 05/12] lib/bb/ui/uihelper: indicate to caller of eventHandler() if events handled Paul Eggleton
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/command.py | 2 ++
lib/bb/event.py | 32 ++++++++++++++++++++++++++++++++
lib/bb/runqueue.py | 6 ++++++
lib/bb/ui/knotty.py | 37 ++++++-------------------------------
lib/bb/ui/ncurses.py | 2 +-
lib/bb/ui/taskexp.py | 20 ++------------------
lib/bb/ui/toasterui.py | 25 ++++---------------------
7 files changed, 53 insertions(+), 71 deletions(-)
diff --git a/lib/bb/command.py b/lib/bb/command.py
index 2c8223e..01a2c61 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -50,6 +50,8 @@ class CommandFailed(CommandExit):
def __init__(self, message):
self.error = message
CommandExit.__init__(self, 1)
+ def __str__(self):
+ return "Command execution failed: %s" % self.error
class CommandError(Exception):
pass
diff --git a/lib/bb/event.py b/lib/bb/event.py
index d5c5ef3..b2d0d1a 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -516,6 +516,28 @@ class NoProvider(Event):
def isRuntime(self):
return self._runtime
+ def __str__(self):
+ msg = ''
+ if self._runtime:
+ r = "R"
+ else:
+ r = ""
+
+ extra = ''
+ if not self._reasons:
+ if self._close_matches:
+ extra = ". Close matches:\n %s" % '\n '.join(self._close_matches)
+
+ if self._dependees:
+ msg = "Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)%s" % (r, self._item, ", ".join(self._dependees), r, extra)
+ else:
+ msg = "Nothing %sPROVIDES '%s'%s" % (r, self._item, extra)
+ if self._reasons:
+ for reason in self._reasons:
+ msg += '\n' + reason
+ return msg
+
+
class MultipleProviders(Event):
"""Multiple Providers"""
@@ -543,6 +565,16 @@ class MultipleProviders(Event):
"""
return self._candidates
+ def __str__(self):
+ msg = "Multiple providers are available for %s%s (%s)" % (self._is_runtime and "runtime " or "",
+ self._item,
+ ", ".join(self._candidates))
+ rtime = ""
+ if self._is_runtime:
+ rtime = "R"
+ msg += "\nConsider defining a PREFERRED_%sPROVIDER entry to match %s" % (rtime, self._item)
+ return msg
+
class ParseStarted(OperationStarted):
"""Recipe parsing for the runqueue has begun"""
def __init__(self, total):
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 21520d3..7dd964d 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2488,6 +2488,9 @@ class runQueueTaskFailed(runQueueEvent):
runQueueEvent.__init__(self, task, stats, rq)
self.exitcode = exitcode
+ def __str__(self):
+ return "Task (%s) failed with exit code '%s'" % (self.taskstring, self.exitcode)
+
class sceneQueueTaskFailed(sceneQueueEvent):
"""
Event notifying a setscene task failed
@@ -2496,6 +2499,9 @@ class sceneQueueTaskFailed(sceneQueueEvent):
sceneQueueEvent.__init__(self, task, stats, rq)
self.exitcode = exitcode
+ def __str__(self):
+ return "Setscene task (%s) failed with exit code '%s' - real task will be run instead" % (self.taskstring, self.exitcode)
+
class sceneQueueComplete(sceneQueueEvent):
"""
Event when all the sceneQueue tasks are complete
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 1aa5ebb..c301982 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -560,7 +560,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
return_value = event.exitcode
if event.error:
errors = errors + 1
- logger.error("Command execution failed: %s", event.error)
+ logger.error(str(event))
main.shutdown = 2
continue
if isinstance(event, bb.command.CommandExit):
@@ -571,39 +571,16 @@ def main(server, eventHandler, params, tf = TerminalFilter):
main.shutdown = 2
continue
if isinstance(event, bb.event.MultipleProviders):
- logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
- event._item,
- ", ".join(event._candidates))
- rtime = ""
- if event._is_runtime:
- rtime = "R"
- logger.info("consider defining a PREFERRED_%sPROVIDER entry to match %s" % (rtime, event._item))
+ logger.info(str(event))
continue
if isinstance(event, bb.event.NoProvider):
- if event._runtime:
- r = "R"
- else:
- r = ""
-
- extra = ''
- if not event._reasons:
- if event._close_matches:
- extra = ". Close matches:\n %s" % '\n '.join(event._close_matches)
-
# For universe builds, only show these as warnings, not errors
- h = logger.warning
if not universe:
return_value = 1
errors = errors + 1
- h = logger.error
-
- if event._dependees:
- h("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)%s", r, event._item, ", ".join(event._dependees), r, extra)
+ logger.error(str(event))
else:
- h("Nothing %sPROVIDES '%s'%s", r, event._item, extra)
- if event._reasons:
- for reason in event._reasons:
- h("%s", reason)
+ logger.warning(str(event))
continue
if isinstance(event, bb.runqueue.sceneQueueTaskStarted):
@@ -625,13 +602,11 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if isinstance(event, bb.runqueue.runQueueTaskFailed):
return_value = 1
taskfailures.append(event.taskstring)
- logger.error("Task (%s) failed with exit code '%s'",
- event.taskstring, event.exitcode)
+ logger.error(str(event))
continue
if isinstance(event, bb.runqueue.sceneQueueTaskFailed):
- logger.warning("Setscene task (%s) failed with exit code '%s' - real task will be run instead",
- event.taskstring, event.exitcode)
+ logger.warning(str(event))
continue
if isinstance(event, bb.event.DepTreeGenerated):
diff --git a/lib/bb/ui/ncurses.py b/lib/bb/ui/ncurses.py
index ca845a3..8690c52 100644
--- a/lib/bb/ui/ncurses.py
+++ b/lib/bb/ui/ncurses.py
@@ -315,7 +315,7 @@ class NCursesUI:
# also allow them to now exit with a single ^C
shutdown = 2
if isinstance(event, bb.command.CommandFailed):
- mw.appendText("Command execution failed: %s" % event.error)
+ mw.appendText(str(event))
time.sleep(2)
exitflag = True
if isinstance(event, bb.command.CommandExit):
diff --git a/lib/bb/ui/taskexp.py b/lib/bb/ui/taskexp.py
index 9d14ece..0d0f9f5 100644
--- a/lib/bb/ui/taskexp.py
+++ b/lib/bb/ui/taskexp.py
@@ -286,23 +286,7 @@ def main(server, eventHandler, params):
continue
if isinstance(event, bb.event.NoProvider):
- if event._runtime:
- r = "R"
- else:
- r = ""
-
- extra = ''
- if not event._reasons:
- if event._close_matches:
- extra = ". Close matches:\n %s" % '\n '.join(event._close_matches)
-
- if event._dependees:
- print("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)%s" % (r, event._item, ", ".join(event._dependees), r, extra))
- else:
- print("Nothing %sPROVIDES '%s'%s" % (r, event._item, extra))
- if event._reasons:
- for reason in event._reasons:
- print(reason)
+ print(str(event))
_, error = server.runCommand(["stateShutdown"])
if error:
@@ -310,7 +294,7 @@ def main(server, eventHandler, params):
break
if isinstance(event, bb.command.CommandFailed):
- print("Command execution failed: %s" % event.error)
+ print(str(event))
return event.exitcode
if isinstance(event, bb.command.CommandExit):
diff --git a/lib/bb/ui/toasterui.py b/lib/bb/ui/toasterui.py
index 71f04fa..88cec37 100644
--- a/lib/bb/ui/toasterui.py
+++ b/lib/bb/ui/toasterui.py
@@ -320,29 +320,13 @@ def main(server, eventHandler, params):
if isinstance(event, bb.event.CacheLoadCompleted):
continue
if isinstance(event, bb.event.MultipleProviders):
- logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
- event._item,
- ", ".join(event._candidates))
- logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
+ logger.info(str(event))
continue
if isinstance(event, bb.event.NoProvider):
errors = errors + 1
- if event._runtime:
- r = "R"
- else:
- r = ""
-
- if event._dependees:
- text = "Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)" % (r, event._item, ", ".join(event._dependees), r)
- else:
- text = "Nothing %sPROVIDES '%s'" % (r, event._item)
-
+ text = str(event)
logger.error(text)
- if event._reasons:
- for reason in event._reasons:
- logger.error("%s", reason)
- text += reason
buildinfohelper.store_log_error(text)
continue
@@ -364,8 +348,7 @@ def main(server, eventHandler, params):
if isinstance(event, bb.runqueue.runQueueTaskFailed):
buildinfohelper.update_and_store_task(event)
taskfailures.append(event.taskstring)
- logger.error("Task (%s) failed with exit code '%s'",
- event.taskstring, event.exitcode)
+ logger.error(str(event))
continue
if isinstance(event, (bb.runqueue.sceneQueueTaskCompleted, bb.runqueue.sceneQueueTaskFailed)):
@@ -382,7 +365,7 @@ def main(server, eventHandler, params):
if isinstance(event, bb.command.CommandFailed):
errors += 1
errorcode = 1
- logger.error("Command execution failed: %s", event.error)
+ logger.error(str(event))
elif isinstance(event, bb.event.BuildCompleted):
buildinfohelper.scan_image_artifacts()
buildinfohelper.clone_required_sdk_artifacts()
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 05/12] lib/bb/ui/uihelper: indicate to caller of eventHandler() if events handled
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
` (3 preceding siblings ...)
2017-07-19 9:56 ` [PATCH 04/12] lib/bb/event: refactor printing events Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 06/12] knotty: make it possible to use termfilter without either console Paul Eggleton
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
It is useful for the caller to know whether the uihelper has handled the
event passed so that it can skip other event handling code if so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/ui/uihelper.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/bb/ui/uihelper.py b/lib/bb/ui/uihelper.py
index 113fced..963c1ea 100644
--- a/lib/bb/ui/uihelper.py
+++ b/lib/bb/ui/uihelper.py
@@ -61,6 +61,9 @@ class BBUIHelper:
self.running_tasks[event.pid]['progress'] = event.progress
self.running_tasks[event.pid]['rate'] = event.rate
self.needUpdate = True
+ else:
+ return False
+ return True
def getTasks(self):
self.needUpdate = False
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 06/12] knotty: make it possible to use termfilter without either console
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
` (4 preceding siblings ...)
2017-07-19 9:56 ` [PATCH 05/12] lib/bb/ui/uihelper: indicate to caller of eventHandler() if events handled Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 07/12] tinfoil: add functionality for running full builds Paul Eggleton
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
This isn't useful for knotty itself, but for use from tinfoil
in case we can't get access to either the console or errconsole, allow
either to be unspecified (None).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/ui/knotty.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index c301982..f3900bd 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -207,8 +207,10 @@ class TerminalFilter(object):
self.interactive = False
bb.note("Unable to use interactive mode for this terminal, using fallback")
return
- console.addFilter(InteractConsoleLogFilter(self, format))
- errconsole.addFilter(InteractConsoleLogFilter(self, format))
+ if console:
+ console.addFilter(InteractConsoleLogFilter(self, format))
+ if errconsole:
+ errconsole.addFilter(InteractConsoleLogFilter(self, format))
self.main_progress = None
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 07/12] tinfoil: add functionality for running full builds
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
` (5 preceding siblings ...)
2017-07-19 9:56 ` [PATCH 06/12] knotty: make it possible to use termfilter without either console Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 08/12] tinfoil: enable access to additional cached items Paul Eggleton
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
Up to this point, if you wanted to run build tasks in the normal way
they get run from a python script, there was no other way than to shell
out to bitbake. Worse than that, you couldn't have tinfoil active during
that because only one bitbake instance could be running at once. As long
as we're prepared to handle the events produced, we can create a wrapper
around calling the buildTargets command. Borrow code from knotty to do
this in such a way that we get the expected running task display
(courtesy of TermFilter) and Ctrl+C handling.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/tinfoil.py | 154 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 154 insertions(+)
diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index cba157c..580ce8d 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -2,6 +2,7 @@
#
# Copyright (C) 2012-2017 Intel Corporation
# Copyright (C) 2011 Mentor Graphics Corporation
+# Copyright (C) 2006-2012 Richard Purdie
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
@@ -218,6 +219,7 @@ class Tinfoil:
self.ui_module = None
self.server_connection = None
self.recipes_parsed = False
+ self.quiet = 0
if setup_logging:
# This is the *client-side* logger, nothing to do with
# logging messages from the server
@@ -230,6 +232,8 @@ class Tinfoil:
self.shutdown()
def prepare(self, config_only=False, config_params=None, quiet=0, extra_features=None):
+ self.quiet = quiet
+
if self.tracking:
extrafeatures = [bb.cooker.CookerFeatures.BASEDATASTORE_TRACKING]
else:
@@ -434,6 +438,156 @@ class Tinfoil:
"""
return self.run_command('buildFile', buildfile, task, internal)
+ def build_targets(self, targets, task=None, handle_events=True, extra_events=None, event_callback=None):
+ """
+ Builds the specified targets. This is equivalent to a normal invocation
+ of bitbake. Has built-in event handling which is enabled by default and
+ can be extended if needed.
+ Parameters:
+ targets:
+ One or more targets to build. Can be a list or a
+ space-separated string.
+ task:
+ The task to run; if None then the value of BB_DEFAULT_TASK
+ will be used. Default None.
+ handle_events:
+ True to handle events in a similar way to normal bitbake
+ invocation with knotty; False to return immediately (on the
+ assumption that the caller will handle the events instead).
+ Default True.
+ extra_events:
+ An optional list of events to add to the event mask (if
+ handle_events=True). If you add events here you also need
+ to specify a callback function in event_callback that will
+ handle the additional events. Default None.
+ event_callback:
+ An optional function taking a single parameter which
+ will be called first upon receiving any event (if
+ handle_events=True) so that the caller can override or
+ extend the event handling. Default None.
+ """
+ if isinstance(targets, str):
+ targets = targets.split()
+ if not task:
+ task = self.config_data.getVar('BB_DEFAULT_TASK')
+
+ if handle_events:
+ # A reasonable set of default events matching up with those we handle below
+ eventmask = [
+ 'bb.event.BuildStarted',
+ 'bb.event.BuildCompleted',
+ 'logging.LogRecord',
+ 'bb.event.NoProvider',
+ 'bb.command.CommandCompleted',
+ 'bb.command.CommandFailed',
+ 'bb.build.TaskStarted',
+ 'bb.build.TaskFailed',
+ 'bb.build.TaskSucceeded',
+ 'bb.build.TaskFailedSilent',
+ 'bb.build.TaskProgress',
+ 'bb.runqueue.runQueueTaskStarted',
+ 'bb.runqueue.sceneQueueTaskStarted',
+ 'bb.event.ProcessStarted',
+ 'bb.event.ProcessProgress',
+ 'bb.event.ProcessFinished',
+ ]
+ if extra_events:
+ eventmask.extend(extra_events)
+ ret = self.set_event_mask(eventmask)
+
+ ret = self.run_command('buildTargets', targets, task)
+ if handle_events:
+ result = False
+ # Borrowed from knotty, instead somewhat hackily we use the helper
+ # as the object to store "shutdown" on
+ helper = bb.ui.uihelper.BBUIHelper()
+ # We set up logging optionally in the constructor so now we need to
+ # grab the handlers to pass to TerminalFilter
+ console = None
+ errconsole = None
+ for handler in self.logger.handlers:
+ if isinstance(handler, logging.StreamHandler):
+ if handler.stream == sys.stdout:
+ console = handler
+ elif handler.stream == sys.stderr:
+ errconsole = handler
+ format_str = "%(levelname)s: %(message)s"
+ format = bb.msg.BBLogFormatter(format_str)
+ helper.shutdown = 0
+ parseprogress = None
+ termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, format, quiet=self.quiet)
+ try:
+ while True:
+ try:
+ event = self.wait_event(0.25)
+ if event:
+ if event_callback and event_callback(event):
+ continue
+ if helper.eventHandler(event):
+ continue
+ if isinstance(event, bb.event.ProcessStarted):
+ if self.quiet > 1:
+ continue
+ parseprogress = bb.ui.knotty.new_progress(event.processname, event.total)
+ parseprogress.start(False)
+ continue
+ if isinstance(event, bb.event.ProcessProgress):
+ if self.quiet > 1:
+ continue
+ if parseprogress:
+ parseprogress.update(event.progress)
+ else:
+ bb.warn("Got ProcessProgress event for someting that never started?")
+ continue
+ if isinstance(event, bb.event.ProcessFinished):
+ if self.quiet > 1:
+ continue
+ if parseprogress:
+ parseprogress.finish()
+ parseprogress = None
+ continue
+ if isinstance(event, bb.command.CommandCompleted):
+ result = True
+ break
+ if isinstance(event, bb.command.CommandFailed):
+ self.logger.error(str(event))
+ result = False
+ break
+ if isinstance(event, logging.LogRecord):
+ if event.taskpid == 0 or event.levelno > logging.INFO:
+ self.logger.handle(event)
+ continue
+ if isinstance(event, bb.event.NoProvider):
+ self.logger.error(str(event))
+ result = False
+ break
+
+ elif helper.shutdown > 1:
+ break
+ termfilter.updateFooter()
+ except KeyboardInterrupt:
+ termfilter.clearFooter()
+ if helper.shutdown == 1:
+ print("\nSecond Keyboard Interrupt, stopping...\n")
+ ret = self.run_command("stateForceShutdown")
+ if ret and ret[2]:
+ self.logger.error("Unable to cleanly stop: %s" % ret[2])
+ elif helper.shutdown == 0:
+ print("\nKeyboard Interrupt, closing down...\n")
+ interrupted = True
+ ret = self.run_command("stateShutdown")
+ if ret and ret[2]:
+ self.logger.error("Unable to cleanly shutdown: %s" % ret[2])
+ helper.shutdown = helper.shutdown + 1
+ termfilter.clearFooter()
+ finally:
+ termfilter.finish()
+ if helper.failed_tasks:
+ result = False
+ return result
+ else:
+ return ret
+
def shutdown(self):
if self.server_connection:
self.run_command('clientComplete')
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 08/12] tinfoil: enable access to additional cached items
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
` (6 preceding siblings ...)
2017-07-19 9:56 ` [PATCH 07/12] tinfoil: add functionality for running full builds Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 09/12] tinfoil: add simple API for getting cached recipe information Paul Eggleton
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
Add access to fn_provides, packages, packages_dynamic and rproviders on
the recipecache object. This requires an additional corresponding
command plumbing to be added.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/command.py | 32 ++++++++++++++++++++++++++++++++
lib/bb/tinfoil.py | 8 ++++++++
2 files changed, 40 insertions(+)
diff --git a/lib/bb/command.py b/lib/bb/command.py
index 01a2c61..4c00a34 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -307,6 +307,38 @@ class CommandsSync:
return command.cooker.recipecaches[mc].pkg_pepvpr
getRecipeVersions.readonly = True
+ def getRecipeProvides(self, command, params):
+ try:
+ mc = params[0]
+ except IndexError:
+ mc = ''
+ return command.cooker.recipecaches[mc].fn_provides
+ getRecipeProvides.readonly = True
+
+ def getRecipePackages(self, command, params):
+ try:
+ mc = params[0]
+ except IndexError:
+ mc = ''
+ return command.cooker.recipecaches[mc].packages
+ getRecipePackages.readonly = True
+
+ def getRecipePackagesDynamic(self, command, params):
+ try:
+ mc = params[0]
+ except IndexError:
+ mc = ''
+ return command.cooker.recipecaches[mc].packages_dynamic
+ getRecipePackagesDynamic.readonly = True
+
+ def getRProviders(self, command, params):
+ try:
+ mc = params[0]
+ except IndexError:
+ mc = ''
+ return command.cooker.recipecaches[mc].rproviders
+ getRProviders.readonly = True
+
def getRuntimeDepends(self, command, params):
ret = []
try:
diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index 580ce8d..feaeb56 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -173,6 +173,14 @@ class TinfoilCookerAdapter:
attrvalue = self.tinfoil.run_command('getBbFilePriority') or {}
elif name == 'pkg_dp':
attrvalue = self.tinfoil.run_command('getDefaultPreference') or {}
+ elif name == 'fn_provides':
+ attrvalue = self.tinfoil.run_command('getRecipeProvides') or {}
+ elif name == 'packages':
+ attrvalue = self.tinfoil.run_command('getRecipePackages') or {}
+ elif name == 'packages_dynamic':
+ attrvalue = self.tinfoil.run_command('getRecipePackagesDynamic') or {}
+ elif name == 'rproviders':
+ attrvalue = self.tinfoil.run_command('getRProviders') or {}
else:
raise AttributeError("%s instance has no attribute '%s'" % (self.__class__.__name__, name))
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 09/12] tinfoil: add simple API for getting cached recipe information
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
` (7 preceding siblings ...)
2017-07-19 9:56 ` [PATCH 08/12] tinfoil: enable access to additional cached items Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 10/12] tinfoil: add more doc comments Paul Eggleton
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
A common task for tinfoil-using scripts is to iterate over all recipes.
This isn't too difficult with the current API, but the pkg_* variables
are a little awkward and are really designed for bitbake's internal
usage - and it gets a bit more difficult when you want to access some of
the other information such as packages and rprovides. To resolve this,
create a new recipe info class and add an all_recipes() function to
generate this for all recipes. Also add a get_recipe_info() function to
get the information for a specific recipe (by PN).
(It might perhaps be suggested that we already have a structure similar
to this in the cache, however the one we add here is designed for
external use and allows the internal structures to change if needed
without affecting the API).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/tinfoil.py | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 142 insertions(+)
diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index feaeb56..7209cc9 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -217,6 +217,82 @@ class TinfoilCookerAdapter:
return self.tinfoil.find_best_provider(pn)
+class TinfoilRecipeInfo:
+ """
+ Provides a convenient representation of the cached information for a single recipe.
+ Some attributes are set on construction, others are read on-demand (which internally
+ may result in a remote procedure call to the bitbake server the first time).
+ Note that only information which is cached is available through this object - if
+ you need other variable values you will need to parse the recipe using
+ Tinfoil.parse_recipe().
+ """
+ def __init__(self, recipecache, d, pn, fn, fns):
+ self._recipecache = recipecache
+ self._d = d
+ self.pn = pn
+ self.fn = fn
+ self.fns = fns
+ self.inherit_files = recipecache.inherits[fn]
+ self.depends = recipecache.deps[fn]
+ (self.pe, self.pv, self.pr) = recipecache.pkg_pepvpr[fn]
+ self._cached_packages = None
+ self._cached_rprovides = None
+ self._cached_packages_dynamic = None
+
+ def __getattr__(self, name):
+ if name == 'alternates':
+ return [x for x in self.fns if x != self.fn]
+ elif name == 'rdepends':
+ return self._recipecache.rundeps[self.fn]
+ elif name == 'rrecommends':
+ return self._recipecache.runrecs[self.fn]
+ elif name == 'provides':
+ return self._recipecache.fn_provides[self.fn]
+ elif name == 'packages':
+ if self._cached_packages is None:
+ self._cached_packages = []
+ for pkg, fns in self._recipecache.packages.items():
+ if self.fn in fns:
+ self._cached_packages.append(pkg)
+ return self._cached_packages
+ elif name == 'packages_dynamic':
+ if self._cached_packages_dynamic is None:
+ self._cached_packages_dynamic = []
+ for pkg, fns in self._recipecache.packages_dynamic.items():
+ if self.fn in fns:
+ self._cached_packages_dynamic.append(pkg)
+ return self._cached_packages_dynamic
+ elif name == 'rprovides':
+ if self._cached_rprovides is None:
+ self._cached_rprovides = []
+ for pkg, fns in self._recipecache.rproviders.items():
+ if self.fn in fns:
+ self._cached_rprovides.append(pkg)
+ return self._cached_rprovides
+ else:
+ raise AttributeError("%s instance has no attribute '%s'" % (self.__class__.__name__, name))
+ def inherits(self, only_recipe=False):
+ """
+ Get the inherited classes for a recipe. Returns the class names only.
+ Parameters:
+ only_recipe: True to return only the classes inherited by the recipe
+ itself, False to return all classes inherited within
+ the context for the recipe (which includes globally
+ inherited classes).
+ """
+ if only_recipe:
+ global_inherit = [x for x in (self._d.getVar('BBINCLUDED') or '').split() if x.endswith('.bbclass')]
+ else:
+ global_inherit = []
+ for clsfile in self.inherit_files:
+ if only_recipe and clsfile in global_inherit:
+ continue
+ clsname = os.path.splitext(os.path.basename(clsfile))[0]
+ yield clsname
+ def __str__(self):
+ return '%s' % self.pn
+
+
class Tinfoil:
def __init__(self, output=sys.stdout, tracking=False, setup_logging=True):
@@ -401,6 +477,72 @@ class Tinfoil:
def get_file_appends(self, fn):
return self.run_command('getFileAppends', fn)
+ def all_recipes(self, mc='', sort=True):
+ """
+ Enable iterating over all recipes in the current configuration.
+ Returns an iterator over TinfoilRecipeInfo objects created on demand.
+ Parameters:
+ mc: The multiconfig, default of '' uses the main configuration.
+ sort: True to sort recipes alphabetically (default), False otherwise
+ """
+ recipecache = self.cooker.recipecaches[mc]
+ if sort:
+ recipes = sorted(recipecache.pkg_pn.items())
+ else:
+ recipes = recipecache.pkg_pn.items()
+ for pn, fns in recipes:
+ prov = self.find_best_provider(pn)
+ recipe = TinfoilRecipeInfo(recipecache,
+ self.config_data,
+ pn=pn,
+ fn=prov[3],
+ fns=fns)
+ yield recipe
+
+ def all_recipe_files(self, mc='', variants=True, preferred_only=False):
+ """
+ Enable iterating over all recipe files in the current configuration.
+ Returns an iterator over file paths.
+ Parameters:
+ mc: The multiconfig, default of '' uses the main configuration.
+ variants: True to include variants of recipes created through
+ BBCLASSEXTEND (default) or False to exclude them
+ preferred_only: True to include only the preferred recipe where
+ multiple exist providing the same PN, False to list
+ all recipes
+ """
+ recipecache = self.cooker.recipecaches[mc]
+ if preferred_only:
+ files = []
+ for pn in recipecache.pkg_pn.keys():
+ prov = self.find_best_provider(pn)
+ files.append(prov[3])
+ else:
+ files = recipecache.pkg_fn.keys()
+ for fn in sorted(files):
+ if not variants and fn.startswith('virtual:'):
+ continue
+ yield fn
+
+
+ def get_recipe_info(self, pn, mc=''):
+ """
+ Get information on a specific recipe in the current configuration by name (PN).
+ Returns a TinfoilRecipeInfo object created on demand.
+ Parameters:
+ mc: The multiconfig, default of '' uses the main configuration.
+ """
+ recipecache = self.cooker.recipecaches[mc]
+ prov = self.find_best_provider(pn)
+ fn = prov[3]
+ actual_pn = recipecache.pkg_fn[fn]
+ recipe = TinfoilRecipeInfo(recipecache,
+ self.config_data,
+ pn=actual_pn,
+ fn=fn,
+ fns=recipecache.pkg_pn[actual_pn])
+ return recipe
+
def parse_recipe(self, pn):
"""
Parse the specified recipe and return a datastore object
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 10/12] tinfoil: add more doc comments
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
` (8 preceding siblings ...)
2017-07-19 9:56 ` [PATCH 09/12] tinfoil: add simple API for getting cached recipe information Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 11/12] fetch2: allow hiding checksum warning Paul Eggleton
2017-07-19 9:56 ` [PATCH 12/12] fetch2: fire an event when there are missing checksums Paul Eggleton
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
We want this API to be easier to use, so add missing function
documentation to help with that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/tinfoil.py | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 65 insertions(+)
diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index 7209cc9..5d1d7f8 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -55,6 +55,7 @@ class TinfoilCommandFailed(Exception):
"""Exception raised when run_command fails"""
class TinfoilDataStoreConnector:
+ """Connector object used to enable access to datastore objects via tinfoil"""
def __init__(self, tinfoil, dsindex):
self.tinfoil = tinfoil
@@ -294,8 +295,25 @@ class TinfoilRecipeInfo:
class Tinfoil:
+ """
+ Tinfoil - an API for scripts and utilities to query
+ BitBake internals and perform build operations.
+ """
def __init__(self, output=sys.stdout, tracking=False, setup_logging=True):
+ """
+ Create a new tinfoil object.
+ Parameters:
+ output: specifies where console output should be sent. Defaults
+ to sys.stdout.
+ tracking: True to enable variable history tracking, False to
+ disable it (default). Enabling this has a minor
+ performance impact so typically it isn't enabled
+ unless you need to query variable history.
+ setup_logging: True to setup a logger so that things like
+ bb.warn() will work immediately and timeout warnings
+ are visible; False to let BitBake do this itself.
+ """
self.logger = logging.getLogger('BitBake')
self.config_data = None
self.cooker = None
@@ -316,6 +334,37 @@ class Tinfoil:
self.shutdown()
def prepare(self, config_only=False, config_params=None, quiet=0, extra_features=None):
+ """
+ Prepares the underlying BitBake system to be used via tinfoil.
+ This function must be called prior to calling any of the other
+ functions in the API.
+ NOTE: if you call prepare() you must absolutely call shutdown()
+ before your code terminates. You can use a "with" block to ensure
+ this happens e.g.
+
+ with bb.tinfoil.Tinfoil() as tinfoil:
+ tinfoil.prepare()
+ ...
+
+ Parameters:
+ config_only: True to read only the configuration and not load
+ the cache / parse recipes. This is useful if you just
+ want to query the value of a variable at the global
+ level or you want to do anything else that doesn't
+ involve knowing anything about the recipes in the
+ current configuration. False loads the cache / parses
+ recipes.
+ config_params: optionally specify your own configuration
+ parameters. If not specified an instance of
+ TinfoilConfigParameters will be created internally.
+ quiet: quiet level controlling console output - equivalent
+ to bitbake's -q/--quiet option. Default of 0 gives
+ the same output level as normal bitbake execution.
+ extra_features: extra features to be added to the feature
+ set requested from the server. See
+ CookerFeatures._feature_list for possible
+ features.
+ """
self.quiet = quiet
if self.tracking:
@@ -440,9 +489,16 @@ class Tinfoil:
return self.server_connection.events.waitEvent(timeout)
def get_overlayed_recipes(self):
+ """
+ Find recipes which are overlayed (i.e. where recipes exist in multiple layers)
+ """
return defaultdict(list, self.run_command('getOverlayedRecipes'))
def get_skipped_recipes(self):
+ """
+ Find recipes which were skipped (i.e. SkipRecipe was raised
+ during parsing).
+ """
return OrderedDict(self.run_command('getSkippedRecipes'))
def get_all_providers(self):
@@ -475,6 +531,9 @@ class Tinfoil:
return best[3]
def get_file_appends(self, fn):
+ """
+ Find the bbappends for a recipe file
+ """
return self.run_command('getFileAppends', fn)
def all_recipes(self, mc='', sort=True):
@@ -739,6 +798,12 @@ class Tinfoil:
return ret
def shutdown(self):
+ """
+ Shut down tinfoil. Disconnects from the server and gracefully
+ releases any associated resources. You must call this function if
+ prepare() has been called, or use a with... block when you create
+ the tinfoil object which will ensure that it gets called.
+ """
if self.server_connection:
self.run_command('clientComplete')
_server_connections.remove(self.server_connection)
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 11/12] fetch2: allow hiding checksum warning
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
` (9 preceding siblings ...)
2017-07-19 9:56 ` [PATCH 10/12] tinfoil: add more doc comments Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
2017-07-19 9:56 ` [PATCH 12/12] fetch2: fire an event when there are missing checksums Paul Eggleton
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
If BB_STRICT_CHECKSUMS is set to "ignore" then don't display a warning
if no checksums are specified in the recipe. This is not intended to be
used from recipes - it is needed when we move to using more standard
code paths to fetch new files from scripts i.e. where we don't know what
the checksums are in advance.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/fetch2/__init__.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 5c02317..2e615b2 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -584,6 +584,12 @@ def verify_checksum(ud, d, precomputed={}):
ud.sha256_name, sha256data))
raise NoChecksumError('Missing SRC_URI checksum', ud.url)
+ if strict == "ignore":
+ return {
+ _MD5_KEY: md5data,
+ _SHA256_KEY: sha256data
+ }
+
# Log missing sums so user can more easily add them
logger.warning('Missing md5 SRC_URI checksum for %s, consider adding to the recipe:\n'
'SRC_URI[%s] = "%s"',
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 12/12] fetch2: fire an event when there are missing checksums
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
` (10 preceding siblings ...)
2017-07-19 9:56 ` [PATCH 11/12] fetch2: allow hiding checksum warning Paul Eggleton
@ 2017-07-19 9:56 ` Paul Eggleton
11 siblings, 0 replies; 13+ messages in thread
From: Paul Eggleton @ 2017-07-19 9:56 UTC (permalink / raw)
To: bitbake-devel
If BB_STRICT_CHECKSUMS is set to anything other than "1" i.e. we're not
going to raise an error, then fire an event so that scripts can listen
for it and get the checksums.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/fetch2/__init__.py | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 2e615b2..74ba37f 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -39,6 +39,7 @@ import errno
import bb.persist_data, bb.utils
import bb.checksum
import bb.process
+import bb.event
__version__ = "2"
_checksum_cache = bb.checksum.FileChecksumCache()
@@ -142,6 +143,13 @@ class NonLocalMethod(Exception):
def __init__(self):
Exception.__init__(self)
+class MissingChecksumEvent(bb.event.Event):
+ def __init__(self, url, md5sum, sha256sum):
+ self.url = url
+ self.checksums = {'md5sum': md5sum,
+ 'sha256sum': sha256sum}
+ bb.event.Event.__init__(self)
+
class URI(object):
"""
@@ -584,6 +592,8 @@ def verify_checksum(ud, d, precomputed={}):
ud.sha256_name, sha256data))
raise NoChecksumError('Missing SRC_URI checksum', ud.url)
+ bb.event.fire(MissingChecksumEvent(ud.url, md5data, sha256data), d)
+
if strict == "ignore":
return {
_MD5_KEY: md5data,
--
2.9.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
end of thread, other threads:[~2017-07-19 9:56 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-19 9:55 [PATCH 00/12] Tinfoil enhancements / fetch fixes Paul Eggleton
2017-07-19 9:56 ` [PATCH 01/12] tinfoil: add internal mode to build_file() function Paul Eggleton
2017-07-19 9:56 ` [PATCH 02/12] tinfoil: add a parse_recipes() function Paul Eggleton
2017-07-19 9:56 ` [PATCH 03/12] tinfoil: set a flag when recipes have been parsed Paul Eggleton
2017-07-19 9:56 ` [PATCH 04/12] lib/bb/event: refactor printing events Paul Eggleton
2017-07-19 9:56 ` [PATCH 05/12] lib/bb/ui/uihelper: indicate to caller of eventHandler() if events handled Paul Eggleton
2017-07-19 9:56 ` [PATCH 06/12] knotty: make it possible to use termfilter without either console Paul Eggleton
2017-07-19 9:56 ` [PATCH 07/12] tinfoil: add functionality for running full builds Paul Eggleton
2017-07-19 9:56 ` [PATCH 08/12] tinfoil: enable access to additional cached items Paul Eggleton
2017-07-19 9:56 ` [PATCH 09/12] tinfoil: add simple API for getting cached recipe information Paul Eggleton
2017-07-19 9:56 ` [PATCH 10/12] tinfoil: add more doc comments Paul Eggleton
2017-07-19 9:56 ` [PATCH 11/12] fetch2: allow hiding checksum warning Paul Eggleton
2017-07-19 9:56 ` [PATCH 12/12] fetch2: fire an event when there are missing checksums Paul Eggleton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.