* [PATCH 00/10] Support progress reporting
@ 2016-06-13 2:52 Paul Eggleton
2016-06-13 2:52 ` [PATCH 01/10] knotty: provide a symlink to the latest console log Paul Eggleton
` (9 more replies)
0 siblings, 10 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
Add code to support progress reporting within OpenEmbedded, as well as a
couple of related changes to allow reworking the extensible SDK
installation process.
The following changes since commit 6d1379c8818400e5cdc442e6142f08a110fd5b95:
lib/bb/main.py: Fix use of BBPOSTCONF and BBPRECONF (2016-06-09 18:00:45 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib paule/startup-bb
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/startup-bb
Paul Eggleton (10):
knotty: provide a symlink to the latest console log
knotty: import latest python-progressbar
lib: implement basic task progress support
lib/bb/progress: add MultiStageProgressReporter
fetch2: implement progress support
knotty: add code to support showing progress for sstate object querying
knotty: show task progress bar
knotty: add quiet output mode
runqueue: add ability to enforce that tasks are setscened
runqueue: report progress for "Preparing RunQueue" step
lib/bb/build.py | 34 ++++
lib/bb/event.py | 27 +++
lib/bb/fetch2/__init__.py | 4 +-
lib/bb/fetch2/git.py | 52 +++++-
lib/bb/fetch2/wget.py | 26 ++-
lib/bb/main.py | 9 +
lib/bb/msg.py | 5 +-
lib/bb/progress.py | 266 ++++++++++++++++++++++++++++
lib/bb/runqueue.py | 131 +++++++++++++-
lib/bb/ui/knotty.py | 143 ++++++++++++---
lib/bb/ui/uihelper.py | 7 +-
lib/progressbar.py | 384 ----------------------------------------
lib/progressbar/LICENSE.txt | 52 ++++++
lib/progressbar/__init__.py | 49 ++++++
lib/progressbar/compat.py | 44 +++++
lib/progressbar/progressbar.py | 315 +++++++++++++++++++++++++++++++++
lib/progressbar/widgets.py | 391 +++++++++++++++++++++++++++++++++++++++++
17 files changed, 1519 insertions(+), 420 deletions(-)
create mode 100644 lib/bb/progress.py
delete mode 100644 lib/progressbar.py
create mode 100644 lib/progressbar/LICENSE.txt
create mode 100644 lib/progressbar/__init__.py
create mode 100644 lib/progressbar/compat.py
create mode 100644 lib/progressbar/progressbar.py
create mode 100644 lib/progressbar/widgets.py
--
2.5.5
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 01/10] knotty: provide a symlink to the latest console log
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-13 2:52 ` [PATCH 02/10] knotty: import latest python-progressbar Paul Eggleton
` (8 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
If you're looking to find the latest console log repeatedly it can be a bit
tedious - let's just create a symlink just as we do with other logs to
make it easy to find.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/ui/knotty.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 9605c8e..ddd36d5 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -287,6 +287,12 @@ def main(server, eventHandler, params, tf = TerminalFilter):
bb.msg.addDefaultlogFilter(consolelog)
consolelog.setFormatter(conlogformat)
logger.addHandler(consolelog)
+ loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log')
+ bb.utils.remove(loglink)
+ try:
+ os.symlink(os.path.basename(consolelogfile), loglink)
+ except OSError:
+ pass
llevel, debug_domains = bb.msg.constructLogOptions()
server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list])
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 02/10] knotty: import latest python-progressbar
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
2016-06-13 2:52 ` [PATCH 01/10] knotty: provide a symlink to the latest console log Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-13 2:52 ` [PATCH 03/10] lib: implement basic task progress support Paul Eggleton
` (7 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
Since we're going to make some minor extensions to it, it makes sense to
bring in the latest version of python-progressbar. Its structure has
changed a little but the API hasn't; however we do need to ensure our
overridden _needs_update() function's signature in BBProgress() matches
properly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/ui/knotty.py | 2 +-
lib/progressbar.py | 384 -----------------------------------------
lib/progressbar/LICENSE.txt | 52 ++++++
lib/progressbar/__init__.py | 49 ++++++
lib/progressbar/compat.py | 44 +++++
lib/progressbar/progressbar.py | 307 ++++++++++++++++++++++++++++++++
lib/progressbar/widgets.py | 355 +++++++++++++++++++++++++++++++++++++
7 files changed, 808 insertions(+), 385 deletions(-)
delete mode 100644 lib/progressbar.py
create mode 100644 lib/progressbar/LICENSE.txt
create mode 100644 lib/progressbar/__init__.py
create mode 100644 lib/progressbar/compat.py
create mode 100644 lib/progressbar/progressbar.py
create mode 100644 lib/progressbar/widgets.py
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index ddd36d5..6a6f688 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -51,7 +51,7 @@ class BBProgress(progressbar.ProgressBar):
self._resize_default = None
progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets, fd=sys.stdout)
- def _handle_resize(self, signum, frame):
+ def _handle_resize(self, signum=None, frame=None):
progressbar.ProgressBar._handle_resize(self, signum, frame)
if self._resize_default:
self._resize_default(signum, frame)
diff --git a/lib/progressbar.py b/lib/progressbar.py
deleted file mode 100644
index 114cdc1..0000000
--- a/lib/progressbar.py
+++ /dev/null
@@ -1,384 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: iso-8859-1 -*-
-#
-# progressbar - Text progressbar library for python.
-# Copyright (c) 2005 Nilton Volpato
-#
-# This library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-#
-# This library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with this library; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
-
-
-"""Text progressbar library for python.
-
-This library provides a text mode progressbar. This is typically used
-to display the progress of a long running operation, providing a
-visual clue that processing is underway.
-
-The ProgressBar class manages the progress, and the format of the line
-is given by a number of widgets. A widget is an object that may
-display diferently depending on the state of the progress. There are
-three types of widget:
-- a string, which always shows itself;
-- a ProgressBarWidget, which may return a diferent value every time
-it's update method is called; and
-- a ProgressBarWidgetHFill, which is like ProgressBarWidget, except it
-expands to fill the remaining width of the line.
-
-The progressbar module is very easy to use, yet very powerful. And
-automatically supports features like auto-resizing when available.
-"""
-
-from __future__ import division
-
-__author__ = "Nilton Volpato"
-__author_email__ = "first-name dot last-name @ gmail.com"
-__date__ = "2006-05-07"
-__version__ = "2.3-dev"
-
-import sys, time, os
-from array import array
-try:
- from fcntl import ioctl
- import termios
-except ImportError:
- pass
-import signal
-try:
- basestring
-except NameError:
- basestring = (str,)
-
-class ProgressBarWidget(object):
- """This is an element of ProgressBar formatting.
-
- The ProgressBar object will call it's update value when an update
- is needed. It's size may change between call, but the results will
- not be good if the size changes drastically and repeatedly.
- """
- def update(self, pbar):
- """Returns the string representing the widget.
-
- The parameter pbar is a reference to the calling ProgressBar,
- where one can access attributes of the class for knowing how
- the update must be made.
-
- At least this function must be overriden."""
- pass
-
-class ProgressBarWidgetHFill(object):
- """This is a variable width element of ProgressBar formatting.
-
- The ProgressBar object will call it's update value, informing the
- width this object must the made. This is like TeX \\hfill, it will
- expand to fill the line. You can use more than one in the same
- line, and they will all have the same width, and together will
- fill the line.
- """
- def update(self, pbar, width):
- """Returns the string representing the widget.
-
- The parameter pbar is a reference to the calling ProgressBar,
- where one can access attributes of the class for knowing how
- the update must be made. The parameter width is the total
- horizontal width the widget must have.
-
- At least this function must be overriden."""
- pass
-
-
-class ETA(ProgressBarWidget):
- "Widget for the Estimated Time of Arrival"
- def format_time(self, seconds):
- return time.strftime('%H:%M:%S', time.gmtime(seconds))
- def update(self, pbar):
- if pbar.currval == 0:
- return 'ETA: --:--:--'
- elif pbar.finished:
- return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
- else:
- elapsed = pbar.seconds_elapsed
- eta = elapsed * pbar.maxval / pbar.currval - elapsed
- return 'ETA: %s' % self.format_time(eta)
-
-class FileTransferSpeed(ProgressBarWidget):
- "Widget for showing the transfer speed (useful for file transfers)."
- def __init__(self, unit='B'):
- self.unit = unit
- self.fmt = '%6.2f %s'
- self.prefixes = ['', 'K', 'M', 'G', 'T', 'P']
- def update(self, pbar):
- if pbar.seconds_elapsed < 2e-6:#== 0:
- bps = 0.0
- else:
- bps = pbar.currval / pbar.seconds_elapsed
- spd = bps
- for u in self.prefixes:
- if spd < 1000:
- break
- spd /= 1000
- return self.fmt % (spd, u + self.unit + '/s')
-
-class RotatingMarker(ProgressBarWidget):
- "A rotating marker for filling the bar of progress."
- def __init__(self, markers='|/-\\'):
- self.markers = markers
- self.curmark = -1
- def update(self, pbar):
- if pbar.finished:
- return self.markers[0]
- self.curmark = (self.curmark + 1) % len(self.markers)
- return self.markers[self.curmark]
-
-class Percentage(ProgressBarWidget):
- "Just the percentage done."
- def update(self, pbar):
- return '%3d%%' % pbar.percentage()
-
-class SimpleProgress(ProgressBarWidget):
- "Returns what is already done and the total, e.g.: '5 of 47'"
- def __init__(self, sep=' of '):
- self.sep = sep
- def update(self, pbar):
- return '%d%s%d' % (pbar.currval, self.sep, pbar.maxval)
-
-class Bar(ProgressBarWidgetHFill):
- "The bar of progress. It will stretch to fill the line."
- def __init__(self, marker='#', left='|', right='|'):
- self.marker = marker
- self.left = left
- self.right = right
- def _format_marker(self, pbar):
- if isinstance(self.marker, basestring):
- return self.marker
- else:
- return self.marker.update(pbar)
- def update(self, pbar, width):
- percent = pbar.percentage()
- cwidth = width - len(self.left) - len(self.right)
- marked_width = int(percent * cwidth // 100)
- m = self._format_marker(pbar)
- bar = (self.left + (m * marked_width).ljust(cwidth) + self.right)
- return bar
-
-class ReverseBar(Bar):
- "The reverse bar of progress, or bar of regress. :)"
- def update(self, pbar, width):
- percent = pbar.percentage()
- cwidth = width - len(self.left) - len(self.right)
- marked_width = int(percent * cwidth // 100)
- m = self._format_marker(pbar)
- bar = (self.left + (m*marked_width).rjust(cwidth) + self.right)
- return bar
-
-default_widgets = [Percentage(), ' ', Bar()]
-class ProgressBar(object):
- """This is the ProgressBar class, it updates and prints the bar.
-
- A common way of using it is like:
- >>> pbar = ProgressBar().start()
- >>> for i in xrange(100):
- ... # do something
- ... pbar.update(i+1)
- ...
- >>> pbar.finish()
-
- You can also use a progressbar as an iterator:
- >>> progress = ProgressBar()
- >>> for i in progress(some_iterable):
- ... # do something
- ...
-
- But anything you want to do is possible (well, almost anything).
- You can supply different widgets of any type in any order. And you
- can even write your own widgets! There are many widgets already
- shipped and you should experiment with them.
-
- The term_width parameter must be an integer or None. In the latter case
- it will try to guess it, if it fails it will default to 80 columns.
-
- When implementing a widget update method you may access any
- attribute or function of the ProgressBar object calling the
- widget's update method. The most important attributes you would
- like to access are:
- - currval: current value of the progress, 0 <= currval <= maxval
- - maxval: maximum (and final) value of the progress
- - finished: True if the bar has finished (reached 100%), False o/w
- - start_time: the time when start() method of ProgressBar was called
- - seconds_elapsed: seconds elapsed since start_time
- - percentage(): percentage of the progress [0..100]. This is a method.
-
- The attributes above are unlikely to change between different versions,
- the other ones may change or cease to exist without notice, so try to rely
- only on the ones documented above if you are extending the progress bar.
- """
-
- __slots__ = ('currval', 'fd', 'finished', 'last_update_time', 'maxval',
- 'next_update', 'num_intervals', 'seconds_elapsed',
- 'signal_set', 'start_time', 'term_width', 'update_interval',
- 'widgets', '_iterable')
-
- _DEFAULT_MAXVAL = 100
-
- def __init__(self, maxval=None, widgets=default_widgets, term_width=None,
- fd=sys.stderr):
- self.maxval = maxval
- self.widgets = widgets
- self.fd = fd
- self.signal_set = False
- if term_width is not None:
- self.term_width = term_width
- else:
- try:
- self._handle_resize(None, None)
- signal.signal(signal.SIGWINCH, self._handle_resize)
- self.signal_set = True
- except (SystemExit, KeyboardInterrupt):
- raise
- except:
- self.term_width = int(os.environ.get('COLUMNS', 80)) - 1
-
- self.currval = 0
- self.finished = False
- self.start_time = None
- self.last_update_time = None
- self.seconds_elapsed = 0
- self._iterable = None
-
- def __call__(self, iterable):
- try:
- self.maxval = len(iterable)
- except TypeError:
- # If the iterable has no length, then rely on the value provided
- # by the user, otherwise fail.
- if not (isinstance(self.maxval, (int, long)) and self.maxval > 0):
- raise RuntimeError('Could not determine maxval from iterable. '
- 'You must explicitly provide a maxval.')
- self._iterable = iter(iterable)
- self.start()
- return self
-
- def __iter__(self):
- return self
-
- def next(self):
- try:
- next = self._iterable.next()
- self.update(self.currval + 1)
- return next
- except StopIteration:
- self.finish()
- raise
-
- def _handle_resize(self, signum, frame):
- h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2]
- self.term_width = w
-
- def percentage(self):
- "Returns the percentage of the progress."
- return self.currval * 100.0 / self.maxval
-
- def _format_widgets(self):
- r = []
- hfill_inds = []
- num_hfill = 0
- currwidth = 0
- for i, w in enumerate(self.widgets):
- if isinstance(w, ProgressBarWidgetHFill):
- r.append(w)
- hfill_inds.append(i)
- num_hfill += 1
- elif isinstance(w, basestring):
- r.append(w)
- currwidth += len(w)
- else:
- weval = w.update(self)
- currwidth += len(weval)
- r.append(weval)
- for iw in hfill_inds:
- widget_width = int((self.term_width - currwidth) // num_hfill)
- r[iw] = r[iw].update(self, widget_width)
- return r
-
- def _format_line(self):
- return ''.join(self._format_widgets()).ljust(self.term_width)
-
- def _next_update(self):
- return int((int(self.num_intervals *
- (self.currval / self.maxval)) + 1) *
- self.update_interval)
-
- def _need_update(self):
- """Returns true when the progressbar should print an updated line.
-
- You can override this method if you want finer grained control over
- updates.
-
- The current implementation is optimized to be as fast as possible and
- as economical as possible in the number of updates. However, depending
- on your usage you may want to do more updates. For instance, if your
- progressbar stays in the same percentage for a long time, and you want
- to update other widgets, like ETA, then you could return True after
- some time has passed with no updates.
-
- Ideally you could call self._format_line() and see if it's different
- from the previous _format_line() call, but calling _format_line() takes
- around 20 times more time than calling this implementation of
- _need_update().
- """
- return self.currval >= self.next_update
-
- def update(self, value):
- "Updates the progress bar to a new value."
- assert 0 <= value <= self.maxval, '0 <= %d <= %d' % (value, self.maxval)
- self.currval = value
- if not self._need_update():
- return
- if self.start_time is None:
- raise RuntimeError('You must call start() before calling update()')
- now = time.time()
- self.seconds_elapsed = now - self.start_time
- self.next_update = self._next_update()
- self.fd.write(self._format_line() + '\r')
- self.last_update_time = now
-
- def start(self):
- """Starts measuring time, and prints the bar at 0%.
-
- It returns self so you can use it like this:
- >>> pbar = ProgressBar().start()
- >>> for i in xrange(100):
- ... # do something
- ... pbar.update(i+1)
- ...
- >>> pbar.finish()
- """
- if self.maxval is None:
- self.maxval = self._DEFAULT_MAXVAL
- assert self.maxval > 0
-
- self.num_intervals = max(100, self.term_width)
- self.update_interval = self.maxval / self.num_intervals
- self.next_update = 0
-
- self.start_time = self.last_update_time = time.time()
- self.update(0)
- return self
-
- def finish(self):
- """Used to tell the progress is finished."""
- self.finished = True
- self.update(self.maxval)
- self.fd.write('\n')
- if self.signal_set:
- signal.signal(signal.SIGWINCH, signal.SIG_DFL)
diff --git a/lib/progressbar/LICENSE.txt b/lib/progressbar/LICENSE.txt
new file mode 100644
index 0000000..fc8ccdc
--- /dev/null
+++ b/lib/progressbar/LICENSE.txt
@@ -0,0 +1,52 @@
+You can redistribute and/or modify this library under the terms of the
+GNU LGPL license or BSD license (or both).
+
+---
+
+progressbar - Text progress bar library for python.
+Copyright (C) 2005 Nilton Volpato
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Lesser General Public
+License as published by the Free Software Foundation; either
+version 2.1 of the License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+Lesser General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public
+License along with this library; if not, write to the Free Software
+Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+---
+
+progressbar - Text progress bar library for python
+Copyright (c) 2008 Nilton Volpato
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+ a. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ b. Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the distribution.
+ c. Neither the name of the author nor the names of its contributors
+ may be used to endorse or promote products derived from this software
+ without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+DAMAGE.
diff --git a/lib/progressbar/__init__.py b/lib/progressbar/__init__.py
new file mode 100644
index 0000000..fbab744
--- /dev/null
+++ b/lib/progressbar/__init__.py
@@ -0,0 +1,49 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Text progress bar library for Python.
+
+A text progress bar is typically used to display the progress of a long
+running operation, providing a visual cue that processing is underway.
+
+The ProgressBar class manages the current progress, and the format of the line
+is given by a number of widgets. A widget is an object that may display
+differently depending on the state of the progress bar. There are three types
+of widgets:
+ - a string, which always shows itself
+
+ - a ProgressBarWidget, which may return a different value every time its
+ update method is called
+
+ - a ProgressBarWidgetHFill, which is like ProgressBarWidget, except it
+ expands to fill the remaining width of the line.
+
+The progressbar module is very easy to use, yet very powerful. It will also
+automatically enable features like auto-resizing when the system supports it.
+"""
+
+__author__ = 'Nilton Volpato'
+__author_email__ = 'first-name dot last-name @ gmail.com'
+__date__ = '2011-05-14'
+__version__ = '2.3'
+
+from .compat import *
+from .widgets import *
+from .progressbar import *
diff --git a/lib/progressbar/compat.py b/lib/progressbar/compat.py
new file mode 100644
index 0000000..a39f4a1
--- /dev/null
+++ b/lib/progressbar/compat.py
@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Compatibility methods and classes for the progressbar module."""
+
+
+# Python 3.x (and backports) use a modified iterator syntax
+# This will allow 2.x to behave with 3.x iterators
+try:
+ next
+except NameError:
+ def next(iter):
+ try:
+ # Try new style iterators
+ return iter.__next__()
+ except AttributeError:
+ # Fallback in case of a "native" iterator
+ return iter.next()
+
+
+# Python < 2.5 does not have "any"
+try:
+ any
+except NameError:
+ def any(iterator):
+ for item in iterator:
+ if item: return True
+ return False
diff --git a/lib/progressbar/progressbar.py b/lib/progressbar/progressbar.py
new file mode 100644
index 0000000..0b9dcf7
--- /dev/null
+++ b/lib/progressbar/progressbar.py
@@ -0,0 +1,307 @@
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Main ProgressBar class."""
+
+from __future__ import division
+
+import math
+import os
+import signal
+import sys
+import time
+
+try:
+ from fcntl import ioctl
+ from array import array
+ import termios
+except ImportError:
+ pass
+
+from .compat import * # for: any, next
+from . import widgets
+
+
+class UnknownLength: pass
+
+
+class ProgressBar(object):
+ """The ProgressBar class which updates and prints the bar.
+
+ A common way of using it is like:
+ >>> pbar = ProgressBar().start()
+ >>> for i in range(100):
+ ... # do something
+ ... pbar.update(i+1)
+ ...
+ >>> pbar.finish()
+
+ You can also use a ProgressBar as an iterator:
+ >>> progress = ProgressBar()
+ >>> for i in progress(some_iterable):
+ ... # do something
+ ...
+
+ Since the progress bar is incredibly customizable you can specify
+ different widgets of any type in any order. You can even write your own
+ widgets! However, since there are already a good number of widgets you
+ should probably play around with them before moving on to create your own
+ widgets.
+
+ The term_width parameter represents the current terminal width. If the
+ parameter is set to an integer then the progress bar will use that,
+ otherwise it will attempt to determine the terminal width falling back to
+ 80 columns if the width cannot be determined.
+
+ When implementing a widget's update method you are passed a reference to
+ the current progress bar. As a result, you have access to the
+ ProgressBar's methods and attributes. Although there is nothing preventing
+ you from changing the ProgressBar you should treat it as read only.
+
+ Useful methods and attributes include (Public API):
+ - currval: current progress (0 <= currval <= maxval)
+ - maxval: maximum (and final) value
+ - finished: True if the bar has finished (reached 100%)
+ - start_time: the time when start() method of ProgressBar was called
+ - seconds_elapsed: seconds elapsed since start_time and last call to
+ update
+ - percentage(): progress in percent [0..100]
+ """
+
+ __slots__ = ('currval', 'fd', 'finished', 'last_update_time',
+ 'left_justify', 'maxval', 'next_update', 'num_intervals',
+ 'poll', 'seconds_elapsed', 'signal_set', 'start_time',
+ 'term_width', 'update_interval', 'widgets', '_time_sensitive',
+ '__iterable')
+
+ _DEFAULT_MAXVAL = 100
+ _DEFAULT_TERMSIZE = 80
+ _DEFAULT_WIDGETS = [widgets.Percentage(), ' ', widgets.Bar()]
+
+ def __init__(self, maxval=None, widgets=None, term_width=None, poll=1,
+ left_justify=True, fd=sys.stderr):
+ """Initializes a progress bar with sane defaults."""
+
+ # Don't share a reference with any other progress bars
+ if widgets is None:
+ widgets = list(self._DEFAULT_WIDGETS)
+
+ self.maxval = maxval
+ self.widgets = widgets
+ self.fd = fd
+ self.left_justify = left_justify
+
+ self.signal_set = False
+ if term_width is not None:
+ self.term_width = term_width
+ else:
+ try:
+ self._handle_resize(None, None)
+ signal.signal(signal.SIGWINCH, self._handle_resize)
+ self.signal_set = True
+ except (SystemExit, KeyboardInterrupt): raise
+ except Exception as e:
+ print("DEBUG 5 %s" % e)
+ self.term_width = self._env_size()
+
+ self.__iterable = None
+ self._update_widgets()
+ self.currval = 0
+ self.finished = False
+ self.last_update_time = None
+ self.poll = poll
+ self.seconds_elapsed = 0
+ self.start_time = None
+ self.update_interval = 1
+ self.next_update = 0
+
+
+ def __call__(self, iterable):
+ """Use a ProgressBar to iterate through an iterable."""
+
+ try:
+ self.maxval = len(iterable)
+ except:
+ if self.maxval is None:
+ self.maxval = UnknownLength
+
+ self.__iterable = iter(iterable)
+ return self
+
+
+ def __iter__(self):
+ return self
+
+
+ def __next__(self):
+ try:
+ value = next(self.__iterable)
+ if self.start_time is None:
+ self.start()
+ else:
+ self.update(self.currval + 1)
+ return value
+ except StopIteration:
+ if self.start_time is None:
+ self.start()
+ self.finish()
+ raise
+
+
+ # Create an alias so that Python 2.x won't complain about not being
+ # an iterator.
+ next = __next__
+
+
+ def _env_size(self):
+ """Tries to find the term_width from the environment."""
+
+ return int(os.environ.get('COLUMNS', self._DEFAULT_TERMSIZE)) - 1
+
+
+ def _handle_resize(self, signum=None, frame=None):
+ """Tries to catch resize signals sent from the terminal."""
+
+ h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2]
+ self.term_width = w
+
+
+ def percentage(self):
+ """Returns the progress as a percentage."""
+ if self.currval >= self.maxval:
+ return 100.0
+ return (self.currval * 100.0 / self.maxval) if self.maxval else 100.00
+
+ percent = property(percentage)
+
+
+ def _format_widgets(self):
+ result = []
+ expanding = []
+ width = self.term_width
+
+ for index, widget in enumerate(self.widgets):
+ if isinstance(widget, widgets.WidgetHFill):
+ result.append(widget)
+ expanding.insert(0, index)
+ else:
+ widget = widgets.format_updatable(widget, self)
+ result.append(widget)
+ width -= len(widget)
+
+ count = len(expanding)
+ while count:
+ portion = max(int(math.ceil(width * 1. / count)), 0)
+ index = expanding.pop()
+ count -= 1
+
+ widget = result[index].update(self, portion)
+ width -= len(widget)
+ result[index] = widget
+
+ return result
+
+
+ def _format_line(self):
+ """Joins the widgets and justifies the line."""
+
+ widgets = ''.join(self._format_widgets())
+
+ if self.left_justify: return widgets.ljust(self.term_width)
+ else: return widgets.rjust(self.term_width)
+
+
+ def _need_update(self):
+ """Returns whether the ProgressBar should redraw the line."""
+ if self.currval >= self.next_update or self.finished: return True
+
+ delta = time.time() - self.last_update_time
+ return self._time_sensitive and delta > self.poll
+
+
+ def _update_widgets(self):
+ """Checks all widgets for the time sensitive bit."""
+
+ self._time_sensitive = any(getattr(w, 'TIME_SENSITIVE', False)
+ for w in self.widgets)
+
+
+ def update(self, value=None):
+ """Updates the ProgressBar to a new value."""
+
+ if value is not None and value is not UnknownLength:
+ if (self.maxval is not UnknownLength
+ and not 0 <= value <= self.maxval):
+
+ raise ValueError('Value out of range')
+
+ self.currval = value
+
+
+ if not self._need_update(): return
+ if self.start_time is None:
+ raise RuntimeError('You must call "start" before calling "update"')
+
+ now = time.time()
+ self.seconds_elapsed = now - self.start_time
+ self.next_update = self.currval + self.update_interval
+ self.fd.write(self._format_line() + '\r')
+ self.fd.flush()
+ self.last_update_time = now
+
+
+ def start(self):
+ """Starts measuring time, and prints the bar at 0%.
+
+ It returns self so you can use it like this:
+ >>> pbar = ProgressBar().start()
+ >>> for i in range(100):
+ ... # do something
+ ... pbar.update(i+1)
+ ...
+ >>> pbar.finish()
+ """
+
+ if self.maxval is None:
+ self.maxval = self._DEFAULT_MAXVAL
+
+ self.num_intervals = max(100, self.term_width)
+ self.next_update = 0
+
+ if self.maxval is not UnknownLength:
+ if self.maxval < 0: raise ValueError('Value out of range')
+ self.update_interval = self.maxval / self.num_intervals
+
+
+ self.start_time = self.last_update_time = time.time()
+ self.update(0)
+
+ return self
+
+
+ def finish(self):
+ """Puts the ProgressBar bar in the finished state."""
+
+ if self.finished:
+ return
+ self.finished = True
+ self.update(self.maxval)
+ self.fd.write('\n')
+ if self.signal_set:
+ signal.signal(signal.SIGWINCH, signal.SIG_DFL)
diff --git a/lib/progressbar/widgets.py b/lib/progressbar/widgets.py
new file mode 100644
index 0000000..6434ad5
--- /dev/null
+++ b/lib/progressbar/widgets.py
@@ -0,0 +1,355 @@
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Default ProgressBar widgets."""
+
+from __future__ import division
+
+import datetime
+import math
+
+try:
+ from abc import ABCMeta, abstractmethod
+except ImportError:
+ AbstractWidget = object
+ abstractmethod = lambda fn: fn
+else:
+ AbstractWidget = ABCMeta('AbstractWidget', (object,), {})
+
+
+def format_updatable(updatable, pbar):
+ if hasattr(updatable, 'update'): return updatable.update(pbar)
+ else: return updatable
+
+
+class Widget(AbstractWidget):
+ """The base class for all widgets.
+
+ The ProgressBar will call the widget's update value when the widget should
+ be updated. The widget's size may change between calls, but the widget may
+ display incorrectly if the size changes drastically and repeatedly.
+
+ The boolean TIME_SENSITIVE informs the ProgressBar that it should be
+ updated more often because it is time sensitive.
+ """
+
+ TIME_SENSITIVE = False
+ __slots__ = ()
+
+ @abstractmethod
+ def update(self, pbar):
+ """Updates the widget.
+
+ pbar - a reference to the calling ProgressBar
+ """
+
+
+class WidgetHFill(Widget):
+ """The base class for all variable width widgets.
+
+ This widget is much like the \\hfill command in TeX, it will expand to
+ fill the line. You can use more than one in the same line, and they will
+ all have the same width, and together will fill the line.
+ """
+
+ @abstractmethod
+ def update(self, pbar, width):
+ """Updates the widget providing the total width the widget must fill.
+
+ pbar - a reference to the calling ProgressBar
+ width - The total width the widget must fill
+ """
+
+
+class Timer(Widget):
+ """Widget which displays the elapsed seconds."""
+
+ __slots__ = ('format_string',)
+ TIME_SENSITIVE = True
+
+ def __init__(self, format='Elapsed Time: %s'):
+ self.format_string = format
+
+ @staticmethod
+ def format_time(seconds):
+ """Formats time as the string "HH:MM:SS"."""
+
+ return str(datetime.timedelta(seconds=int(seconds)))
+
+
+ def update(self, pbar):
+ """Updates the widget to show the elapsed time."""
+
+ return self.format_string % self.format_time(pbar.seconds_elapsed)
+
+
+class ETA(Timer):
+ """Widget which attempts to estimate the time of arrival."""
+
+ TIME_SENSITIVE = True
+
+ def update(self, pbar):
+ """Updates the widget to show the ETA or total time when finished."""
+
+ if pbar.currval == 0:
+ return 'ETA: --:--:--'
+ elif pbar.finished:
+ return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
+ else:
+ elapsed = pbar.seconds_elapsed
+ eta = elapsed * pbar.maxval / pbar.currval - elapsed
+ return 'ETA: %s' % self.format_time(eta)
+
+
+class AdaptiveETA(Timer):
+ """Widget which attempts to estimate the time of arrival.
+
+ Uses a weighted average of two estimates:
+ 1) ETA based on the total progress and time elapsed so far
+ 2) ETA based on the progress as per the last 10 update reports
+
+ The weight depends on the current progress so that to begin with the
+ total progress is used and at the end only the most recent progress is
+ used.
+ """
+
+ TIME_SENSITIVE = True
+ NUM_SAMPLES = 10
+
+ def _update_samples(self, currval, elapsed):
+ sample = (currval, elapsed)
+ if not hasattr(self, 'samples'):
+ self.samples = [sample] * (self.NUM_SAMPLES + 1)
+ else:
+ self.samples.append(sample)
+ return self.samples.pop(0)
+
+ def _eta(self, maxval, currval, elapsed):
+ return elapsed * maxval / float(currval) - elapsed
+
+ def update(self, pbar):
+ """Updates the widget to show the ETA or total time when finished."""
+ if pbar.currval == 0:
+ return 'ETA: --:--:--'
+ elif pbar.finished:
+ return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
+ else:
+ elapsed = pbar.seconds_elapsed
+ currval1, elapsed1 = self._update_samples(pbar.currval, elapsed)
+ eta = self._eta(pbar.maxval, pbar.currval, elapsed)
+ if pbar.currval > currval1:
+ etasamp = self._eta(pbar.maxval - currval1,
+ pbar.currval - currval1,
+ elapsed - elapsed1)
+ weight = (pbar.currval / float(pbar.maxval)) ** 0.5
+ eta = (1 - weight) * eta + weight * etasamp
+ return 'ETA: %s' % self.format_time(eta)
+
+
+class FileTransferSpeed(Widget):
+ """Widget for showing the transfer speed (useful for file transfers)."""
+
+ FORMAT = '%6.2f %s%s/s'
+ PREFIXES = ' kMGTPEZY'
+ __slots__ = ('unit',)
+
+ def __init__(self, unit='B'):
+ self.unit = unit
+
+ def update(self, pbar):
+ """Updates the widget with the current SI prefixed speed."""
+
+ if pbar.seconds_elapsed < 2e-6 or pbar.currval < 2e-6: # =~ 0
+ scaled = power = 0
+ else:
+ speed = pbar.currval / pbar.seconds_elapsed
+ power = int(math.log(speed, 1000))
+ scaled = speed / 1000.**power
+
+ return self.FORMAT % (scaled, self.PREFIXES[power], self.unit)
+
+
+class AnimatedMarker(Widget):
+ """An animated marker for the progress bar which defaults to appear as if
+ it were rotating.
+ """
+
+ __slots__ = ('markers', 'curmark')
+
+ def __init__(self, markers='|/-\\'):
+ self.markers = markers
+ self.curmark = -1
+
+ def update(self, pbar):
+ """Updates the widget to show the next marker or the first marker when
+ finished"""
+
+ if pbar.finished: return self.markers[0]
+
+ self.curmark = (self.curmark + 1) % len(self.markers)
+ return self.markers[self.curmark]
+
+# Alias for backwards compatibility
+RotatingMarker = AnimatedMarker
+
+
+class Counter(Widget):
+ """Displays the current count."""
+
+ __slots__ = ('format_string',)
+
+ def __init__(self, format='%d'):
+ self.format_string = format
+
+ def update(self, pbar):
+ return self.format_string % pbar.currval
+
+
+class Percentage(Widget):
+ """Displays the current percentage as a number with a percent sign."""
+
+ def update(self, pbar):
+ return '%3d%%' % pbar.percentage()
+
+
+class FormatLabel(Timer):
+ """Displays a formatted label."""
+
+ mapping = {
+ 'elapsed': ('seconds_elapsed', Timer.format_time),
+ 'finished': ('finished', None),
+ 'last_update': ('last_update_time', None),
+ 'max': ('maxval', None),
+ 'seconds': ('seconds_elapsed', None),
+ 'start': ('start_time', None),
+ 'value': ('currval', None)
+ }
+
+ __slots__ = ('format_string',)
+ def __init__(self, format):
+ self.format_string = format
+
+ def update(self, pbar):
+ context = {}
+ for name, (key, transform) in self.mapping.items():
+ try:
+ value = getattr(pbar, key)
+
+ if transform is None:
+ context[name] = value
+ else:
+ context[name] = transform(value)
+ except: pass
+
+ return self.format_string % context
+
+
+class SimpleProgress(Widget):
+ """Returns progress as a count of the total (e.g.: "5 of 47")."""
+
+ __slots__ = ('sep',)
+
+ def __init__(self, sep=' of '):
+ self.sep = sep
+
+ def update(self, pbar):
+ return '%d%s%d' % (pbar.currval, self.sep, pbar.maxval)
+
+
+class Bar(WidgetHFill):
+ """A progress bar which stretches to fill the line."""
+
+ __slots__ = ('marker', 'left', 'right', 'fill', 'fill_left')
+
+ def __init__(self, marker='#', left='|', right='|', fill=' ',
+ fill_left=True):
+ """Creates a customizable progress bar.
+
+ marker - string or updatable object to use as a marker
+ left - string or updatable object to use as a left border
+ right - string or updatable object to use as a right border
+ fill - character to use for the empty part of the progress bar
+ fill_left - whether to fill from the left or the right
+ """
+ self.marker = marker
+ self.left = left
+ self.right = right
+ self.fill = fill
+ self.fill_left = fill_left
+
+
+ def update(self, pbar, width):
+ """Updates the progress bar and its subcomponents."""
+
+ left, marked, right = (format_updatable(i, pbar) for i in
+ (self.left, self.marker, self.right))
+
+ width -= len(left) + len(right)
+ # Marked must *always* have length of 1
+ if pbar.maxval:
+ marked *= int(pbar.currval / pbar.maxval * width)
+ else:
+ marked = ''
+
+ if self.fill_left:
+ return '%s%s%s' % (left, marked.ljust(width, self.fill), right)
+ else:
+ return '%s%s%s' % (left, marked.rjust(width, self.fill), right)
+
+
+class ReverseBar(Bar):
+ """A bar which has a marker which bounces from side to side."""
+
+ def __init__(self, marker='#', left='|', right='|', fill=' ',
+ fill_left=False):
+ """Creates a customizable progress bar.
+
+ marker - string or updatable object to use as a marker
+ left - string or updatable object to use as a left border
+ right - string or updatable object to use as a right border
+ fill - character to use for the empty part of the progress bar
+ fill_left - whether to fill from the left or the right
+ """
+ self.marker = marker
+ self.left = left
+ self.right = right
+ self.fill = fill
+ self.fill_left = fill_left
+
+
+class BouncingBar(Bar):
+ def update(self, pbar, width):
+ """Updates the progress bar and its subcomponents."""
+
+ left, marker, right = (format_updatable(i, pbar) for i in
+ (self.left, self.marker, self.right))
+
+ width -= len(left) + len(right)
+
+ if pbar.finished: return '%s%s%s' % (left, width * marker, right)
+
+ position = int(pbar.currval % (width * 2 - 1))
+ if position > width: position = width * 2 - position
+ lpad = self.fill * (position - 1)
+ rpad = self.fill * (width - len(marker) - len(lpad))
+
+ # Swap if we want to bounce the other way
+ if not self.fill_left: rpad, lpad = lpad, rpad
+
+ return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 03/10] lib: implement basic task progress support
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
2016-06-13 2:52 ` [PATCH 01/10] knotty: provide a symlink to the latest console log Paul Eggleton
2016-06-13 2:52 ` [PATCH 02/10] knotty: import latest python-progressbar Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-13 2:52 ` [PATCH 04/10] lib/bb/progress: add MultiStageProgressReporter Paul Eggleton
` (6 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
For long-running tasks where we have some output from the task that
gives us some idea of the progress of the task (such as a percentage
complete), provide the means to scrape the output for that progress
information and show it to the user in the default knotty terminal
output in the form of a progress bar. This is implemented using a new
TaskProgress event as well as some code we can insert to do output
scanning/filtering.
Any task can fire TaskProgress events; however, if you have a shell task
whose output you wish to scan for progress information, you just need to
set the "progress" varflag on the task. This can be set to:
* "percent" to just look for a number followed by a % sign
* "percent:<regex>" to specify your own regex matching a percentage
value (must have a single group which matches the percentage number)
* "outof:<regex>" to look for the specified regex matching x out of y
items completed (must have two groups - first group needs to be x,
second y).
We can potentially extend this in future but this should be a good
start.
Part of the implementation for [YOCTO #5383].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/build.py | 34 +++++++++++++++++
lib/bb/progress.py | 86 ++++++++++++++++++++++++++++++++++++++++++
lib/bb/ui/knotty.py | 74 +++++++++++++++++++++++++++++++-----
lib/bb/ui/uihelper.py | 7 +++-
lib/progressbar/progressbar.py | 16 ++++++--
lib/progressbar/widgets.py | 36 ++++++++++++++++++
6 files changed, 239 insertions(+), 14 deletions(-)
create mode 100644 lib/bb/progress.py
diff --git a/lib/bb/build.py b/lib/bb/build.py
index e016ae3..04bd580 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -35,6 +35,7 @@ import stat
import bb
import bb.msg
import bb.process
+import bb.progress
from bb import data, event, utils
bblogger = logging.getLogger('BitBake')
@@ -137,6 +138,25 @@ class TaskInvalid(TaskBase):
super(TaskInvalid, self).__init__(task, None, metadata)
self._message = "No such task '%s'" % task
+class TaskProgress(event.Event):
+ """
+ Task made some progress that could be reported to the user, usually in
+ the form of a progress bar or similar.
+ NOTE: this class does not inherit from TaskBase since it doesn't need
+ to - it's fired within the task context itself, so we don't have any of
+ the context information that you do in the case of the other events.
+ The event PID can be used to determine which task it came from.
+ The progress value is normally 0-100, but can also be negative
+ indicating that progress has been made but we aren't able to determine
+ how much.
+ The rate is optional, this is simply an extra string to display to the
+ user if specified.
+ """
+ def __init__(self, progress, rate=None):
+ self.progress = progress
+ self.rate = rate
+ event.Event.__init__(self)
+
class LogTee(object):
def __init__(self, logger, outfile):
@@ -340,6 +360,20 @@ exit $ret
else:
logfile = sys.stdout
+ progress = d.getVarFlag(func, 'progress', True)
+ if progress:
+ if progress == 'percent':
+ # Use default regex
+ logfile = bb.progress.BasicProgressHandler(d, outfile=logfile)
+ elif progress.startswith('percent:'):
+ # Use specified regex
+ logfile = bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+ elif progress.startswith('outof:'):
+ # Use specified regex
+ logfile = bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+ else:
+ bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
+
def readfifo(data):
lines = data.split(b'\0')
for line in lines:
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
new file mode 100644
index 0000000..bab8e94
--- /dev/null
+++ b/lib/bb/progress.py
@@ -0,0 +1,86 @@
+"""
+BitBake progress handling code
+"""
+
+# Copyright (C) 2016 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import sys
+import re
+import time
+import bb.event
+import bb.build
+
+class ProgressHandler(object):
+ """
+ Base class that can pretend to be a file object well enough to be
+ used to build objects to intercept console output and determine the
+ progress of some operation.
+ """
+ def __init__(self, d, outfile=None):
+ self._progress = 0
+ self._data = d
+ self._lastevent = 0
+ if outfile:
+ self._outfile = outfile
+ else:
+ self._outfile = sys.stdout
+
+ def _fire_progress(self, taskprogress, rate=None):
+ """Internal function to fire the progress event"""
+ bb.event.fire(bb.build.TaskProgress(taskprogress, rate), self._data)
+
+ def write(self, string):
+ self._outfile.write(string)
+
+ def flush(self):
+ self._outfile.flush()
+
+ def update(self, progress, rate=None):
+ ts = time.time()
+ if progress > 100:
+ progress = 100
+ if progress != self._progress or self._lastevent + 1 < ts:
+ self._fire_progress(progress, rate)
+ self._lastevent = ts
+ self._progress = progress
+
+class BasicProgressHandler(ProgressHandler):
+ def __init__(self, d, regex=r'(\d+)%', outfile=None):
+ super(BasicProgressHandler, self).__init__(d, outfile)
+ self._regex = re.compile(regex)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(0)
+
+ def write(self, string):
+ percs = self._regex.findall(string)
+ if percs:
+ progress = int(percs[-1])
+ self.update(progress)
+ super(BasicProgressHandler, self).write(string)
+
+class OutOfProgressHandler(ProgressHandler):
+ def __init__(self, d, regex, outfile=None):
+ super(OutOfProgressHandler, self).__init__(d, outfile)
+ self._regex = re.compile(regex)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(0)
+
+ def write(self, string):
+ nums = self._regex.findall(string)
+ if nums:
+ progress = (float(nums[-1][0]) / float(nums[-1][1])) * 100
+ self.update(progress)
+ super(OutOfProgressHandler, self).write(string)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 6a6f688..2513501 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -40,10 +40,13 @@ logger = logging.getLogger("BitBake")
interactive = sys.stdout.isatty()
class BBProgress(progressbar.ProgressBar):
- def __init__(self, msg, maxval):
+ def __init__(self, msg, maxval, widgets=None):
self.msg = msg
- widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
- progressbar.ETA()]
+ self.extrapos = -1
+ if not widgets:
+ widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
+ progressbar.ETA()]
+ self.extrapos = 4
try:
self._resize_default = signal.getsignal(signal.SIGWINCH)
@@ -55,11 +58,31 @@ class BBProgress(progressbar.ProgressBar):
progressbar.ProgressBar._handle_resize(self, signum, frame)
if self._resize_default:
self._resize_default(signum, frame)
+
def finish(self):
progressbar.ProgressBar.finish(self)
if self._resize_default:
signal.signal(signal.SIGWINCH, self._resize_default)
+ def setmessage(self, msg):
+ self.msg = msg
+ self.widgets[0] = msg
+
+ def setextra(self, extra):
+ if extra:
+ extrastr = str(extra)
+ if extrastr[0] != ' ':
+ extrastr = ' ' + extrastr
+ if extrastr[-1] != ' ':
+ extrastr += ' '
+ else:
+ extrastr = ' '
+ self.widgets[self.extrapos] = extrastr
+
+ def _need_update(self):
+ # We always want the bar to print when update() is called
+ return True
+
class NonInteractiveProgress(object):
fobj = sys.stdout
@@ -195,15 +218,31 @@ class TerminalFilter(object):
activetasks = self.helper.running_tasks
failedtasks = self.helper.failed_tasks
runningpids = self.helper.running_pids
- if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
+ if self.footer_present and not self.helper.needUpdate:
return
+ self.helper.needUpdate = False
if self.footer_present:
self.clearFooter()
if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
return
tasks = []
for t in runningpids:
- tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+ progress = activetasks[t].get("progress", None)
+ if progress is not None:
+ pbar = activetasks[t].get("progressbar", None)
+ rate = activetasks[t].get("rate", None)
+ start_time = activetasks[t].get("starttime", None)
+ if not pbar or pbar.bouncing != (progress < 0):
+ if progress < 0:
+ pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100, widgets=[progressbar.BouncingSlider()])
+ pbar.bouncing = True
+ else:
+ pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100)
+ pbar.bouncing = False
+ activetasks[t]["progressbar"] = pbar
+ tasks.append((pbar, progress, rate, start_time))
+ else:
+ tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
if self.main.shutdown:
content = "Waiting for %s running tasks to finish:" % len(activetasks)
@@ -214,8 +253,23 @@ class TerminalFilter(object):
print(content)
lines = 1 + int(len(content) / (self.columns + 1))
for tasknum, task in enumerate(tasks[:(self.rows - 2)]):
- content = "%s: %s" % (tasknum, task)
- print(content)
+ if isinstance(task, tuple):
+ pbar, progress, rate, start_time = task
+ if not pbar.start_time:
+ pbar.start(False)
+ if start_time:
+ pbar.start_time = start_time
+ pbar.setmessage('%s:%s' % (tasknum, pbar.msg.split(':', 1)[1]))
+ if progress > -1:
+ pbar.setextra(rate)
+ output = pbar.update(progress)
+ else:
+ output = pbar.update(1)
+ if not output or (len(output) <= pbar.term_width):
+ print('')
+ else:
+ content = "%s: %s" % (tasknum, task)
+ print(content)
lines = lines + 1 + int(len(content) / (self.columns + 1))
self.footer_present = lines
self.lastpids = runningpids[:]
@@ -249,7 +303,8 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
"bb.command.CommandExit", "bb.command.CommandCompleted", "bb.cooker.CookerExit",
"bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted",
"bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed",
- "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent"]
+ "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
+ "bb.build.TaskProgress"]
def main(server, eventHandler, params, tf = TerminalFilter):
@@ -535,7 +590,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
bb.event.OperationStarted,
bb.event.OperationCompleted,
bb.event.OperationProgress,
- bb.event.DiskFull)):
+ bb.event.DiskFull,
+ bb.build.TaskProgress)):
continue
logger.error("Unknown event: %s", event)
diff --git a/lib/bb/ui/uihelper.py b/lib/bb/ui/uihelper.py
index db70b76..1915e47 100644
--- a/lib/bb/ui/uihelper.py
+++ b/lib/bb/ui/uihelper.py
@@ -18,6 +18,7 @@
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import bb.build
+import time
class BBUIHelper:
def __init__(self):
@@ -31,7 +32,7 @@ class BBUIHelper:
def eventHandler(self, event):
if isinstance(event, bb.build.TaskStarted):
- self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task) }
+ self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time() }
self.running_pids.append(event.pid)
self.needUpdate = True
if isinstance(event, bb.build.TaskSucceeded):
@@ -52,6 +53,10 @@ class BBUIHelper:
self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
self.tasknumber_total = event.stats.total
self.needUpdate = True
+ if isinstance(event, bb.build.TaskProgress):
+ self.running_tasks[event.pid]['progress'] = event.progress
+ self.running_tasks[event.pid]['rate'] = event.rate
+ self.needUpdate = True
def getTasks(self):
self.needUpdate = False
diff --git a/lib/progressbar/progressbar.py b/lib/progressbar/progressbar.py
index 0b9dcf7..2873ad6 100644
--- a/lib/progressbar/progressbar.py
+++ b/lib/progressbar/progressbar.py
@@ -3,6 +3,8 @@
# progressbar - Text progress bar library for Python.
# Copyright (c) 2005 Nilton Volpato
#
+# (With some small changes after importing into BitBake)
+#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
@@ -261,12 +263,14 @@ class ProgressBar(object):
now = time.time()
self.seconds_elapsed = now - self.start_time
self.next_update = self.currval + self.update_interval
- self.fd.write(self._format_line() + '\r')
+ output = self._format_line()
+ self.fd.write(output + '\r')
self.fd.flush()
self.last_update_time = now
+ return output
- def start(self):
+ def start(self, update=True):
"""Starts measuring time, and prints the bar at 0%.
It returns self so you can use it like this:
@@ -289,8 +293,12 @@ class ProgressBar(object):
self.update_interval = self.maxval / self.num_intervals
- self.start_time = self.last_update_time = time.time()
- self.update(0)
+ self.start_time = time.time()
+ if update:
+ self.last_update_time = self.start_time
+ self.update(0)
+ else:
+ self.last_update_time = 0
return self
diff --git a/lib/progressbar/widgets.py b/lib/progressbar/widgets.py
index 6434ad5..77285ca 100644
--- a/lib/progressbar/widgets.py
+++ b/lib/progressbar/widgets.py
@@ -353,3 +353,39 @@ class BouncingBar(Bar):
if not self.fill_left: rpad, lpad = lpad, rpad
return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
+
+
+class BouncingSlider(Bar):
+ """
+ A slider that bounces back and forth in response to update() calls
+ without reference to the actual value. Based on a combination of
+ BouncingBar from a newer version of this module and RotatingMarker.
+ """
+ def __init__(self, marker='<=>'):
+ self.curmark = -1
+ self.forward = True
+ Bar.__init__(self, marker=marker)
+ def update(self, pbar, width):
+ left, marker, right = (format_updatable(i, pbar) for i in
+ (self.left, self.marker, self.right))
+
+ width -= len(left) + len(right)
+ if width < 0:
+ return ''
+
+ if pbar.finished: return '%s%s%s' % (left, width * '=', right)
+
+ self.curmark = self.curmark + 1
+ position = int(self.curmark % (width * 2 - 1))
+ if position + len(marker) > width:
+ self.forward = not self.forward
+ self.curmark = 1
+ position = 1
+ lpad = ' ' * (position - 1)
+ rpad = ' ' * (width - len(marker) - len(lpad))
+
+ if not self.forward:
+ temp = lpad
+ lpad = rpad
+ rpad = temp
+ return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 04/10] lib/bb/progress: add MultiStageProgressReporter
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
` (2 preceding siblings ...)
2016-06-13 2:52 ` [PATCH 03/10] lib: implement basic task progress support Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-13 2:52 ` [PATCH 05/10] fetch2: implement progress support Paul Eggleton
` (5 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
Add a class to help report progress in a task that consists of multiple
stages, some of which may have internal progress (do_rootfs within
OpenEmbedded is one example). Each stage is weighted to try to give
a reasonable representation of progress over time.
Part of the implementation for [YOCTO #5383].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/progress.py | 107 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 107 insertions(+)
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
index bab8e94..7ee9f04 100644
--- a/lib/bb/progress.py
+++ b/lib/bb/progress.py
@@ -20,6 +20,7 @@ BitBake progress handling code
import sys
import re
import time
+import inspect
import bb.event
import bb.build
@@ -84,3 +85,109 @@ class OutOfProgressHandler(ProgressHandler):
progress = (float(nums[-1][0]) / float(nums[-1][1])) * 100
self.update(progress)
super(OutOfProgressHandler, self).write(string)
+
+class MultiStageProgressReporter(object):
+ """
+ Class which allows reporting progress without the caller
+ having to know where they are in the overall sequence. Useful
+ for tasks made up of python code spread across multiple
+ classes / functions - the progress reporter object can
+ be passed around or stored at the object level and calls
+ to next_stage() and update() made whereever needed.
+ """
+ def __init__(self, d, stage_weights, debug=False):
+ """
+ Initialise the progress reporter.
+
+ Parameters:
+ * d: the datastore (needed for firing the events)
+ * stage_weights: a list of weight values, one for each stage.
+ The value is scaled internally so you only need to specify
+ values relative to other values in the list, so if there
+ are two stages and the first takes 2s and the second takes
+ 10s you would specify [2, 10] (or [1, 5], it doesn't matter).
+ * debug: specify True (and ensure you call finish() at the end)
+ in order to show a printout of the calculated stage weights
+ based on timing each stage. Use this to determine what the
+ weights should be when you're not sure.
+ """
+ self._data = d
+ total = sum(stage_weights)
+ self._stage_weights = [float(x)/total for x in stage_weights]
+ self._stage = -1
+ self._base_progress = 0
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(0)
+ self._debug = debug
+ if self._debug:
+ self._last_time = time.time()
+ self._stage_times = []
+ self._stage_total = None
+ self._callers = []
+
+ def _fire_progress(self, taskprogress):
+ bb.event.fire(bb.build.TaskProgress(taskprogress), self._data)
+
+ def next_stage(self, stage_total=None):
+ """
+ Move to the next stage.
+ Parameters:
+ * stage_total: optional total for progress within the stage,
+ see update() for details
+ NOTE: you need to call this before the first stage.
+ """
+ self._stage += 1
+ self._stage_total = stage_total
+ if self._stage == 0:
+ # First stage
+ if self._debug:
+ self._last_time = time.time()
+ else:
+ if self._stage < len(self._stage_weights):
+ self._base_progress = sum(self._stage_weights[:self._stage]) * 100
+ if self._debug:
+ currtime = time.time()
+ self._stage_times.append(currtime - self._last_time)
+ self._last_time = currtime
+ self._callers.append(inspect.getouterframes(inspect.currentframe())[1])
+ elif not self._debug:
+ bb.warn('ProgressReporter: current stage beyond declared number of stages')
+ self._base_progress = 100
+ self._fire_progress(self._base_progress)
+
+ def update(self, stage_progress):
+ """
+ Update progress within the current stage.
+ Parameters:
+ * stage_progress: progress value within the stage. If stage_total
+ was specified when next_stage() was last called, then this
+ value is considered to be out of stage_total, otherwise it should
+ be a percentage value from 0 to 100.
+ """
+ if self._stage_total:
+ stage_progress = (float(stage_progress) / self._stage_total) * 100
+ if self._stage < 0:
+ bb.warn('ProgressReporter: update called before first call to next_stage()')
+ elif self._stage < len(self._stage_weights):
+ progress = self._base_progress + (stage_progress * self._stage_weights[self._stage])
+ else:
+ progress = self._base_progress
+ if progress > 100:
+ progress = 100
+ self._fire_progress(progress)
+
+ def finish(self):
+ if self._debug:
+ import math
+ self._stage_times.append(time.time() - self._last_time)
+ mintime = max(min(self._stage_times), 0.01)
+ self._callers.append(None)
+ stage_weights = [int(math.ceil(x / mintime)) for x in self._stage_times]
+ bb.warn('Stage weights: %s' % stage_weights)
+ out = []
+ for stage_weight, caller in zip(stage_weights, self._callers):
+ if caller:
+ out.append('Up to %s:%d: %d' % (caller[1], caller[2], stage_weight))
+ else:
+ out.append('Up to finish: %d' % stage_weight)
+ bb.warn('Stage times:\n %s' % '\n '.join(out))
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 05/10] fetch2: implement progress support
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
` (3 preceding siblings ...)
2016-06-13 2:52 ` [PATCH 04/10] lib/bb/progress: add MultiStageProgressReporter Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-28 10:23 ` Burton, Ross
2016-06-13 2:52 ` [PATCH 06/10] knotty: add code to support showing progress for sstate object querying Paul Eggleton
` (4 subsequent siblings)
9 siblings, 1 reply; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.
At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).
Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.
Part of the implementation for [YOCTO #5383].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/fetch2/__init__.py | 4 ++--
lib/bb/fetch2/git.py | 52 +++++++++++++++++++++++++++++++++++++++++++----
lib/bb/fetch2/wget.py | 26 +++++++++++++++++++++++-
lib/bb/progress.py | 31 ++++++++++++++++++++++++++++
4 files changed, 106 insertions(+), 7 deletions(-)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index f612318..8ba00e0 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -779,7 +779,7 @@ def localpath(url, d):
fetcher = bb.fetch2.Fetch([url], d)
return fetcher.localpath(url)
-def runfetchcmd(cmd, d, quiet=False, cleanup=None):
+def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None):
"""
Run cmd returning the command output
Raise an error if interrupted or cmd fails
@@ -820,7 +820,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None):
error_message = ""
try:
- (output, errors) = bb.process.run(cmd, shell=True, stderr=subprocess.PIPE)
+ (output, errors) = bb.process.run(cmd, log=log, shell=True, stderr=subprocess.PIPE)
success = True
except bb.process.NotFoundError as e:
error_message = "Fetch command %s" % (e.command)
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 59827e3..4e2dcec 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -71,11 +71,53 @@ import os
import re
import bb
import errno
+import bb.progress
from bb import data
from bb.fetch2 import FetchMethod
from bb.fetch2 import runfetchcmd
from bb.fetch2 import logger
+
+class GitProgressHandler(bb.progress.LineFilterProgressHandler):
+ """Extract progress information from git output"""
+ def __init__(self, d):
+ self._buffer = ''
+ self._count = 0
+ super(GitProgressHandler, self).__init__(d)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(-1)
+
+ def write(self, string):
+ self._buffer += string
+ stages = ['Counting objects', 'Compressing objects', 'Receiving objects', 'Resolving deltas']
+ stage_weights = [0.2, 0.05, 0.5, 0.25]
+ stagenum = 0
+ for i, stage in reversed(list(enumerate(stages))):
+ if stage in self._buffer:
+ stagenum = i
+ self._buffer = ''
+ break
+ self._status = stages[stagenum]
+ percs = re.findall(r'(\d+)%', string)
+ if percs:
+ progress = int(round((int(percs[-1]) * stage_weights[stagenum]) + (sum(stage_weights[:stagenum]) * 100)))
+ rates = re.findall(r'([\d.]+ [a-zA-Z]*/s+)', string)
+ if rates:
+ rate = rates[-1]
+ else:
+ rate = None
+ self.update(progress, rate)
+ else:
+ if stagenum == 0:
+ percs = re.findall(r': (\d+)', string)
+ if percs:
+ count = int(percs[-1])
+ if count > self._count:
+ self._count = count
+ self._fire_progress(-count)
+ super(GitProgressHandler, self).write(string)
+
+
class Git(FetchMethod):
"""Class to fetch a module or modules from git repositories"""
def init(self, d):
@@ -196,10 +238,11 @@ class Git(FetchMethod):
# We do this since git will use a "-l" option automatically for local urls where possible
if repourl.startswith("file://"):
repourl = repourl[7:]
- clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
+ clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, repourl, ud.clonedir)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, clone_cmd)
- runfetchcmd(clone_cmd, d)
+ progresshandler = GitProgressHandler(d)
+ runfetchcmd(clone_cmd, d, log=progresshandler)
os.chdir(ud.clonedir)
# Update the checkout if needed
@@ -214,10 +257,11 @@ class Git(FetchMethod):
logger.debug(1, "No Origin")
runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, repourl), d)
- fetch_cmd = "%s fetch -f --prune %s refs/*:refs/*" % (ud.basecmd, repourl)
+ fetch_cmd = "LANG=C %s fetch -f --prune --progress %s refs/*:refs/*" % (ud.basecmd, repourl)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
- runfetchcmd(fetch_cmd, d)
+ progresshandler = GitProgressHandler(d)
+ runfetchcmd(fetch_cmd, d, log=progresshandler)
runfetchcmd("%s prune-packed" % ud.basecmd, d)
runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
try:
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index d688fd9..3a381b7 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -31,6 +31,7 @@ import subprocess
import os
import logging
import bb
+import bb.progress
import urllib.request, urllib.parse, urllib.error
from bb import data
from bb.fetch2 import FetchMethod
@@ -41,6 +42,27 @@ from bb.utils import export_proxies
from bs4 import BeautifulSoup
from bs4 import SoupStrainer
+class WgetProgressHandler(bb.progress.LineFilterProgressHandler):
+ """
+ Extract progress information from wget output.
+ Note: relies on --progress=dot --show-progress being specified on the
+ wget command line.
+ """
+ def __init__(self, d):
+ super(WgetProgressHandler, self).__init__(d)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(0)
+
+ def writeline(self, line):
+ percs = re.findall(r'(\d+)%\s+([\d.]+[A-Z])', line)
+ if percs:
+ progress = int(percs[-1][0])
+ rate = percs[-1][1] + '/s'
+ self.update(progress, rate)
+ return False
+ return True
+
+
class Wget(FetchMethod):
"""Class to fetch urls via 'wget'"""
def supports(self, ud, d):
@@ -70,9 +92,11 @@ class Wget(FetchMethod):
def _runwget(self, ud, d, command, quiet):
+ progresshandler = WgetProgressHandler(d)
+
logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command))
bb.fetch2.check_network_access(d, command)
- runfetchcmd(command, d, quiet)
+ runfetchcmd(command + ' --progress=dot --show-progress', d, quiet, log=progresshandler)
def download(self, ud, d):
"""Fetch urls"""
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
index 7ee9f04..73e3f05 100644
--- a/lib/bb/progress.py
+++ b/lib/bb/progress.py
@@ -58,6 +58,37 @@ class ProgressHandler(object):
self._lastevent = ts
self._progress = progress
+class LineFilterProgressHandler(ProgressHandler):
+ """
+ A ProgressHandler variant that provides the ability to filter out
+ the lines if they contain progress information. Additionally, it
+ filters out anything before the last line feed on a line. This can
+ be used to keep the logs clean of output that we've only enabled for
+ getting progress, assuming that that can be done on a per-line
+ basis.
+ """
+ def __init__(self, d, outfile=None):
+ self._linebuffer = ''
+ super(LineFilterProgressHandler, self).__init__(d, outfile)
+
+ def write(self, string):
+ self._linebuffer += string
+ while True:
+ breakpos = self._linebuffer.find('\n') + 1
+ if breakpos == 0:
+ break
+ line = self._linebuffer[:breakpos]
+ self._linebuffer = self._linebuffer[breakpos:]
+ # Drop any line feeds and anything that precedes them
+ lbreakpos = line.rfind('\r') + 1
+ if lbreakpos:
+ line = line[lbreakpos:]
+ if self.writeline(line):
+ super(LineFilterProgressHandler, self).write(line)
+
+ def writeline(self, line):
+ return True
+
class BasicProgressHandler(ProgressHandler):
def __init__(self, d, regex=r'(\d+)%', outfile=None):
super(BasicProgressHandler, self).__init__(d, outfile)
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 06/10] knotty: add code to support showing progress for sstate object querying
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
` (4 preceding siblings ...)
2016-06-13 2:52 ` [PATCH 05/10] fetch2: implement progress support Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-13 2:52 ` [PATCH 07/10] knotty: show task progress bar Paul Eggleton
` (3 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
Add support code on the BitBake side to allow sstate.bbclass in
OpenEmbedded to report progress when it is checking for availability of
artifacts from shared state mirrors.
Part of the implementation for [YOCTO #5853].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/event.py | 27 +++++++++++++++++++++++++++
lib/bb/ui/knotty.py | 15 +++++++++++++--
2 files changed, 40 insertions(+), 2 deletions(-)
diff --git a/lib/bb/event.py b/lib/bb/event.py
index 6fb3712..3485d73 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -640,6 +640,33 @@ class MetadataEvent(Event):
self.type = eventtype
self._localdata = eventdata
+class ProcessStarted(Event):
+ """
+ Generic process started event (usually part of the initial startup)
+ where further progress events will be delivered
+ """
+ def __init__(self, processname, total):
+ Event.__init__(self)
+ self.processname = processname
+ self.total = total
+
+class ProcessProgress(Event):
+ """
+ Generic process progress event (usually part of the initial startup)
+ """
+ def __init__(self, processname, progress):
+ Event.__init__(self)
+ self.processname = processname
+ self.progress = progress
+
+class ProcessFinished(Event):
+ """
+ Generic process finished event (usually part of the initial startup)
+ """
+ def __init__(self, processname):
+ Event.__init__(self)
+ self.processname = processname
+
class SanityCheck(Event):
"""
Event to run sanity checks, either raise errors or generate events as return status.
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 2513501..6fdaafe 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -90,7 +90,7 @@ class NonInteractiveProgress(object):
self.msg = msg
self.maxval = maxval
- def start(self):
+ def start(self, update=True):
self.fobj.write("%s..." % self.msg)
self.fobj.flush()
return self
@@ -304,7 +304,7 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
"bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted",
"bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed",
"bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
- "bb.build.TaskProgress"]
+ "bb.build.TaskProgress", "bb.event.ProcessStarted", "bb.event.ProcessProgress", "bb.event.ProcessFinished"]
def main(server, eventHandler, params, tf = TerminalFilter):
@@ -579,6 +579,17 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if isinstance(event, bb.event.DepTreeGenerated):
continue
+ if isinstance(event, bb.event.ProcessStarted):
+ parseprogress = new_progress(event.processname, event.total)
+ parseprogress.start(False)
+ continue
+ if isinstance(event, bb.event.ProcessProgress):
+ parseprogress.update(event.progress)
+ continue
+ if isinstance(event, bb.event.ProcessFinished):
+ parseprogress.finish()
+ continue
+
# ignore
if isinstance(event, (bb.event.BuildBase,
bb.event.MetadataEvent,
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 07/10] knotty: show task progress bar
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
` (5 preceding siblings ...)
2016-06-13 2:52 ` [PATCH 06/10] knotty: add code to support showing progress for sstate object querying Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-13 2:52 ` [PATCH 08/10] knotty: add quiet output mode Paul Eggleton
` (2 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
In addition to the "currently running n tasks (x of y)" message, show a
progress bar for another view on how much of the build is left. We have
to take care to reset it when moving from the scenequeue to the
runqueue, and explicitly don't include an ETA since not all tasks take
equal time and thus it isn't possible to estimate the time remaining
with the information available.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/ui/knotty.py | 23 +++++++++++++++++++----
1 file changed, 19 insertions(+), 4 deletions(-)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 6fdaafe..c245c22 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -89,6 +89,7 @@ class NonInteractiveProgress(object):
def __init__(self, msg, maxval):
self.msg = msg
self.maxval = maxval
+ self.finished = False
def start(self, update=True):
self.fobj.write("%s..." % self.msg)
@@ -99,8 +100,11 @@ class NonInteractiveProgress(object):
pass
def finish(self):
+ if self.finished:
+ return
self.fobj.write("done.\n")
self.fobj.flush()
+ self.finished = True
def new_progress(msg, maxval):
if interactive:
@@ -204,6 +208,8 @@ class TerminalFilter(object):
console.addFilter(InteractConsoleLogFilter(self, format))
errconsole.addFilter(InteractConsoleLogFilter(self, format))
+ self.main_progress = None
+
def clearFooter(self):
if self.footer_present:
lines = self.footer_present
@@ -246,11 +252,20 @@ class TerminalFilter(object):
if self.main.shutdown:
content = "Waiting for %s running tasks to finish:" % len(activetasks)
- elif not len(activetasks):
- content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+ print(content)
else:
- content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
- print(content)
+ if not len(activetasks):
+ content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+ else:
+ content = "Currently %2s running tasks (%s of %s)" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
+ maxtask = self.helper.tasknumber_total + 1
+ if not self.main_progress or self.main_progress.maxval != maxtask:
+ widgets = [' ', progressbar.Percentage(), ' ', progressbar.Bar()]
+ self.main_progress = BBProgress("Running tasks", maxtask, widgets=widgets)
+ self.main_progress.start(False)
+ self.main_progress.setmessage(content)
+ self.main_progress.update(self.helper.tasknumber_current)
+ print('')
lines = 1 + int(len(content) / (self.columns + 1))
for tasknum, task in enumerate(tasks[:(self.rows - 2)]):
if isinstance(task, tuple):
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 08/10] knotty: add quiet output mode
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
` (6 preceding siblings ...)
2016-06-13 2:52 ` [PATCH 07/10] knotty: show task progress bar Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-13 2:52 ` [PATCH 09/10] runqueue: add ability to enforce that tasks are setscened Paul Eggleton
2016-06-13 2:52 ` [PATCH 10/10] runqueue: report progress for "Preparing RunQueue" step Paul Eggleton
9 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
Quiet output mode disables printing most messages (below warnings) to
the console; however these messages still go to the console log file.
This is primarily for cases where bitbake is being launched
interactively from some other process, but where full console output is
not needed.
Because of the need to keep logging all normal events to the console
log, this functionality was implemented within the knotty UI rather
than in bb.msg (where verbose mode is implemented). We don't currently
have a means of registering command line options from the UI end, thus
the option actually has to be registered in main.py regardless of the
UI, however I didn't feel like it was worth setting up such a mechanism
just for this option.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/main.py | 9 +++++++++
lib/bb/msg.py | 5 ++++-
lib/bb/ui/knotty.py | 57 ++++++++++++++++++++++++++++++-----------------------
3 files changed, 45 insertions(+), 26 deletions(-)
diff --git a/lib/bb/main.py b/lib/bb/main.py
index 283f29b..3fc3ff5 100755
--- a/lib/bb/main.py
+++ b/lib/bb/main.py
@@ -179,6 +179,9 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
parser.add_option("-D", "--debug", action="count", dest="debug", default=0,
help="Increase the debug level. You can specify this more than once.")
+ parser.add_option("-q", "--quiet", action="store_true", dest="quiet", default=False,
+ help="Output less log message data to the terminal.")
+
parser.add_option("-n", "--dry-run", action="store_true", dest="dry_run", default=False,
help="Don't execute, just go through the motions.")
@@ -279,6 +282,12 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
options, targets = parser.parse_args(argv)
+ if options.quiet and options.verbose:
+ parser.error("options --quiet and --verbose are mutually exclusive")
+
+ if options.quiet and options.debug:
+ parser.error("options --quiet and --debug are mutually exclusive")
+
# use configuration files from environment variables
if "BBPRECONF" in os.environ:
options.prefile.append(os.environ["BBPRECONF"])
diff --git a/lib/bb/msg.py b/lib/bb/msg.py
index 8c3ab47..b7c39fa 100644
--- a/lib/bb/msg.py
+++ b/lib/bb/msg.py
@@ -182,9 +182,12 @@ def constructLogOptions():
debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
return level, debug_domains
-def addDefaultlogFilter(handler, cls = BBLogFilter):
+def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
level, debug_domains = constructLogOptions()
+ if forcelevel is not None:
+ level = forcelevel
+
cls(handler, level, debug_domains)
#
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index c245c22..dbcb9c4 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -161,7 +161,7 @@ class TerminalFilter(object):
cr = (25, 80)
return cr
- def __init__(self, main, helper, console, errconsole, format):
+ def __init__(self, main, helper, console, errconsole, format, quiet):
self.main = main
self.helper = helper
self.cuu = None
@@ -169,6 +169,7 @@ class TerminalFilter(object):
self.interactive = sys.stdout.isatty()
self.footer_present = False
self.lastpids = []
+ self.quiet = quiet
if not self.interactive:
return
@@ -267,25 +268,26 @@ class TerminalFilter(object):
self.main_progress.update(self.helper.tasknumber_current)
print('')
lines = 1 + int(len(content) / (self.columns + 1))
- for tasknum, task in enumerate(tasks[:(self.rows - 2)]):
- if isinstance(task, tuple):
- pbar, progress, rate, start_time = task
- if not pbar.start_time:
- pbar.start(False)
- if start_time:
- pbar.start_time = start_time
- pbar.setmessage('%s:%s' % (tasknum, pbar.msg.split(':', 1)[1]))
- if progress > -1:
- pbar.setextra(rate)
- output = pbar.update(progress)
+ if not self.quiet:
+ for tasknum, task in enumerate(tasks[:(self.rows - 2)]):
+ if isinstance(task, tuple):
+ pbar, progress, rate, start_time = task
+ if not pbar.start_time:
+ pbar.start(False)
+ if start_time:
+ pbar.start_time = start_time
+ pbar.setmessage('%s:%s' % (tasknum, pbar.msg.split(':', 1)[1]))
+ if progress > -1:
+ pbar.setextra(rate)
+ output = pbar.update(progress)
+ else:
+ output = pbar.update(1)
+ if not output or (len(output) <= pbar.term_width):
+ print('')
else:
- output = pbar.update(1)
- if not output or (len(output) <= pbar.term_width):
- print('')
- else:
- content = "%s: %s" % (tasknum, task)
- print(content)
- lines = lines + 1 + int(len(content) / (self.columns + 1))
+ content = "%s: %s" % (tasknum, task)
+ print(content)
+ lines = lines + 1 + int(len(content) / (self.columns + 1))
self.footer_present = lines
self.lastpids = runningpids[:]
self.lastcount = self.helper.tasknumber_current
@@ -336,7 +338,10 @@ def main(server, eventHandler, params, tf = TerminalFilter):
errconsole = logging.StreamHandler(sys.stderr)
format_str = "%(levelname)s: %(message)s"
format = bb.msg.BBLogFormatter(format_str)
- bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut)
+ if params.options.quiet:
+ bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut, bb.msg.BBLogFormatter.WARNING)
+ else:
+ bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut)
bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
console.setFormatter(format)
errconsole.setFormatter(format)
@@ -399,7 +404,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
warnings = 0
taskfailures = []
- termfilter = tf(main, helper, console, errconsole, format)
+ termfilter = tf(main, helper, console, errconsole, format, params.options.quiet)
atexit.register(termfilter.finish)
while True:
@@ -498,8 +503,9 @@ def main(server, eventHandler, params, tf = TerminalFilter):
continue
parseprogress.finish()
- print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
- % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
+ if not params.options.quiet:
+ print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
+ % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
continue
if isinstance(event, bb.event.CacheLoadStarted):
@@ -510,7 +516,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
continue
if isinstance(event, bb.event.CacheLoadCompleted):
cacheprogress.finish()
- print("Loaded %d entries from dependency cache." % event.num_entries)
+ if not params.options.quiet:
+ print("Loaded %d entries from dependency cache." % event.num_entries)
continue
if isinstance(event, bb.command.CommandFailed):
@@ -670,7 +677,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if return_value and errors:
summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
"\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
- if summary:
+ if summary and not params.options.quiet:
print(summary)
if interrupted:
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 09/10] runqueue: add ability to enforce that tasks are setscened
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
` (7 preceding siblings ...)
2016-06-13 2:52 ` [PATCH 08/10] knotty: add quiet output mode Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-13 2:52 ` [PATCH 10/10] runqueue: report progress for "Preparing RunQueue" step Paul Eggleton
9 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
Add the ability to enter a mode where only a specified whitelist of
tasks can be executed outright; everything else must be successfully
provided in the form of a setscene task (or covered by a setscene task).
Any setscene failure outside of the whitelist will cause the build to
fail immediately instead of running the real task, and any real tasks
that would execute outside of the whitelist cause an immediate build
failure when it comes to executing the runqueue as well.
The mode is enabled by setting BB_SETSCENE_ENFORCE="1", and the
whitelist is specified through BB_SETSCENE_ENFORCE_WHITELIST, consisting
of pn:taskname pairs. A single % character can be substituted for the pn
value to match any target explicitly specified on the bitbake command
line. Wildcards * and ? can also be used as per standard unix file name
matching for both pn and taskname.
Part of the implementation of [YOCTO #9367].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/runqueue.py | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 65 insertions(+), 1 deletion(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 376c9f5..c68a3fe 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -224,6 +224,7 @@ class RunQueueData:
self.stampwhitelist = cfgData.getVar("BB_STAMP_WHITELIST", True) or ""
self.multi_provider_whitelist = (cfgData.getVar("MULTI_PROVIDER_WHITELIST", True) or "").split()
+ self.setscenewhitelist = get_setscene_enforce_whitelist(cfgData)
self.reset()
@@ -1629,8 +1630,35 @@ class RunQueueExecuteTasks(RunQueueExecute):
Run the tasks in a queue prepared by rqdata.prepare()
"""
+ if self.rqdata.setscenewhitelist:
+ # Check tasks that are going to run against the whitelist
+ unexpected = False
+ for taskid in range(len(self.rqdata.runq_fnid)):
+ fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[taskid]]
+ taskname = self.rqdata.runq_task[taskid]
+ # Ignore covered tasks
+ if taskid in self.rq.scenequeue_covered:
+ continue
+ # Ignore stamped tasks
+ if self.rq.check_stamp_task(taskid, taskname, cache=self.stampcache):
+ continue
+ # Ignore noexec tasks
+ taskdep = self.rqdata.dataCache.task_deps[fn]
+ if 'noexec' in taskdep and taskname in taskdep['noexec']:
+ continue
+
+ pn = self.rqdata.dataCache.pkg_fn[fn]
+ if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
+ unexpected = True
+ if taskid in self.rqdata.runq_setscene:
+ logger.error('Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname))
+ else:
+ logger.error('Task %s.%s attempted to execute unexpectedly' % (pn, taskname))
+ if unexpected:
+ self.rq.state = runQueueCleanUp
+ return True
+
self.rq.read_workers()
-
if self.stats.total == 0:
# nothing to do
@@ -1982,6 +2010,16 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.scenequeue_covered.add(task)
self.scenequeue_updatecounters(task)
+ def check_taskfail(self, task):
+ if self.rqdata.setscenewhitelist:
+ realtask = self.rqdata.runq_setscene[task]
+ fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[realtask]]
+ pn = self.rqdata.dataCache.pkg_fn[fn]
+ taskname = self.rqdata.runq_task[realtask]
+ if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
+ logger.error('Task %s.%s failed' % (pn, taskname + "_setscene"))
+ self.rq.state = runQueueCleanUp
+
def task_complete(self, task):
self.stats.taskCompleted()
bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
@@ -1992,6 +2030,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
bb.event.fire(sceneQueueTaskFailed(task, self.stats, result, self), self.cfgData)
self.scenequeue_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
+ self.check_taskfail(task)
def task_failoutright(self, task):
self.runq_running[task] = 1
@@ -2001,6 +2040,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
index = self.rqdata.runq_setscene[task]
self.scenequeue_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
+ self.check_taskfail(task)
def task_skip(self, task):
self.runq_running[task] = 1
@@ -2282,3 +2322,27 @@ class runQueuePipe():
if len(self.queue) > 0:
print("Warning, worker left partial message: %s" % self.queue)
self.input.close()
+
+def get_setscene_enforce_whitelist(d):
+ if d.getVar('BB_SETSCENE_ENFORCE', True) != '1':
+ return None
+ whitelist = (d.getVar("BB_SETSCENE_ENFORCE_WHITELIST", True) or "").split()
+ outlist = []
+ for item in whitelist[:]:
+ if item.startswith('%:'):
+ for target in sys.argv[1:]:
+ if not target.startswith('-'):
+ outlist.append(target.split(':')[0] + ':' + item.split(':')[1])
+ else:
+ outlist.append(item)
+ return outlist
+
+def check_setscene_enforce_whitelist(pn, taskname, whitelist):
+ import fnmatch
+ if whitelist:
+ item = '%s:%s' % (pn, taskname)
+ for whitelist_item in whitelist:
+ if fnmatch.fnmatch(item, whitelist_item):
+ return True
+ return False
+ return True
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 10/10] runqueue: report progress for "Preparing RunQueue" step
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
` (8 preceding siblings ...)
2016-06-13 2:52 ` [PATCH 09/10] runqueue: add ability to enforce that tasks are setscened Paul Eggleton
@ 2016-06-13 2:52 ` Paul Eggleton
2016-06-13 11:09 ` Richard Purdie
9 siblings, 1 reply; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 2:52 UTC (permalink / raw)
To: bitbake-devel
When "Preparing RunQueue" shows up you can expect to wait up to 30
seconds while it works - which is a bit long to leave the user waiting
without any kind of output. Since the work being carried out during this
time is divided into stages such that it's practical to determine
internally how it's progressing, replace the message with a progress
bar.
Actually what happens during this time is two major steps rather than
just one - the runqueue preparation itself, followed by the
initialisation prior to running setscene tasks. I elected to have the
progress bar cover both as one (there doesn't appear to be much point in
doing otherwise from a user perspective). I did however describe it as
"initialising tasks".
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/progress.py | 42 +++++++++++++++++++++++++++++++++++
lib/bb/runqueue.py | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 105 insertions(+), 2 deletions(-)
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
index 73e3f05..46574dd 100644
--- a/lib/bb/progress.py
+++ b/lib/bb/progress.py
@@ -222,3 +222,45 @@ class MultiStageProgressReporter(object):
else:
out.append('Up to finish: %d' % stage_weight)
bb.warn('Stage times:\n %s' % '\n '.join(out))
+
+class MultiStageProcessProgressReporter(MultiStageProgressReporter):
+ """
+ Version of MultiStageProgressReporter intended for use with
+ standalone processes (such as preparing the runqueue)
+ """
+ def __init__(self, d, processname, stage_weights, debug=False):
+ self._processname = processname
+ MultiStageProgressReporter.__init__(self, d, stage_weights, debug)
+
+ def start(self):
+ bb.event.fire(bb.event.ProcessStarted(self._processname, 100), self._data)
+
+ def _fire_progress(self, taskprogress):
+ bb.event.fire(bb.event.ProcessProgress(self._processname, taskprogress), self._data)
+
+ def finish(self):
+ MultiStageProgressReporter.finish(self)
+ bb.event.fire(bb.event.ProcessFinished(self._processname), self._data)
+
+class DummyMultiStageProcessProgressReporter(MultiStageProgressReporter):
+ """
+ MultiStageProcessProgressReporter that takes the calls and does nothing
+ with them (to avoid a bunch of "if progress_reporter:" checks)
+ """
+ def __init__(self):
+ MultiStageProcessProgressReporter.__init__(self, "", None, [])
+
+ def _fire_progress(self, taskprogress, rate=None):
+ pass
+
+ def start(self):
+ pass
+
+ def next_stage(self, stage_total=None):
+ pass
+
+ def update(self, stage_progress):
+ pass
+
+ def finish(self):
+ pass
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index c68a3fe..f83a651 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -225,6 +225,7 @@ class RunQueueData:
self.stampwhitelist = cfgData.getVar("BB_STAMP_WHITELIST", True) or ""
self.multi_provider_whitelist = (cfgData.getVar("MULTI_PROVIDER_WHITELIST", True) or "").split()
self.setscenewhitelist = get_setscene_enforce_whitelist(cfgData)
+ self.init_progress_reporter = bb.progress.DummyMultiStageProcessProgressReporter()
self.reset()
@@ -434,7 +435,8 @@ class RunQueueData:
# Nothing to do
return 0
- logger.info("Preparing RunQueue")
+ self.init_progress_reporter.start()
+ self.init_progress_reporter.next_stage()
# Step A - Work out a list of tasks to run
#
@@ -567,8 +569,9 @@ class RunQueueData:
# e.g. do_sometask[recrdeptask] = "do_someothertask"
# (makes sure sometask runs after someothertask of all DEPENDS, RDEPENDS and intertask dependencies, recursively)
# We need to do this separately since we need all of self.runq_depends to be complete before this is processed
+ self.init_progress_reporter.next_stage(len(recursivetasks))
extradeps = {}
- for task in recursivetasks:
+ for taskcounter, task in enumerate(recursivetasks):
extradeps[task] = set(self.runq_depends[task])
tasknames = recursivetasks[task]
seendeps = set()
@@ -589,6 +592,7 @@ class RunQueueData:
if task in recursiveitasks:
for dep in recursiveitasks[task]:
generate_recdeps(dep)
+ self.init_progress_reporter.update(taskcounter)
# Remove circular references so that do_a[recrdeptask] = "do_a do_b" can work
for task in recursivetasks:
@@ -603,6 +607,8 @@ class RunQueueData:
logger.debug(2, "Task %s (%s %s) contains self reference! %s", task, taskData.fn_index[taskData.tasks_fnid[task]], taskData.tasks_name[task], self.runq_depends[task])
self.runq_depends[task].remove(task)
+ self.init_progress_reporter.next_stage()
+
# Step B - Mark all active tasks
#
# Start with the tasks we were asked to run and mark all dependencies
@@ -666,6 +672,8 @@ class RunQueueData:
else:
mark_active(listid, 1)
+ self.init_progress_reporter.next_stage()
+
# Step C - Prune all inactive tasks
#
# Once all active tasks are marked, prune the ones we don't need.
@@ -685,6 +693,8 @@ class RunQueueData:
delcount = delcount + 1
maps.append(-1)
+ self.init_progress_reporter.next_stage()
+
#
# Step D - Sanity checks and computation
#
@@ -698,6 +708,8 @@ class RunQueueData:
logger.verbose("Pruned %s inactive tasks, %s left", delcount, len(self.runq_fnid))
+ self.init_progress_reporter.next_stage()
+
# Remap the dependencies to account for the deleted tasks
# Check we didn't delete a task we depend on
for listid in range(len(self.runq_fnid)):
@@ -711,11 +723,15 @@ class RunQueueData:
logger.verbose("Assign Weightings")
+ self.init_progress_reporter.next_stage()
+
# Generate a list of reverse dependencies to ease future calculations
for listid in range(len(self.runq_fnid)):
for dep in self.runq_depends[listid]:
self.runq_revdeps[dep].add(listid)
+ self.init_progress_reporter.next_stage()
+
# Identify tasks at the end of dependency chains
# Error on circular dependency loops (length two)
endpoints = []
@@ -730,10 +746,14 @@ class RunQueueData:
logger.verbose("Compute totals (have %s endpoint(s))", len(endpoints))
+ self.init_progress_reporter.next_stage()
+
# Calculate task weights
# Check of higher length circular dependencies
self.runq_weight = self.calculate_task_weights(endpoints)
+ self.init_progress_reporter.next_stage()
+
# Sanity Check - Check for multiple tasks building the same provider
prov_list = {}
seen_fn = []
@@ -825,6 +845,8 @@ class RunQueueData:
else:
logger.error(msg)
+ self.init_progress_reporter.next_stage()
+
# Create a whitelist usable by the stamp checks
stampfnwhitelist = []
for entry in self.stampwhitelist.split():
@@ -836,6 +858,8 @@ class RunQueueData:
stampfnwhitelist.append(fn)
self.stampfnwhitelist = stampfnwhitelist
+ self.init_progress_reporter.next_stage()
+
# Iterate over the task list looking for tasks with a 'setscene' function
self.runq_setscene = []
if not self.cooker.configuration.nosetscene:
@@ -859,6 +883,8 @@ class RunQueueData:
logger.verbose("Invalidate task %s, %s", taskname, fn)
bb.parse.siggen.invalidate_task(taskname, self.dataCache, fn)
+ self.init_progress_reporter.next_stage()
+
# Invalidate task if force mode active
if self.cooker.configuration.force:
for (fn, target) in self.target_pairs:
@@ -872,6 +898,8 @@ class RunQueueData:
st = "do_%s" % st
invalidate_task(fn, st, True)
+ self.init_progress_reporter.next_stage()
+
# Create and print to the logs a virtual/xxxx -> PN (fn) table
virtmap = taskData.get_providermap(prefix="virtual/")
virtpnmap = {}
@@ -881,6 +909,8 @@ class RunQueueData:
if hasattr(bb.parse.siggen, "tasks_resolved"):
bb.parse.siggen.tasks_resolved(virtmap, virtpnmap, self.dataCache)
+ self.init_progress_reporter.next_stage()
+
# Iterate over the task list and call into the siggen code
dealtwith = set()
todeal = set(range(len(self.runq_fnid)))
@@ -895,6 +925,7 @@ class RunQueueData:
self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
bb.parse.siggen.writeout_file_checksum_cache()
+
return len(self.runq_fnid)
def dump_data(self, taskQueue):
@@ -1119,14 +1150,19 @@ class RunQueue:
if self.state is runQueuePrepare:
self.rqexe = RunQueueExecuteDummy(self)
+ self.rqdata.init_progress_reporter = bb.progress.MultiStageProcessProgressReporter(self.cooker.data,
+ "Initialising tasks",
+ [27, 870, 3, 33, 1, 3, 3, 2, 3, 6, 1, 13, 1, 1, 132, 32, 1, 61, 1, 13, 2, 243, 151, 299, 81, 1, 15, 72])
if self.rqdata.prepare() == 0:
self.state = runQueueComplete
else:
self.state = runQueueSceneInit
+ self.rqdata.init_progress_reporter.next_stage()
# we are ready to run, emit dependency info to any UI or class which
# needs it
depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
+ self.rqdata.init_progress_reporter.next_stage()
bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
if self.state is runQueueSceneInit:
@@ -1139,7 +1175,9 @@ class RunQueue:
self.write_diffscenetasks(invalidtasks)
self.state = runQueueComplete
else:
+ self.rqdata.init_progress_reporter.next_stage()
self.start_worker()
+ self.rqdata.init_progress_reporter.next_stage()
self.rqexe = RunQueueExecuteScenequeue(self)
if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp]:
@@ -1152,6 +1190,8 @@ class RunQueue:
if self.cooker.configuration.setsceneonly:
self.state = runQueueComplete
else:
+ # Just in case we didn't setscene
+ self.rqdata.init_progress_reporter.finish()
logger.info("Executing RunQueue Tasks")
self.rqexe = RunQueueExecuteTasks(self)
self.state = runQueueRunning
@@ -1792,6 +1832,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.runq_complete.append(0)
self.runq_buildable.append(0)
+ self.rqdata.init_progress_reporter.next_stage()
+
# First process the chains up to the first setscene task.
endpoints = {}
for task in range(len(self.rqdata.runq_fnid)):
@@ -1800,6 +1842,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
if (len(self.rqdata.runq_revdeps[task]) == 0) and task not in self.rqdata.runq_setscene:
endpoints[task] = set()
+ self.rqdata.init_progress_reporter.next_stage()
+
# Secondly process the chains between setscene tasks.
for task in self.rqdata.runq_setscene:
for dep in self.rqdata.runq_depends[task]:
@@ -1807,6 +1851,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
endpoints[dep] = set()
endpoints[dep].add(task)
+ self.rqdata.init_progress_reporter.next_stage()
+
def process_endpoints(endpoints):
newendpoints = {}
for point, task in endpoints.items():
@@ -1831,6 +1877,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
process_endpoints(endpoints)
+ self.rqdata.init_progress_reporter.next_stage()
+
# Build a list of setscene tasks which are "unskippable"
# These are direct endpoints referenced by the build
endpoints2 = {}
@@ -1867,6 +1915,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
if sq_revdeps_new2[task]:
self.unskippable.append(self.rqdata.runq_setscene.index(task))
+ self.rqdata.init_progress_reporter.next_stage(len(self.rqdata.runq_fnid))
+
for task in range(len(self.rqdata.runq_fnid)):
if task in self.rqdata.runq_setscene:
deps = set()
@@ -1875,6 +1925,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
sq_revdeps_squash.append(deps)
elif len(sq_revdeps_new[task]) != 0:
bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
+ self.rqdata.init_progress_reporter.update(task)
+
+ self.rqdata.init_progress_reporter.next_stage()
# Resolve setscene inter-task dependencies
# e.g. do_sometask_setscene[depends] = "targetname:do_someothertask_setscene"
@@ -1902,10 +1955,14 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
# Have to zero this to avoid circular dependencies
sq_revdeps_squash[self.rqdata.runq_setscene.index(taskid)] = set()
+ self.rqdata.init_progress_reporter.next_stage()
+
for task in self.sq_harddeps:
for dep in self.sq_harddeps[task]:
sq_revdeps_squash[dep].add(task)
+ self.rqdata.init_progress_reporter.next_stage()
+
#for task in range(len(sq_revdeps_squash)):
# realtask = self.rqdata.runq_setscene[task]
# bb.warn("Task %s: %s_setscene is %s " % (task, self.rqdata.get_user_idstring(realtask) , sq_revdeps_squash[task]))
@@ -1920,6 +1977,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
for dep in self.sq_revdeps[task]:
self.sq_deps[dep].add(task)
+ self.rqdata.init_progress_reporter.next_stage()
+
for task in range(len(self.sq_revdeps)):
if len(self.sq_revdeps[task]) == 0:
self.runq_buildable[task] = 1
@@ -1977,6 +2036,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.rqdata.get_user_idstring(realtask))
self.outrightfail.append(task)
+ self.rqdata.init_progress_reporter.finish()
+
logger.info('Executing SetScene Tasks')
self.rq.state = runQueueSceneRun
--
2.5.5
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH 10/10] runqueue: report progress for "Preparing RunQueue" step
2016-06-13 2:52 ` [PATCH 10/10] runqueue: report progress for "Preparing RunQueue" step Paul Eggleton
@ 2016-06-13 11:09 ` Richard Purdie
2016-06-13 12:16 ` Paul Eggleton
0 siblings, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2016-06-13 11:09 UTC (permalink / raw)
To: Paul Eggleton, bitbake-devel
On Mon, 2016-06-13 at 14:52 +1200, Paul Eggleton wrote:
> @@ -1119,14 +1150,19 @@ class RunQueue:
>
> if self.state is runQueuePrepare:
> self.rqexe = RunQueueExecuteDummy(self)
> + self.rqdata.init_progress_reporter = bb.progress.MultiStageProcessProgressReporter(self.cooker.data,
> + "Initialising tasks",
> + [27, 870, 3, 33, 1, 3, 3, 2, 3, 6, 1, 13, 1, 1, 132, 32, 1, 61, 1, 13, 2, 243, 151, 299, 81, 1, 15, 72])
> if self.rqdata.prepare() == 0:
> self.state = runQueueComplete
> else:
> self.state = runQueueSceneInit
> + self.rqdata.init_progress_reporter.next_stage()
>
I appreciate the difficulty in figuring some of this out and the
changes look quite neat on the most part. I am however very worried by
the above from a code maintenance standpoint.
If for example I add or remove a step in runqueue, how do I know where
in this array to make a change? How do I calculate what the new
weighting should be?
I did wonder if next_stage() should pass the number instead, that way
at least the call site would be clearer. It doesn't solve the problem
of knowing how many steps there are, or what these weights mean or how
to recalculate them in the future.
Any thoughts on that?
Cheers,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 10/10] runqueue: report progress for "Preparing RunQueue" step
2016-06-13 11:09 ` Richard Purdie
@ 2016-06-13 12:16 ` Paul Eggleton
2016-06-15 21:46 ` Richard Purdie
0 siblings, 1 reply; 17+ messages in thread
From: Paul Eggleton @ 2016-06-13 12:16 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
On Mon, 13 Jun 2016 12:09:33 Richard Purdie wrote:
> On Mon, 2016-06-13 at 14:52 +1200, Paul Eggleton wrote:
> > @@ -1119,14 +1150,19 @@ class RunQueue:
> > if self.state is runQueuePrepare:
> > self.rqexe = RunQueueExecuteDummy(self)
> >
> > + self.rqdata.init_progress_reporter =
> > bb.progress.MultiStageProcessProgressReporter(self.cooker.data, +
> > "Initialising tasks",
> > + [27, 870, 3,
> > 33, 1, 3, 3, 2, 3, 6, 1, 13, 1, 1, 132, 32, 1, 61, 1, 13, 2, 243, 151,
> > 299, 81, 1, 15, 72])>
> > if self.rqdata.prepare() == 0:
> > self.state = runQueueComplete
> >
> > else:
> > self.state = runQueueSceneInit
> >
> > + self.rqdata.init_progress_reporter.next_stage()
>
> I appreciate the difficulty in figuring some of this out and the
> changes look quite neat on the most part. I am however very worried by
> the above from a code maintenance standpoint.
>
> If for example I add or remove a step in runqueue, how do I know where
> in this array to make a change? How do I calculate what the new
> weighting should be?
Actually with the current patchset it's pretty easy. Just temporarily specify
debug=True to the reporter when constructing and it will print out the list of
weightings (based on timing each stage) when finish() is called, along with a
map of the weightings to the lines where next_stage() was called. (Perhaps if
we'd want a comment to that effect for anyone editing this code.)
> I did wonder if next_stage() should pass the number instead, that way
> at least the call site would be clearer. It doesn't solve the problem
> of knowing how many steps there are, or what these weights mean or how
> to recalculate them in the future.
I guess you could do it that way; in the way I'd structured the code earlier
it wouldn't have been practical but now that next_stage() must be called
before every stage, it could be done. The trouble though is that then if you
want to set them all at once (as you often would, since they are relative) you
will have to go to each line and update it instead of just changing the list
passed to the constructor, so I'm not convinced that's an improvement.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 10/10] runqueue: report progress for "Preparing RunQueue" step
2016-06-13 12:16 ` Paul Eggleton
@ 2016-06-15 21:46 ` Richard Purdie
2016-06-15 21:49 ` Paul Eggleton
0 siblings, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2016-06-15 21:46 UTC (permalink / raw)
To: Paul Eggleton; +Cc: bitbake-devel
On Tue, 2016-06-14 at 00:16 +1200, Paul Eggleton wrote:
> On Mon, 13 Jun 2016 12:09:33 Richard Purdie wrote:
> > On Mon, 2016-06-13 at 14:52 +1200, Paul Eggleton wrote:
> > > @@ -1119,14 +1150,19 @@ class RunQueue:
> > > if self.state is runQueuePrepare:
> > > self.rqexe = RunQueueExecuteDummy(self)
> > >
> > > + self.rqdata.init_progress_reporter =
> > > bb.progress.MultiStageProcessProgressReporter(self.cooker.data, +
> > >
> > > "Initialising
> > > tasks",
> > > + [27,
> > > 870, 3,
> > > 33, 1, 3, 3, 2, 3, 6, 1, 13, 1, 1, 132, 32, 1, 61, 1, 13, 2, 243,
> > > 151,
> > > 299, 81, 1, 15, 72])>
> > > if self.rqdata.prepare() == 0:
> > > self.state = runQueueComplete
> > >
> > > else:
> > > self.state = runQueueSceneInit
> > >
> > > + self.rqdata.init_progress_reporter.next_stage()
> >
> > I appreciate the difficulty in figuring some of this out and the
> > changes look quite neat on the most part. I am however very worried
> > by
> > the above from a code maintenance standpoint.
> >
> > If for example I add or remove a step in runqueue, how do I know
> > where
> > in this array to make a change? How do I calculate what the new
> > weighting should be?
>
> Actually with the current patchset it's pretty easy. Just temporarily
> specify
> debug=True to the reporter when constructing and it will print out
> the list of
> weightings (based on timing each stage) when finish() is called,
> along with a
> map of the weightings to the lines where next_stage() was called.
> (Perhaps if
> we'd want a comment to that effect for anyone editing this code.)
I think a comment in there about this would be helpful as it isn't
clear how to handle this. What does it do in the case you have a next()
call without a weight? Will that give a warning?
Cheers,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 10/10] runqueue: report progress for "Preparing RunQueue" step
2016-06-15 21:46 ` Richard Purdie
@ 2016-06-15 21:49 ` Paul Eggleton
0 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-15 21:49 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
On Wed, 15 Jun 2016 22:46:41 Richard Purdie wrote:
> On Tue, 2016-06-14 at 00:16 +1200, Paul Eggleton wrote:
> > On Mon, 13 Jun 2016 12:09:33 Richard Purdie wrote:
> > > On Mon, 2016-06-13 at 14:52 +1200, Paul Eggleton wrote:
> > > > @@ -1119,14 +1150,19 @@ class RunQueue:
> > > > if self.state is runQueuePrepare:
> > > > self.rqexe = RunQueueExecuteDummy(self)
> > > >
> > > > + self.rqdata.init_progress_reporter =
> > > > bb.progress.MultiStageProcessProgressReporter(self.cooker.data, +
> > > >
> > > > "Initialising
> > > >
> > > > tasks",
> > > > + [27,
> > > > 870, 3,
> > > > 33, 1, 3, 3, 2, 3, 6, 1, 13, 1, 1, 132, 32, 1, 61, 1, 13, 2, 243,
> > > > 151,
> > > > 299, 81, 1, 15, 72])>
> > > >
> > > > if self.rqdata.prepare() == 0:
> > > > self.state = runQueueComplete
> > > >
> > > > else:
> > > > self.state = runQueueSceneInit
> > > >
> > > > + self.rqdata.init_progress_reporter.next_stage()
> > >
> > > I appreciate the difficulty in figuring some of this out and the
> > > changes look quite neat on the most part. I am however very worried
> > > by
> > > the above from a code maintenance standpoint.
> > >
> > > If for example I add or remove a step in runqueue, how do I know
> > > where
> > > in this array to make a change? How do I calculate what the new
> > > weighting should be?
> >
> > Actually with the current patchset it's pretty easy. Just temporarily
> > specify
> > debug=True to the reporter when constructing and it will print out
> > the list of
> > weightings (based on timing each stage) when finish() is called,
> > along with a
> > map of the weightings to the lines where next_stage() was called.
> > (Perhaps if
> > we'd want a comment to that effect for anyone editing this code.)
>
> I think a comment in there about this would be helpful as it isn't
> clear how to handle this.
OK, I'll add one - I need to rebase this patchset anyway.
> What does it do in the case you have a next()
> call without a weight? Will that give a warning?
It will yes.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 05/10] fetch2: implement progress support
2016-06-13 2:52 ` [PATCH 05/10] fetch2: implement progress support Paul Eggleton
@ 2016-06-28 10:23 ` Burton, Ross
2016-06-28 10:33 ` Paul Eggleton
0 siblings, 1 reply; 17+ messages in thread
From: Burton, Ross @ 2016-06-28 10:23 UTC (permalink / raw)
To: Paul Eggleton; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 319 bytes --]
On 13 June 2016 at 03:52, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:
> + runfetchcmd(command + ' --progress=dot --show-progress', d,
> quiet, log=progresshandler)
>
wget: unrecognized option '--show-progress'
Turns out wget on Ubuntu 14.04 (wget 1.15) doesn't support this option.
Ross
[-- Attachment #2: Type: text/html, Size: 851 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 05/10] fetch2: implement progress support
2016-06-28 10:23 ` Burton, Ross
@ 2016-06-28 10:33 ` Paul Eggleton
0 siblings, 0 replies; 17+ messages in thread
From: Paul Eggleton @ 2016-06-28 10:33 UTC (permalink / raw)
To: Burton, Ross; +Cc: bitbake-devel
On Tue, 28 Jun 2016 11:23:32 Burton, Ross wrote:
> On 13 June 2016 at 03:52, Paul Eggleton <paul.eggleton@linux.intel.com>
>
> wrote:
> > + runfetchcmd(command + ' --progress=dot --show-progress', d,
> > quiet, log=progresshandler)
>
> wget: unrecognized option '--show-progress'
>
> Turns out wget on Ubuntu 14.04 (wget 1.15) doesn't support this option.
Turns out it was introduced in 1.16. Sigh...
I will try to figure out what we can do with this. Worst case I suppose we
could run wget once to determine the version and just disable progress
reporting if it's too old.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2016-06-28 10:33 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-13 2:52 [PATCH 00/10] Support progress reporting Paul Eggleton
2016-06-13 2:52 ` [PATCH 01/10] knotty: provide a symlink to the latest console log Paul Eggleton
2016-06-13 2:52 ` [PATCH 02/10] knotty: import latest python-progressbar Paul Eggleton
2016-06-13 2:52 ` [PATCH 03/10] lib: implement basic task progress support Paul Eggleton
2016-06-13 2:52 ` [PATCH 04/10] lib/bb/progress: add MultiStageProgressReporter Paul Eggleton
2016-06-13 2:52 ` [PATCH 05/10] fetch2: implement progress support Paul Eggleton
2016-06-28 10:23 ` Burton, Ross
2016-06-28 10:33 ` Paul Eggleton
2016-06-13 2:52 ` [PATCH 06/10] knotty: add code to support showing progress for sstate object querying Paul Eggleton
2016-06-13 2:52 ` [PATCH 07/10] knotty: show task progress bar Paul Eggleton
2016-06-13 2:52 ` [PATCH 08/10] knotty: add quiet output mode Paul Eggleton
2016-06-13 2:52 ` [PATCH 09/10] runqueue: add ability to enforce that tasks are setscened Paul Eggleton
2016-06-13 2:52 ` [PATCH 10/10] runqueue: report progress for "Preparing RunQueue" step Paul Eggleton
2016-06-13 11:09 ` Richard Purdie
2016-06-13 12:16 ` Paul Eggleton
2016-06-15 21:46 ` Richard Purdie
2016-06-15 21:49 ` Paul Eggleton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.