* [PATCH v2 00/10] Support progress reporting
@ 2016-06-23 10:59 Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 01/10] knotty: provide a symlink to the latest console log Paul Eggleton
` (9 more replies)
0 siblings, 10 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
Add code to support progress reporting within OpenEmbedded, as well as a
couple of related changes to allow reworking the extensible SDK
installation process. (See the corresponding OE-Core series for further
details).
Changes since v1:
* Rebase on top of recent runqueue changes
* Fix debug=True in MultiStageProgressReporter printing weightings if
finish() was called a second time (as it is in the runqueue).
* Fix a couple of bugs with the BB_SETSCENE_ENFORCE code
* Add comments in the runqueue code explaining how to update the weightings
The following changes since commit 31977e7bb98f676197c6cee66f6ab4c12d4dcbde:
cooker: clean up EvertWriter (2016-06-20 17:23:53 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib paule/startup-bb
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/startup-bb
Paul Eggleton (10):
knotty: provide a symlink to the latest console log
knotty: import latest python-progressbar
lib: implement basic task progress support
lib/bb/progress: add MultiStageProgressReporter
fetch2: implement progress support
knotty: add code to support showing progress for sstate object
querying
knotty: show task progress bar
knotty: add quiet output mode
runqueue: add ability to enforce that tasks are setscened
runqueue: report progress for "Preparing RunQueue" step
lib/bb/build.py | 34 ++++
lib/bb/event.py | 27 +++
lib/bb/fetch2/__init__.py | 4 +-
lib/bb/fetch2/git.py | 52 +++++-
lib/bb/fetch2/wget.py | 26 ++-
lib/bb/main.py | 9 +
lib/bb/msg.py | 5 +-
lib/bb/progress.py | 270 ++++++++++++++++++++++++++++
lib/bb/runqueue.py | 151 +++++++++++++++-
lib/bb/ui/knotty.py | 143 ++++++++++++---
lib/bb/ui/uihelper.py | 7 +-
lib/progressbar.py | 384 ----------------------------------------
lib/progressbar/LICENSE.txt | 52 ++++++
lib/progressbar/__init__.py | 49 ++++++
lib/progressbar/compat.py | 44 +++++
lib/progressbar/progressbar.py | 315 +++++++++++++++++++++++++++++++++
lib/progressbar/widgets.py | 391 +++++++++++++++++++++++++++++++++++++++++
17 files changed, 1542 insertions(+), 421 deletions(-)
create mode 100644 lib/bb/progress.py
delete mode 100644 lib/progressbar.py
create mode 100644 lib/progressbar/LICENSE.txt
create mode 100644 lib/progressbar/__init__.py
create mode 100644 lib/progressbar/compat.py
create mode 100644 lib/progressbar/progressbar.py
create mode 100644 lib/progressbar/widgets.py
--
2.5.5
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v2 01/10] knotty: provide a symlink to the latest console log
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 02/10] knotty: import latest python-progressbar Paul Eggleton
` (8 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
If you're looking to find the latest console log repeatedly it can be a bit
tedious - let's just create a symlink just as we do with other logs to
make it easy to find.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/ui/knotty.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 9605c8e..ddd36d5 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -287,6 +287,12 @@ def main(server, eventHandler, params, tf = TerminalFilter):
bb.msg.addDefaultlogFilter(consolelog)
consolelog.setFormatter(conlogformat)
logger.addHandler(consolelog)
+ loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log')
+ bb.utils.remove(loglink)
+ try:
+ os.symlink(os.path.basename(consolelogfile), loglink)
+ except OSError:
+ pass
llevel, debug_domains = bb.msg.constructLogOptions()
server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list])
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 02/10] knotty: import latest python-progressbar
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 01/10] knotty: provide a symlink to the latest console log Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 03/10] lib: implement basic task progress support Paul Eggleton
` (7 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
Since we're going to make some minor extensions to it, it makes sense to
bring in the latest version of python-progressbar. Its structure has
changed a little but the API hasn't; however we do need to ensure our
overridden _needs_update() function's signature in BBProgress() matches
properly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/ui/knotty.py | 2 +-
lib/progressbar.py | 384 -----------------------------------------
lib/progressbar/LICENSE.txt | 52 ++++++
lib/progressbar/__init__.py | 49 ++++++
lib/progressbar/compat.py | 44 +++++
lib/progressbar/progressbar.py | 307 ++++++++++++++++++++++++++++++++
lib/progressbar/widgets.py | 355 +++++++++++++++++++++++++++++++++++++
7 files changed, 808 insertions(+), 385 deletions(-)
delete mode 100644 lib/progressbar.py
create mode 100644 lib/progressbar/LICENSE.txt
create mode 100644 lib/progressbar/__init__.py
create mode 100644 lib/progressbar/compat.py
create mode 100644 lib/progressbar/progressbar.py
create mode 100644 lib/progressbar/widgets.py
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index ddd36d5..6a6f688 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -51,7 +51,7 @@ class BBProgress(progressbar.ProgressBar):
self._resize_default = None
progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets, fd=sys.stdout)
- def _handle_resize(self, signum, frame):
+ def _handle_resize(self, signum=None, frame=None):
progressbar.ProgressBar._handle_resize(self, signum, frame)
if self._resize_default:
self._resize_default(signum, frame)
diff --git a/lib/progressbar.py b/lib/progressbar.py
deleted file mode 100644
index 114cdc1..0000000
--- a/lib/progressbar.py
+++ /dev/null
@@ -1,384 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: iso-8859-1 -*-
-#
-# progressbar - Text progressbar library for python.
-# Copyright (c) 2005 Nilton Volpato
-#
-# This library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-#
-# This library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with this library; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
-
-
-"""Text progressbar library for python.
-
-This library provides a text mode progressbar. This is typically used
-to display the progress of a long running operation, providing a
-visual clue that processing is underway.
-
-The ProgressBar class manages the progress, and the format of the line
-is given by a number of widgets. A widget is an object that may
-display diferently depending on the state of the progress. There are
-three types of widget:
-- a string, which always shows itself;
-- a ProgressBarWidget, which may return a diferent value every time
-it's update method is called; and
-- a ProgressBarWidgetHFill, which is like ProgressBarWidget, except it
-expands to fill the remaining width of the line.
-
-The progressbar module is very easy to use, yet very powerful. And
-automatically supports features like auto-resizing when available.
-"""
-
-from __future__ import division
-
-__author__ = "Nilton Volpato"
-__author_email__ = "first-name dot last-name @ gmail.com"
-__date__ = "2006-05-07"
-__version__ = "2.3-dev"
-
-import sys, time, os
-from array import array
-try:
- from fcntl import ioctl
- import termios
-except ImportError:
- pass
-import signal
-try:
- basestring
-except NameError:
- basestring = (str,)
-
-class ProgressBarWidget(object):
- """This is an element of ProgressBar formatting.
-
- The ProgressBar object will call it's update value when an update
- is needed. It's size may change between call, but the results will
- not be good if the size changes drastically and repeatedly.
- """
- def update(self, pbar):
- """Returns the string representing the widget.
-
- The parameter pbar is a reference to the calling ProgressBar,
- where one can access attributes of the class for knowing how
- the update must be made.
-
- At least this function must be overriden."""
- pass
-
-class ProgressBarWidgetHFill(object):
- """This is a variable width element of ProgressBar formatting.
-
- The ProgressBar object will call it's update value, informing the
- width this object must the made. This is like TeX \\hfill, it will
- expand to fill the line. You can use more than one in the same
- line, and they will all have the same width, and together will
- fill the line.
- """
- def update(self, pbar, width):
- """Returns the string representing the widget.
-
- The parameter pbar is a reference to the calling ProgressBar,
- where one can access attributes of the class for knowing how
- the update must be made. The parameter width is the total
- horizontal width the widget must have.
-
- At least this function must be overriden."""
- pass
-
-
-class ETA(ProgressBarWidget):
- "Widget for the Estimated Time of Arrival"
- def format_time(self, seconds):
- return time.strftime('%H:%M:%S', time.gmtime(seconds))
- def update(self, pbar):
- if pbar.currval == 0:
- return 'ETA: --:--:--'
- elif pbar.finished:
- return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
- else:
- elapsed = pbar.seconds_elapsed
- eta = elapsed * pbar.maxval / pbar.currval - elapsed
- return 'ETA: %s' % self.format_time(eta)
-
-class FileTransferSpeed(ProgressBarWidget):
- "Widget for showing the transfer speed (useful for file transfers)."
- def __init__(self, unit='B'):
- self.unit = unit
- self.fmt = '%6.2f %s'
- self.prefixes = ['', 'K', 'M', 'G', 'T', 'P']
- def update(self, pbar):
- if pbar.seconds_elapsed < 2e-6:#== 0:
- bps = 0.0
- else:
- bps = pbar.currval / pbar.seconds_elapsed
- spd = bps
- for u in self.prefixes:
- if spd < 1000:
- break
- spd /= 1000
- return self.fmt % (spd, u + self.unit + '/s')
-
-class RotatingMarker(ProgressBarWidget):
- "A rotating marker for filling the bar of progress."
- def __init__(self, markers='|/-\\'):
- self.markers = markers
- self.curmark = -1
- def update(self, pbar):
- if pbar.finished:
- return self.markers[0]
- self.curmark = (self.curmark + 1) % len(self.markers)
- return self.markers[self.curmark]
-
-class Percentage(ProgressBarWidget):
- "Just the percentage done."
- def update(self, pbar):
- return '%3d%%' % pbar.percentage()
-
-class SimpleProgress(ProgressBarWidget):
- "Returns what is already done and the total, e.g.: '5 of 47'"
- def __init__(self, sep=' of '):
- self.sep = sep
- def update(self, pbar):
- return '%d%s%d' % (pbar.currval, self.sep, pbar.maxval)
-
-class Bar(ProgressBarWidgetHFill):
- "The bar of progress. It will stretch to fill the line."
- def __init__(self, marker='#', left='|', right='|'):
- self.marker = marker
- self.left = left
- self.right = right
- def _format_marker(self, pbar):
- if isinstance(self.marker, basestring):
- return self.marker
- else:
- return self.marker.update(pbar)
- def update(self, pbar, width):
- percent = pbar.percentage()
- cwidth = width - len(self.left) - len(self.right)
- marked_width = int(percent * cwidth // 100)
- m = self._format_marker(pbar)
- bar = (self.left + (m * marked_width).ljust(cwidth) + self.right)
- return bar
-
-class ReverseBar(Bar):
- "The reverse bar of progress, or bar of regress. :)"
- def update(self, pbar, width):
- percent = pbar.percentage()
- cwidth = width - len(self.left) - len(self.right)
- marked_width = int(percent * cwidth // 100)
- m = self._format_marker(pbar)
- bar = (self.left + (m*marked_width).rjust(cwidth) + self.right)
- return bar
-
-default_widgets = [Percentage(), ' ', Bar()]
-class ProgressBar(object):
- """This is the ProgressBar class, it updates and prints the bar.
-
- A common way of using it is like:
- >>> pbar = ProgressBar().start()
- >>> for i in xrange(100):
- ... # do something
- ... pbar.update(i+1)
- ...
- >>> pbar.finish()
-
- You can also use a progressbar as an iterator:
- >>> progress = ProgressBar()
- >>> for i in progress(some_iterable):
- ... # do something
- ...
-
- But anything you want to do is possible (well, almost anything).
- You can supply different widgets of any type in any order. And you
- can even write your own widgets! There are many widgets already
- shipped and you should experiment with them.
-
- The term_width parameter must be an integer or None. In the latter case
- it will try to guess it, if it fails it will default to 80 columns.
-
- When implementing a widget update method you may access any
- attribute or function of the ProgressBar object calling the
- widget's update method. The most important attributes you would
- like to access are:
- - currval: current value of the progress, 0 <= currval <= maxval
- - maxval: maximum (and final) value of the progress
- - finished: True if the bar has finished (reached 100%), False o/w
- - start_time: the time when start() method of ProgressBar was called
- - seconds_elapsed: seconds elapsed since start_time
- - percentage(): percentage of the progress [0..100]. This is a method.
-
- The attributes above are unlikely to change between different versions,
- the other ones may change or cease to exist without notice, so try to rely
- only on the ones documented above if you are extending the progress bar.
- """
-
- __slots__ = ('currval', 'fd', 'finished', 'last_update_time', 'maxval',
- 'next_update', 'num_intervals', 'seconds_elapsed',
- 'signal_set', 'start_time', 'term_width', 'update_interval',
- 'widgets', '_iterable')
-
- _DEFAULT_MAXVAL = 100
-
- def __init__(self, maxval=None, widgets=default_widgets, term_width=None,
- fd=sys.stderr):
- self.maxval = maxval
- self.widgets = widgets
- self.fd = fd
- self.signal_set = False
- if term_width is not None:
- self.term_width = term_width
- else:
- try:
- self._handle_resize(None, None)
- signal.signal(signal.SIGWINCH, self._handle_resize)
- self.signal_set = True
- except (SystemExit, KeyboardInterrupt):
- raise
- except:
- self.term_width = int(os.environ.get('COLUMNS', 80)) - 1
-
- self.currval = 0
- self.finished = False
- self.start_time = None
- self.last_update_time = None
- self.seconds_elapsed = 0
- self._iterable = None
-
- def __call__(self, iterable):
- try:
- self.maxval = len(iterable)
- except TypeError:
- # If the iterable has no length, then rely on the value provided
- # by the user, otherwise fail.
- if not (isinstance(self.maxval, (int, long)) and self.maxval > 0):
- raise RuntimeError('Could not determine maxval from iterable. '
- 'You must explicitly provide a maxval.')
- self._iterable = iter(iterable)
- self.start()
- return self
-
- def __iter__(self):
- return self
-
- def next(self):
- try:
- next = self._iterable.next()
- self.update(self.currval + 1)
- return next
- except StopIteration:
- self.finish()
- raise
-
- def _handle_resize(self, signum, frame):
- h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2]
- self.term_width = w
-
- def percentage(self):
- "Returns the percentage of the progress."
- return self.currval * 100.0 / self.maxval
-
- def _format_widgets(self):
- r = []
- hfill_inds = []
- num_hfill = 0
- currwidth = 0
- for i, w in enumerate(self.widgets):
- if isinstance(w, ProgressBarWidgetHFill):
- r.append(w)
- hfill_inds.append(i)
- num_hfill += 1
- elif isinstance(w, basestring):
- r.append(w)
- currwidth += len(w)
- else:
- weval = w.update(self)
- currwidth += len(weval)
- r.append(weval)
- for iw in hfill_inds:
- widget_width = int((self.term_width - currwidth) // num_hfill)
- r[iw] = r[iw].update(self, widget_width)
- return r
-
- def _format_line(self):
- return ''.join(self._format_widgets()).ljust(self.term_width)
-
- def _next_update(self):
- return int((int(self.num_intervals *
- (self.currval / self.maxval)) + 1) *
- self.update_interval)
-
- def _need_update(self):
- """Returns true when the progressbar should print an updated line.
-
- You can override this method if you want finer grained control over
- updates.
-
- The current implementation is optimized to be as fast as possible and
- as economical as possible in the number of updates. However, depending
- on your usage you may want to do more updates. For instance, if your
- progressbar stays in the same percentage for a long time, and you want
- to update other widgets, like ETA, then you could return True after
- some time has passed with no updates.
-
- Ideally you could call self._format_line() and see if it's different
- from the previous _format_line() call, but calling _format_line() takes
- around 20 times more time than calling this implementation of
- _need_update().
- """
- return self.currval >= self.next_update
-
- def update(self, value):
- "Updates the progress bar to a new value."
- assert 0 <= value <= self.maxval, '0 <= %d <= %d' % (value, self.maxval)
- self.currval = value
- if not self._need_update():
- return
- if self.start_time is None:
- raise RuntimeError('You must call start() before calling update()')
- now = time.time()
- self.seconds_elapsed = now - self.start_time
- self.next_update = self._next_update()
- self.fd.write(self._format_line() + '\r')
- self.last_update_time = now
-
- def start(self):
- """Starts measuring time, and prints the bar at 0%.
-
- It returns self so you can use it like this:
- >>> pbar = ProgressBar().start()
- >>> for i in xrange(100):
- ... # do something
- ... pbar.update(i+1)
- ...
- >>> pbar.finish()
- """
- if self.maxval is None:
- self.maxval = self._DEFAULT_MAXVAL
- assert self.maxval > 0
-
- self.num_intervals = max(100, self.term_width)
- self.update_interval = self.maxval / self.num_intervals
- self.next_update = 0
-
- self.start_time = self.last_update_time = time.time()
- self.update(0)
- return self
-
- def finish(self):
- """Used to tell the progress is finished."""
- self.finished = True
- self.update(self.maxval)
- self.fd.write('\n')
- if self.signal_set:
- signal.signal(signal.SIGWINCH, signal.SIG_DFL)
diff --git a/lib/progressbar/LICENSE.txt b/lib/progressbar/LICENSE.txt
new file mode 100644
index 0000000..fc8ccdc
--- /dev/null
+++ b/lib/progressbar/LICENSE.txt
@@ -0,0 +1,52 @@
+You can redistribute and/or modify this library under the terms of the
+GNU LGPL license or BSD license (or both).
+
+---
+
+progressbar - Text progress bar library for python.
+Copyright (C) 2005 Nilton Volpato
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Lesser General Public
+License as published by the Free Software Foundation; either
+version 2.1 of the License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+Lesser General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public
+License along with this library; if not, write to the Free Software
+Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+---
+
+progressbar - Text progress bar library for python
+Copyright (c) 2008 Nilton Volpato
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+ a. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ b. Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in the
+ documentation and/or other materials provided with the distribution.
+ c. Neither the name of the author nor the names of its contributors
+ may be used to endorse or promote products derived from this software
+ without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+DAMAGE.
diff --git a/lib/progressbar/__init__.py b/lib/progressbar/__init__.py
new file mode 100644
index 0000000..fbab744
--- /dev/null
+++ b/lib/progressbar/__init__.py
@@ -0,0 +1,49 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Text progress bar library for Python.
+
+A text progress bar is typically used to display the progress of a long
+running operation, providing a visual cue that processing is underway.
+
+The ProgressBar class manages the current progress, and the format of the line
+is given by a number of widgets. A widget is an object that may display
+differently depending on the state of the progress bar. There are three types
+of widgets:
+ - a string, which always shows itself
+
+ - a ProgressBarWidget, which may return a different value every time its
+ update method is called
+
+ - a ProgressBarWidgetHFill, which is like ProgressBarWidget, except it
+ expands to fill the remaining width of the line.
+
+The progressbar module is very easy to use, yet very powerful. It will also
+automatically enable features like auto-resizing when the system supports it.
+"""
+
+__author__ = 'Nilton Volpato'
+__author_email__ = 'first-name dot last-name @ gmail.com'
+__date__ = '2011-05-14'
+__version__ = '2.3'
+
+from .compat import *
+from .widgets import *
+from .progressbar import *
diff --git a/lib/progressbar/compat.py b/lib/progressbar/compat.py
new file mode 100644
index 0000000..a39f4a1
--- /dev/null
+++ b/lib/progressbar/compat.py
@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Compatibility methods and classes for the progressbar module."""
+
+
+# Python 3.x (and backports) use a modified iterator syntax
+# This will allow 2.x to behave with 3.x iterators
+try:
+ next
+except NameError:
+ def next(iter):
+ try:
+ # Try new style iterators
+ return iter.__next__()
+ except AttributeError:
+ # Fallback in case of a "native" iterator
+ return iter.next()
+
+
+# Python < 2.5 does not have "any"
+try:
+ any
+except NameError:
+ def any(iterator):
+ for item in iterator:
+ if item: return True
+ return False
diff --git a/lib/progressbar/progressbar.py b/lib/progressbar/progressbar.py
new file mode 100644
index 0000000..0b9dcf7
--- /dev/null
+++ b/lib/progressbar/progressbar.py
@@ -0,0 +1,307 @@
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Main ProgressBar class."""
+
+from __future__ import division
+
+import math
+import os
+import signal
+import sys
+import time
+
+try:
+ from fcntl import ioctl
+ from array import array
+ import termios
+except ImportError:
+ pass
+
+from .compat import * # for: any, next
+from . import widgets
+
+
+class UnknownLength: pass
+
+
+class ProgressBar(object):
+ """The ProgressBar class which updates and prints the bar.
+
+ A common way of using it is like:
+ >>> pbar = ProgressBar().start()
+ >>> for i in range(100):
+ ... # do something
+ ... pbar.update(i+1)
+ ...
+ >>> pbar.finish()
+
+ You can also use a ProgressBar as an iterator:
+ >>> progress = ProgressBar()
+ >>> for i in progress(some_iterable):
+ ... # do something
+ ...
+
+ Since the progress bar is incredibly customizable you can specify
+ different widgets of any type in any order. You can even write your own
+ widgets! However, since there are already a good number of widgets you
+ should probably play around with them before moving on to create your own
+ widgets.
+
+ The term_width parameter represents the current terminal width. If the
+ parameter is set to an integer then the progress bar will use that,
+ otherwise it will attempt to determine the terminal width falling back to
+ 80 columns if the width cannot be determined.
+
+ When implementing a widget's update method you are passed a reference to
+ the current progress bar. As a result, you have access to the
+ ProgressBar's methods and attributes. Although there is nothing preventing
+ you from changing the ProgressBar you should treat it as read only.
+
+ Useful methods and attributes include (Public API):
+ - currval: current progress (0 <= currval <= maxval)
+ - maxval: maximum (and final) value
+ - finished: True if the bar has finished (reached 100%)
+ - start_time: the time when start() method of ProgressBar was called
+ - seconds_elapsed: seconds elapsed since start_time and last call to
+ update
+ - percentage(): progress in percent [0..100]
+ """
+
+ __slots__ = ('currval', 'fd', 'finished', 'last_update_time',
+ 'left_justify', 'maxval', 'next_update', 'num_intervals',
+ 'poll', 'seconds_elapsed', 'signal_set', 'start_time',
+ 'term_width', 'update_interval', 'widgets', '_time_sensitive',
+ '__iterable')
+
+ _DEFAULT_MAXVAL = 100
+ _DEFAULT_TERMSIZE = 80
+ _DEFAULT_WIDGETS = [widgets.Percentage(), ' ', widgets.Bar()]
+
+ def __init__(self, maxval=None, widgets=None, term_width=None, poll=1,
+ left_justify=True, fd=sys.stderr):
+ """Initializes a progress bar with sane defaults."""
+
+ # Don't share a reference with any other progress bars
+ if widgets is None:
+ widgets = list(self._DEFAULT_WIDGETS)
+
+ self.maxval = maxval
+ self.widgets = widgets
+ self.fd = fd
+ self.left_justify = left_justify
+
+ self.signal_set = False
+ if term_width is not None:
+ self.term_width = term_width
+ else:
+ try:
+ self._handle_resize(None, None)
+ signal.signal(signal.SIGWINCH, self._handle_resize)
+ self.signal_set = True
+ except (SystemExit, KeyboardInterrupt): raise
+ except Exception as e:
+ print("DEBUG 5 %s" % e)
+ self.term_width = self._env_size()
+
+ self.__iterable = None
+ self._update_widgets()
+ self.currval = 0
+ self.finished = False
+ self.last_update_time = None
+ self.poll = poll
+ self.seconds_elapsed = 0
+ self.start_time = None
+ self.update_interval = 1
+ self.next_update = 0
+
+
+ def __call__(self, iterable):
+ """Use a ProgressBar to iterate through an iterable."""
+
+ try:
+ self.maxval = len(iterable)
+ except:
+ if self.maxval is None:
+ self.maxval = UnknownLength
+
+ self.__iterable = iter(iterable)
+ return self
+
+
+ def __iter__(self):
+ return self
+
+
+ def __next__(self):
+ try:
+ value = next(self.__iterable)
+ if self.start_time is None:
+ self.start()
+ else:
+ self.update(self.currval + 1)
+ return value
+ except StopIteration:
+ if self.start_time is None:
+ self.start()
+ self.finish()
+ raise
+
+
+ # Create an alias so that Python 2.x won't complain about not being
+ # an iterator.
+ next = __next__
+
+
+ def _env_size(self):
+ """Tries to find the term_width from the environment."""
+
+ return int(os.environ.get('COLUMNS', self._DEFAULT_TERMSIZE)) - 1
+
+
+ def _handle_resize(self, signum=None, frame=None):
+ """Tries to catch resize signals sent from the terminal."""
+
+ h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2]
+ self.term_width = w
+
+
+ def percentage(self):
+ """Returns the progress as a percentage."""
+ if self.currval >= self.maxval:
+ return 100.0
+ return (self.currval * 100.0 / self.maxval) if self.maxval else 100.00
+
+ percent = property(percentage)
+
+
+ def _format_widgets(self):
+ result = []
+ expanding = []
+ width = self.term_width
+
+ for index, widget in enumerate(self.widgets):
+ if isinstance(widget, widgets.WidgetHFill):
+ result.append(widget)
+ expanding.insert(0, index)
+ else:
+ widget = widgets.format_updatable(widget, self)
+ result.append(widget)
+ width -= len(widget)
+
+ count = len(expanding)
+ while count:
+ portion = max(int(math.ceil(width * 1. / count)), 0)
+ index = expanding.pop()
+ count -= 1
+
+ widget = result[index].update(self, portion)
+ width -= len(widget)
+ result[index] = widget
+
+ return result
+
+
+ def _format_line(self):
+ """Joins the widgets and justifies the line."""
+
+ widgets = ''.join(self._format_widgets())
+
+ if self.left_justify: return widgets.ljust(self.term_width)
+ else: return widgets.rjust(self.term_width)
+
+
+ def _need_update(self):
+ """Returns whether the ProgressBar should redraw the line."""
+ if self.currval >= self.next_update or self.finished: return True
+
+ delta = time.time() - self.last_update_time
+ return self._time_sensitive and delta > self.poll
+
+
+ def _update_widgets(self):
+ """Checks all widgets for the time sensitive bit."""
+
+ self._time_sensitive = any(getattr(w, 'TIME_SENSITIVE', False)
+ for w in self.widgets)
+
+
+ def update(self, value=None):
+ """Updates the ProgressBar to a new value."""
+
+ if value is not None and value is not UnknownLength:
+ if (self.maxval is not UnknownLength
+ and not 0 <= value <= self.maxval):
+
+ raise ValueError('Value out of range')
+
+ self.currval = value
+
+
+ if not self._need_update(): return
+ if self.start_time is None:
+ raise RuntimeError('You must call "start" before calling "update"')
+
+ now = time.time()
+ self.seconds_elapsed = now - self.start_time
+ self.next_update = self.currval + self.update_interval
+ self.fd.write(self._format_line() + '\r')
+ self.fd.flush()
+ self.last_update_time = now
+
+
+ def start(self):
+ """Starts measuring time, and prints the bar at 0%.
+
+ It returns self so you can use it like this:
+ >>> pbar = ProgressBar().start()
+ >>> for i in range(100):
+ ... # do something
+ ... pbar.update(i+1)
+ ...
+ >>> pbar.finish()
+ """
+
+ if self.maxval is None:
+ self.maxval = self._DEFAULT_MAXVAL
+
+ self.num_intervals = max(100, self.term_width)
+ self.next_update = 0
+
+ if self.maxval is not UnknownLength:
+ if self.maxval < 0: raise ValueError('Value out of range')
+ self.update_interval = self.maxval / self.num_intervals
+
+
+ self.start_time = self.last_update_time = time.time()
+ self.update(0)
+
+ return self
+
+
+ def finish(self):
+ """Puts the ProgressBar bar in the finished state."""
+
+ if self.finished:
+ return
+ self.finished = True
+ self.update(self.maxval)
+ self.fd.write('\n')
+ if self.signal_set:
+ signal.signal(signal.SIGWINCH, signal.SIG_DFL)
diff --git a/lib/progressbar/widgets.py b/lib/progressbar/widgets.py
new file mode 100644
index 0000000..6434ad5
--- /dev/null
+++ b/lib/progressbar/widgets.py
@@ -0,0 +1,355 @@
+# -*- coding: utf-8 -*-
+#
+# progressbar - Text progress bar library for Python.
+# Copyright (c) 2005 Nilton Volpato
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Default ProgressBar widgets."""
+
+from __future__ import division
+
+import datetime
+import math
+
+try:
+ from abc import ABCMeta, abstractmethod
+except ImportError:
+ AbstractWidget = object
+ abstractmethod = lambda fn: fn
+else:
+ AbstractWidget = ABCMeta('AbstractWidget', (object,), {})
+
+
+def format_updatable(updatable, pbar):
+ if hasattr(updatable, 'update'): return updatable.update(pbar)
+ else: return updatable
+
+
+class Widget(AbstractWidget):
+ """The base class for all widgets.
+
+ The ProgressBar will call the widget's update value when the widget should
+ be updated. The widget's size may change between calls, but the widget may
+ display incorrectly if the size changes drastically and repeatedly.
+
+ The boolean TIME_SENSITIVE informs the ProgressBar that it should be
+ updated more often because it is time sensitive.
+ """
+
+ TIME_SENSITIVE = False
+ __slots__ = ()
+
+ @abstractmethod
+ def update(self, pbar):
+ """Updates the widget.
+
+ pbar - a reference to the calling ProgressBar
+ """
+
+
+class WidgetHFill(Widget):
+ """The base class for all variable width widgets.
+
+ This widget is much like the \\hfill command in TeX, it will expand to
+ fill the line. You can use more than one in the same line, and they will
+ all have the same width, and together will fill the line.
+ """
+
+ @abstractmethod
+ def update(self, pbar, width):
+ """Updates the widget providing the total width the widget must fill.
+
+ pbar - a reference to the calling ProgressBar
+ width - The total width the widget must fill
+ """
+
+
+class Timer(Widget):
+ """Widget which displays the elapsed seconds."""
+
+ __slots__ = ('format_string',)
+ TIME_SENSITIVE = True
+
+ def __init__(self, format='Elapsed Time: %s'):
+ self.format_string = format
+
+ @staticmethod
+ def format_time(seconds):
+ """Formats time as the string "HH:MM:SS"."""
+
+ return str(datetime.timedelta(seconds=int(seconds)))
+
+
+ def update(self, pbar):
+ """Updates the widget to show the elapsed time."""
+
+ return self.format_string % self.format_time(pbar.seconds_elapsed)
+
+
+class ETA(Timer):
+ """Widget which attempts to estimate the time of arrival."""
+
+ TIME_SENSITIVE = True
+
+ def update(self, pbar):
+ """Updates the widget to show the ETA or total time when finished."""
+
+ if pbar.currval == 0:
+ return 'ETA: --:--:--'
+ elif pbar.finished:
+ return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
+ else:
+ elapsed = pbar.seconds_elapsed
+ eta = elapsed * pbar.maxval / pbar.currval - elapsed
+ return 'ETA: %s' % self.format_time(eta)
+
+
+class AdaptiveETA(Timer):
+ """Widget which attempts to estimate the time of arrival.
+
+ Uses a weighted average of two estimates:
+ 1) ETA based on the total progress and time elapsed so far
+ 2) ETA based on the progress as per the last 10 update reports
+
+ The weight depends on the current progress so that to begin with the
+ total progress is used and at the end only the most recent progress is
+ used.
+ """
+
+ TIME_SENSITIVE = True
+ NUM_SAMPLES = 10
+
+ def _update_samples(self, currval, elapsed):
+ sample = (currval, elapsed)
+ if not hasattr(self, 'samples'):
+ self.samples = [sample] * (self.NUM_SAMPLES + 1)
+ else:
+ self.samples.append(sample)
+ return self.samples.pop(0)
+
+ def _eta(self, maxval, currval, elapsed):
+ return elapsed * maxval / float(currval) - elapsed
+
+ def update(self, pbar):
+ """Updates the widget to show the ETA or total time when finished."""
+ if pbar.currval == 0:
+ return 'ETA: --:--:--'
+ elif pbar.finished:
+ return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
+ else:
+ elapsed = pbar.seconds_elapsed
+ currval1, elapsed1 = self._update_samples(pbar.currval, elapsed)
+ eta = self._eta(pbar.maxval, pbar.currval, elapsed)
+ if pbar.currval > currval1:
+ etasamp = self._eta(pbar.maxval - currval1,
+ pbar.currval - currval1,
+ elapsed - elapsed1)
+ weight = (pbar.currval / float(pbar.maxval)) ** 0.5
+ eta = (1 - weight) * eta + weight * etasamp
+ return 'ETA: %s' % self.format_time(eta)
+
+
+class FileTransferSpeed(Widget):
+ """Widget for showing the transfer speed (useful for file transfers)."""
+
+ FORMAT = '%6.2f %s%s/s'
+ PREFIXES = ' kMGTPEZY'
+ __slots__ = ('unit',)
+
+ def __init__(self, unit='B'):
+ self.unit = unit
+
+ def update(self, pbar):
+ """Updates the widget with the current SI prefixed speed."""
+
+ if pbar.seconds_elapsed < 2e-6 or pbar.currval < 2e-6: # =~ 0
+ scaled = power = 0
+ else:
+ speed = pbar.currval / pbar.seconds_elapsed
+ power = int(math.log(speed, 1000))
+ scaled = speed / 1000.**power
+
+ return self.FORMAT % (scaled, self.PREFIXES[power], self.unit)
+
+
+class AnimatedMarker(Widget):
+ """An animated marker for the progress bar which defaults to appear as if
+ it were rotating.
+ """
+
+ __slots__ = ('markers', 'curmark')
+
+ def __init__(self, markers='|/-\\'):
+ self.markers = markers
+ self.curmark = -1
+
+ def update(self, pbar):
+ """Updates the widget to show the next marker or the first marker when
+ finished"""
+
+ if pbar.finished: return self.markers[0]
+
+ self.curmark = (self.curmark + 1) % len(self.markers)
+ return self.markers[self.curmark]
+
+# Alias for backwards compatibility
+RotatingMarker = AnimatedMarker
+
+
+class Counter(Widget):
+ """Displays the current count."""
+
+ __slots__ = ('format_string',)
+
+ def __init__(self, format='%d'):
+ self.format_string = format
+
+ def update(self, pbar):
+ return self.format_string % pbar.currval
+
+
+class Percentage(Widget):
+ """Displays the current percentage as a number with a percent sign."""
+
+ def update(self, pbar):
+ return '%3d%%' % pbar.percentage()
+
+
+class FormatLabel(Timer):
+ """Displays a formatted label."""
+
+ mapping = {
+ 'elapsed': ('seconds_elapsed', Timer.format_time),
+ 'finished': ('finished', None),
+ 'last_update': ('last_update_time', None),
+ 'max': ('maxval', None),
+ 'seconds': ('seconds_elapsed', None),
+ 'start': ('start_time', None),
+ 'value': ('currval', None)
+ }
+
+ __slots__ = ('format_string',)
+ def __init__(self, format):
+ self.format_string = format
+
+ def update(self, pbar):
+ context = {}
+ for name, (key, transform) in self.mapping.items():
+ try:
+ value = getattr(pbar, key)
+
+ if transform is None:
+ context[name] = value
+ else:
+ context[name] = transform(value)
+ except: pass
+
+ return self.format_string % context
+
+
+class SimpleProgress(Widget):
+ """Returns progress as a count of the total (e.g.: "5 of 47")."""
+
+ __slots__ = ('sep',)
+
+ def __init__(self, sep=' of '):
+ self.sep = sep
+
+ def update(self, pbar):
+ return '%d%s%d' % (pbar.currval, self.sep, pbar.maxval)
+
+
+class Bar(WidgetHFill):
+ """A progress bar which stretches to fill the line."""
+
+ __slots__ = ('marker', 'left', 'right', 'fill', 'fill_left')
+
+ def __init__(self, marker='#', left='|', right='|', fill=' ',
+ fill_left=True):
+ """Creates a customizable progress bar.
+
+ marker - string or updatable object to use as a marker
+ left - string or updatable object to use as a left border
+ right - string or updatable object to use as a right border
+ fill - character to use for the empty part of the progress bar
+ fill_left - whether to fill from the left or the right
+ """
+ self.marker = marker
+ self.left = left
+ self.right = right
+ self.fill = fill
+ self.fill_left = fill_left
+
+
+ def update(self, pbar, width):
+ """Updates the progress bar and its subcomponents."""
+
+ left, marked, right = (format_updatable(i, pbar) for i in
+ (self.left, self.marker, self.right))
+
+ width -= len(left) + len(right)
+ # Marked must *always* have length of 1
+ if pbar.maxval:
+ marked *= int(pbar.currval / pbar.maxval * width)
+ else:
+ marked = ''
+
+ if self.fill_left:
+ return '%s%s%s' % (left, marked.ljust(width, self.fill), right)
+ else:
+ return '%s%s%s' % (left, marked.rjust(width, self.fill), right)
+
+
+class ReverseBar(Bar):
+ """A bar which has a marker which bounces from side to side."""
+
+ def __init__(self, marker='#', left='|', right='|', fill=' ',
+ fill_left=False):
+ """Creates a customizable progress bar.
+
+ marker - string or updatable object to use as a marker
+ left - string or updatable object to use as a left border
+ right - string or updatable object to use as a right border
+ fill - character to use for the empty part of the progress bar
+ fill_left - whether to fill from the left or the right
+ """
+ self.marker = marker
+ self.left = left
+ self.right = right
+ self.fill = fill
+ self.fill_left = fill_left
+
+
+class BouncingBar(Bar):
+ def update(self, pbar, width):
+ """Updates the progress bar and its subcomponents."""
+
+ left, marker, right = (format_updatable(i, pbar) for i in
+ (self.left, self.marker, self.right))
+
+ width -= len(left) + len(right)
+
+ if pbar.finished: return '%s%s%s' % (left, width * marker, right)
+
+ position = int(pbar.currval % (width * 2 - 1))
+ if position > width: position = width * 2 - position
+ lpad = self.fill * (position - 1)
+ rpad = self.fill * (width - len(marker) - len(lpad))
+
+ # Swap if we want to bounce the other way
+ if not self.fill_left: rpad, lpad = lpad, rpad
+
+ return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 03/10] lib: implement basic task progress support
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 01/10] knotty: provide a symlink to the latest console log Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 02/10] knotty: import latest python-progressbar Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 04/10] lib/bb/progress: add MultiStageProgressReporter Paul Eggleton
` (6 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
For long-running tasks where we have some output from the task that
gives us some idea of the progress of the task (such as a percentage
complete), provide the means to scrape the output for that progress
information and show it to the user in the default knotty terminal
output in the form of a progress bar. This is implemented using a new
TaskProgress event as well as some code we can insert to do output
scanning/filtering.
Any task can fire TaskProgress events; however, if you have a shell task
whose output you wish to scan for progress information, you just need to
set the "progress" varflag on the task. This can be set to:
* "percent" to just look for a number followed by a % sign
* "percent:<regex>" to specify your own regex matching a percentage
value (must have a single group which matches the percentage number)
* "outof:<regex>" to look for the specified regex matching x out of y
items completed (must have two groups - first group needs to be x,
second y).
We can potentially extend this in future but this should be a good
start.
Part of the implementation for [YOCTO #5383].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/build.py | 34 +++++++++++++++++
lib/bb/progress.py | 86 ++++++++++++++++++++++++++++++++++++++++++
lib/bb/ui/knotty.py | 74 +++++++++++++++++++++++++++++++-----
lib/bb/ui/uihelper.py | 7 +++-
lib/progressbar/progressbar.py | 16 ++++++--
lib/progressbar/widgets.py | 36 ++++++++++++++++++
6 files changed, 239 insertions(+), 14 deletions(-)
create mode 100644 lib/bb/progress.py
diff --git a/lib/bb/build.py b/lib/bb/build.py
index e016ae3..04bd580 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -35,6 +35,7 @@ import stat
import bb
import bb.msg
import bb.process
+import bb.progress
from bb import data, event, utils
bblogger = logging.getLogger('BitBake')
@@ -137,6 +138,25 @@ class TaskInvalid(TaskBase):
super(TaskInvalid, self).__init__(task, None, metadata)
self._message = "No such task '%s'" % task
+class TaskProgress(event.Event):
+ """
+ Task made some progress that could be reported to the user, usually in
+ the form of a progress bar or similar.
+ NOTE: this class does not inherit from TaskBase since it doesn't need
+ to - it's fired within the task context itself, so we don't have any of
+ the context information that you do in the case of the other events.
+ The event PID can be used to determine which task it came from.
+ The progress value is normally 0-100, but can also be negative
+ indicating that progress has been made but we aren't able to determine
+ how much.
+ The rate is optional, this is simply an extra string to display to the
+ user if specified.
+ """
+ def __init__(self, progress, rate=None):
+ self.progress = progress
+ self.rate = rate
+ event.Event.__init__(self)
+
class LogTee(object):
def __init__(self, logger, outfile):
@@ -340,6 +360,20 @@ exit $ret
else:
logfile = sys.stdout
+ progress = d.getVarFlag(func, 'progress', True)
+ if progress:
+ if progress == 'percent':
+ # Use default regex
+ logfile = bb.progress.BasicProgressHandler(d, outfile=logfile)
+ elif progress.startswith('percent:'):
+ # Use specified regex
+ logfile = bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+ elif progress.startswith('outof:'):
+ # Use specified regex
+ logfile = bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+ else:
+ bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
+
def readfifo(data):
lines = data.split(b'\0')
for line in lines:
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
new file mode 100644
index 0000000..bab8e94
--- /dev/null
+++ b/lib/bb/progress.py
@@ -0,0 +1,86 @@
+"""
+BitBake progress handling code
+"""
+
+# Copyright (C) 2016 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import sys
+import re
+import time
+import bb.event
+import bb.build
+
+class ProgressHandler(object):
+ """
+ Base class that can pretend to be a file object well enough to be
+ used to build objects to intercept console output and determine the
+ progress of some operation.
+ """
+ def __init__(self, d, outfile=None):
+ self._progress = 0
+ self._data = d
+ self._lastevent = 0
+ if outfile:
+ self._outfile = outfile
+ else:
+ self._outfile = sys.stdout
+
+ def _fire_progress(self, taskprogress, rate=None):
+ """Internal function to fire the progress event"""
+ bb.event.fire(bb.build.TaskProgress(taskprogress, rate), self._data)
+
+ def write(self, string):
+ self._outfile.write(string)
+
+ def flush(self):
+ self._outfile.flush()
+
+ def update(self, progress, rate=None):
+ ts = time.time()
+ if progress > 100:
+ progress = 100
+ if progress != self._progress or self._lastevent + 1 < ts:
+ self._fire_progress(progress, rate)
+ self._lastevent = ts
+ self._progress = progress
+
+class BasicProgressHandler(ProgressHandler):
+ def __init__(self, d, regex=r'(\d+)%', outfile=None):
+ super(BasicProgressHandler, self).__init__(d, outfile)
+ self._regex = re.compile(regex)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(0)
+
+ def write(self, string):
+ percs = self._regex.findall(string)
+ if percs:
+ progress = int(percs[-1])
+ self.update(progress)
+ super(BasicProgressHandler, self).write(string)
+
+class OutOfProgressHandler(ProgressHandler):
+ def __init__(self, d, regex, outfile=None):
+ super(OutOfProgressHandler, self).__init__(d, outfile)
+ self._regex = re.compile(regex)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(0)
+
+ def write(self, string):
+ nums = self._regex.findall(string)
+ if nums:
+ progress = (float(nums[-1][0]) / float(nums[-1][1])) * 100
+ self.update(progress)
+ super(OutOfProgressHandler, self).write(string)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 6a6f688..2513501 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -40,10 +40,13 @@ logger = logging.getLogger("BitBake")
interactive = sys.stdout.isatty()
class BBProgress(progressbar.ProgressBar):
- def __init__(self, msg, maxval):
+ def __init__(self, msg, maxval, widgets=None):
self.msg = msg
- widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
- progressbar.ETA()]
+ self.extrapos = -1
+ if not widgets:
+ widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
+ progressbar.ETA()]
+ self.extrapos = 4
try:
self._resize_default = signal.getsignal(signal.SIGWINCH)
@@ -55,11 +58,31 @@ class BBProgress(progressbar.ProgressBar):
progressbar.ProgressBar._handle_resize(self, signum, frame)
if self._resize_default:
self._resize_default(signum, frame)
+
def finish(self):
progressbar.ProgressBar.finish(self)
if self._resize_default:
signal.signal(signal.SIGWINCH, self._resize_default)
+ def setmessage(self, msg):
+ self.msg = msg
+ self.widgets[0] = msg
+
+ def setextra(self, extra):
+ if extra:
+ extrastr = str(extra)
+ if extrastr[0] != ' ':
+ extrastr = ' ' + extrastr
+ if extrastr[-1] != ' ':
+ extrastr += ' '
+ else:
+ extrastr = ' '
+ self.widgets[self.extrapos] = extrastr
+
+ def _need_update(self):
+ # We always want the bar to print when update() is called
+ return True
+
class NonInteractiveProgress(object):
fobj = sys.stdout
@@ -195,15 +218,31 @@ class TerminalFilter(object):
activetasks = self.helper.running_tasks
failedtasks = self.helper.failed_tasks
runningpids = self.helper.running_pids
- if self.footer_present and (self.lastcount == self.helper.tasknumber_current) and (self.lastpids == runningpids):
+ if self.footer_present and not self.helper.needUpdate:
return
+ self.helper.needUpdate = False
if self.footer_present:
self.clearFooter()
if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
return
tasks = []
for t in runningpids:
- tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+ progress = activetasks[t].get("progress", None)
+ if progress is not None:
+ pbar = activetasks[t].get("progressbar", None)
+ rate = activetasks[t].get("rate", None)
+ start_time = activetasks[t].get("starttime", None)
+ if not pbar or pbar.bouncing != (progress < 0):
+ if progress < 0:
+ pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100, widgets=[progressbar.BouncingSlider()])
+ pbar.bouncing = True
+ else:
+ pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100)
+ pbar.bouncing = False
+ activetasks[t]["progressbar"] = pbar
+ tasks.append((pbar, progress, rate, start_time))
+ else:
+ tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
if self.main.shutdown:
content = "Waiting for %s running tasks to finish:" % len(activetasks)
@@ -214,8 +253,23 @@ class TerminalFilter(object):
print(content)
lines = 1 + int(len(content) / (self.columns + 1))
for tasknum, task in enumerate(tasks[:(self.rows - 2)]):
- content = "%s: %s" % (tasknum, task)
- print(content)
+ if isinstance(task, tuple):
+ pbar, progress, rate, start_time = task
+ if not pbar.start_time:
+ pbar.start(False)
+ if start_time:
+ pbar.start_time = start_time
+ pbar.setmessage('%s:%s' % (tasknum, pbar.msg.split(':', 1)[1]))
+ if progress > -1:
+ pbar.setextra(rate)
+ output = pbar.update(progress)
+ else:
+ output = pbar.update(1)
+ if not output or (len(output) <= pbar.term_width):
+ print('')
+ else:
+ content = "%s: %s" % (tasknum, task)
+ print(content)
lines = lines + 1 + int(len(content) / (self.columns + 1))
self.footer_present = lines
self.lastpids = runningpids[:]
@@ -249,7 +303,8 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
"bb.command.CommandExit", "bb.command.CommandCompleted", "bb.cooker.CookerExit",
"bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted",
"bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed",
- "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent"]
+ "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
+ "bb.build.TaskProgress"]
def main(server, eventHandler, params, tf = TerminalFilter):
@@ -535,7 +590,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
bb.event.OperationStarted,
bb.event.OperationCompleted,
bb.event.OperationProgress,
- bb.event.DiskFull)):
+ bb.event.DiskFull,
+ bb.build.TaskProgress)):
continue
logger.error("Unknown event: %s", event)
diff --git a/lib/bb/ui/uihelper.py b/lib/bb/ui/uihelper.py
index db70b76..1915e47 100644
--- a/lib/bb/ui/uihelper.py
+++ b/lib/bb/ui/uihelper.py
@@ -18,6 +18,7 @@
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import bb.build
+import time
class BBUIHelper:
def __init__(self):
@@ -31,7 +32,7 @@ class BBUIHelper:
def eventHandler(self, event):
if isinstance(event, bb.build.TaskStarted):
- self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task) }
+ self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time() }
self.running_pids.append(event.pid)
self.needUpdate = True
if isinstance(event, bb.build.TaskSucceeded):
@@ -52,6 +53,10 @@ class BBUIHelper:
self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
self.tasknumber_total = event.stats.total
self.needUpdate = True
+ if isinstance(event, bb.build.TaskProgress):
+ self.running_tasks[event.pid]['progress'] = event.progress
+ self.running_tasks[event.pid]['rate'] = event.rate
+ self.needUpdate = True
def getTasks(self):
self.needUpdate = False
diff --git a/lib/progressbar/progressbar.py b/lib/progressbar/progressbar.py
index 0b9dcf7..2873ad6 100644
--- a/lib/progressbar/progressbar.py
+++ b/lib/progressbar/progressbar.py
@@ -3,6 +3,8 @@
# progressbar - Text progress bar library for Python.
# Copyright (c) 2005 Nilton Volpato
#
+# (With some small changes after importing into BitBake)
+#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
@@ -261,12 +263,14 @@ class ProgressBar(object):
now = time.time()
self.seconds_elapsed = now - self.start_time
self.next_update = self.currval + self.update_interval
- self.fd.write(self._format_line() + '\r')
+ output = self._format_line()
+ self.fd.write(output + '\r')
self.fd.flush()
self.last_update_time = now
+ return output
- def start(self):
+ def start(self, update=True):
"""Starts measuring time, and prints the bar at 0%.
It returns self so you can use it like this:
@@ -289,8 +293,12 @@ class ProgressBar(object):
self.update_interval = self.maxval / self.num_intervals
- self.start_time = self.last_update_time = time.time()
- self.update(0)
+ self.start_time = time.time()
+ if update:
+ self.last_update_time = self.start_time
+ self.update(0)
+ else:
+ self.last_update_time = 0
return self
diff --git a/lib/progressbar/widgets.py b/lib/progressbar/widgets.py
index 6434ad5..77285ca 100644
--- a/lib/progressbar/widgets.py
+++ b/lib/progressbar/widgets.py
@@ -353,3 +353,39 @@ class BouncingBar(Bar):
if not self.fill_left: rpad, lpad = lpad, rpad
return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
+
+
+class BouncingSlider(Bar):
+ """
+ A slider that bounces back and forth in response to update() calls
+ without reference to the actual value. Based on a combination of
+ BouncingBar from a newer version of this module and RotatingMarker.
+ """
+ def __init__(self, marker='<=>'):
+ self.curmark = -1
+ self.forward = True
+ Bar.__init__(self, marker=marker)
+ def update(self, pbar, width):
+ left, marker, right = (format_updatable(i, pbar) for i in
+ (self.left, self.marker, self.right))
+
+ width -= len(left) + len(right)
+ if width < 0:
+ return ''
+
+ if pbar.finished: return '%s%s%s' % (left, width * '=', right)
+
+ self.curmark = self.curmark + 1
+ position = int(self.curmark % (width * 2 - 1))
+ if position + len(marker) > width:
+ self.forward = not self.forward
+ self.curmark = 1
+ position = 1
+ lpad = ' ' * (position - 1)
+ rpad = ' ' * (width - len(marker) - len(lpad))
+
+ if not self.forward:
+ temp = lpad
+ lpad = rpad
+ rpad = temp
+ return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 04/10] lib/bb/progress: add MultiStageProgressReporter
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
` (2 preceding siblings ...)
2016-06-23 10:59 ` [PATCH v2 03/10] lib: implement basic task progress support Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 05/10] fetch2: implement progress support Paul Eggleton
` (5 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
Add a class to help report progress in a task that consists of multiple
stages, some of which may have internal progress (do_rootfs within
OpenEmbedded is one example). Each stage is weighted to try to give
a reasonable representation of progress over time.
Part of the implementation for [YOCTO #5383].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/progress.py | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 111 insertions(+)
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
index bab8e94..93e42df 100644
--- a/lib/bb/progress.py
+++ b/lib/bb/progress.py
@@ -20,6 +20,7 @@ BitBake progress handling code
import sys
import re
import time
+import inspect
import bb.event
import bb.build
@@ -84,3 +85,113 @@ class OutOfProgressHandler(ProgressHandler):
progress = (float(nums[-1][0]) / float(nums[-1][1])) * 100
self.update(progress)
super(OutOfProgressHandler, self).write(string)
+
+class MultiStageProgressReporter(object):
+ """
+ Class which allows reporting progress without the caller
+ having to know where they are in the overall sequence. Useful
+ for tasks made up of python code spread across multiple
+ classes / functions - the progress reporter object can
+ be passed around or stored at the object level and calls
+ to next_stage() and update() made whereever needed.
+ """
+ def __init__(self, d, stage_weights, debug=False):
+ """
+ Initialise the progress reporter.
+
+ Parameters:
+ * d: the datastore (needed for firing the events)
+ * stage_weights: a list of weight values, one for each stage.
+ The value is scaled internally so you only need to specify
+ values relative to other values in the list, so if there
+ are two stages and the first takes 2s and the second takes
+ 10s you would specify [2, 10] (or [1, 5], it doesn't matter).
+ * debug: specify True (and ensure you call finish() at the end)
+ in order to show a printout of the calculated stage weights
+ based on timing each stage. Use this to determine what the
+ weights should be when you're not sure.
+ """
+ self._data = d
+ total = sum(stage_weights)
+ self._stage_weights = [float(x)/total for x in stage_weights]
+ self._stage = -1
+ self._base_progress = 0
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(0)
+ self._debug = debug
+ self._finished = False
+ if self._debug:
+ self._last_time = time.time()
+ self._stage_times = []
+ self._stage_total = None
+ self._callers = []
+
+ def _fire_progress(self, taskprogress):
+ bb.event.fire(bb.build.TaskProgress(taskprogress), self._data)
+
+ def next_stage(self, stage_total=None):
+ """
+ Move to the next stage.
+ Parameters:
+ * stage_total: optional total for progress within the stage,
+ see update() for details
+ NOTE: you need to call this before the first stage.
+ """
+ self._stage += 1
+ self._stage_total = stage_total
+ if self._stage == 0:
+ # First stage
+ if self._debug:
+ self._last_time = time.time()
+ else:
+ if self._stage < len(self._stage_weights):
+ self._base_progress = sum(self._stage_weights[:self._stage]) * 100
+ if self._debug:
+ currtime = time.time()
+ self._stage_times.append(currtime - self._last_time)
+ self._last_time = currtime
+ self._callers.append(inspect.getouterframes(inspect.currentframe())[1])
+ elif not self._debug:
+ bb.warn('ProgressReporter: current stage beyond declared number of stages')
+ self._base_progress = 100
+ self._fire_progress(self._base_progress)
+
+ def update(self, stage_progress):
+ """
+ Update progress within the current stage.
+ Parameters:
+ * stage_progress: progress value within the stage. If stage_total
+ was specified when next_stage() was last called, then this
+ value is considered to be out of stage_total, otherwise it should
+ be a percentage value from 0 to 100.
+ """
+ if self._stage_total:
+ stage_progress = (float(stage_progress) / self._stage_total) * 100
+ if self._stage < 0:
+ bb.warn('ProgressReporter: update called before first call to next_stage()')
+ elif self._stage < len(self._stage_weights):
+ progress = self._base_progress + (stage_progress * self._stage_weights[self._stage])
+ else:
+ progress = self._base_progress
+ if progress > 100:
+ progress = 100
+ self._fire_progress(progress)
+
+ def finish(self):
+ if self._finished:
+ return
+ self._finished = True
+ if self._debug:
+ import math
+ self._stage_times.append(time.time() - self._last_time)
+ mintime = max(min(self._stage_times), 0.01)
+ self._callers.append(None)
+ stage_weights = [int(math.ceil(x / mintime)) for x in self._stage_times]
+ bb.warn('Stage weights: %s' % stage_weights)
+ out = []
+ for stage_weight, caller in zip(stage_weights, self._callers):
+ if caller:
+ out.append('Up to %s:%d: %d' % (caller[1], caller[2], stage_weight))
+ else:
+ out.append('Up to finish: %d' % stage_weight)
+ bb.warn('Stage times:\n %s' % '\n '.join(out))
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 05/10] fetch2: implement progress support
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
` (3 preceding siblings ...)
2016-06-23 10:59 ` [PATCH v2 04/10] lib/bb/progress: add MultiStageProgressReporter Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
2016-07-06 4:26 ` [PATCH v3] " Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 06/10] knotty: add code to support showing progress for sstate object querying Paul Eggleton
` (4 subsequent siblings)
9 siblings, 1 reply; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.
At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).
Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.
Part of the implementation for [YOCTO #5383].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/fetch2/__init__.py | 4 ++--
lib/bb/fetch2/git.py | 52 +++++++++++++++++++++++++++++++++++++++++++----
lib/bb/fetch2/wget.py | 26 +++++++++++++++++++++++-
lib/bb/progress.py | 31 ++++++++++++++++++++++++++++
4 files changed, 106 insertions(+), 7 deletions(-)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index b6fcaaa..a27512c 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -779,7 +779,7 @@ def localpath(url, d):
fetcher = bb.fetch2.Fetch([url], d)
return fetcher.localpath(url)
-def runfetchcmd(cmd, d, quiet=False, cleanup=None):
+def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None):
"""
Run cmd returning the command output
Raise an error if interrupted or cmd fails
@@ -821,7 +821,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None):
error_message = ""
try:
- (output, errors) = bb.process.run(cmd, shell=True, stderr=subprocess.PIPE)
+ (output, errors) = bb.process.run(cmd, log=log, shell=True, stderr=subprocess.PIPE)
success = True
except bb.process.NotFoundError as e:
error_message = "Fetch command %s" % (e.command)
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 59827e3..4e2dcec 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -71,11 +71,53 @@ import os
import re
import bb
import errno
+import bb.progress
from bb import data
from bb.fetch2 import FetchMethod
from bb.fetch2 import runfetchcmd
from bb.fetch2 import logger
+
+class GitProgressHandler(bb.progress.LineFilterProgressHandler):
+ """Extract progress information from git output"""
+ def __init__(self, d):
+ self._buffer = ''
+ self._count = 0
+ super(GitProgressHandler, self).__init__(d)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(-1)
+
+ def write(self, string):
+ self._buffer += string
+ stages = ['Counting objects', 'Compressing objects', 'Receiving objects', 'Resolving deltas']
+ stage_weights = [0.2, 0.05, 0.5, 0.25]
+ stagenum = 0
+ for i, stage in reversed(list(enumerate(stages))):
+ if stage in self._buffer:
+ stagenum = i
+ self._buffer = ''
+ break
+ self._status = stages[stagenum]
+ percs = re.findall(r'(\d+)%', string)
+ if percs:
+ progress = int(round((int(percs[-1]) * stage_weights[stagenum]) + (sum(stage_weights[:stagenum]) * 100)))
+ rates = re.findall(r'([\d.]+ [a-zA-Z]*/s+)', string)
+ if rates:
+ rate = rates[-1]
+ else:
+ rate = None
+ self.update(progress, rate)
+ else:
+ if stagenum == 0:
+ percs = re.findall(r': (\d+)', string)
+ if percs:
+ count = int(percs[-1])
+ if count > self._count:
+ self._count = count
+ self._fire_progress(-count)
+ super(GitProgressHandler, self).write(string)
+
+
class Git(FetchMethod):
"""Class to fetch a module or modules from git repositories"""
def init(self, d):
@@ -196,10 +238,11 @@ class Git(FetchMethod):
# We do this since git will use a "-l" option automatically for local urls where possible
if repourl.startswith("file://"):
repourl = repourl[7:]
- clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
+ clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, repourl, ud.clonedir)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, clone_cmd)
- runfetchcmd(clone_cmd, d)
+ progresshandler = GitProgressHandler(d)
+ runfetchcmd(clone_cmd, d, log=progresshandler)
os.chdir(ud.clonedir)
# Update the checkout if needed
@@ -214,10 +257,11 @@ class Git(FetchMethod):
logger.debug(1, "No Origin")
runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, repourl), d)
- fetch_cmd = "%s fetch -f --prune %s refs/*:refs/*" % (ud.basecmd, repourl)
+ fetch_cmd = "LANG=C %s fetch -f --prune --progress %s refs/*:refs/*" % (ud.basecmd, repourl)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
- runfetchcmd(fetch_cmd, d)
+ progresshandler = GitProgressHandler(d)
+ runfetchcmd(fetch_cmd, d, log=progresshandler)
runfetchcmd("%s prune-packed" % ud.basecmd, d)
runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
try:
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index d688fd9..3a381b7 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -31,6 +31,7 @@ import subprocess
import os
import logging
import bb
+import bb.progress
import urllib.request, urllib.parse, urllib.error
from bb import data
from bb.fetch2 import FetchMethod
@@ -41,6 +42,27 @@ from bb.utils import export_proxies
from bs4 import BeautifulSoup
from bs4 import SoupStrainer
+class WgetProgressHandler(bb.progress.LineFilterProgressHandler):
+ """
+ Extract progress information from wget output.
+ Note: relies on --progress=dot --show-progress being specified on the
+ wget command line.
+ """
+ def __init__(self, d):
+ super(WgetProgressHandler, self).__init__(d)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(0)
+
+ def writeline(self, line):
+ percs = re.findall(r'(\d+)%\s+([\d.]+[A-Z])', line)
+ if percs:
+ progress = int(percs[-1][0])
+ rate = percs[-1][1] + '/s'
+ self.update(progress, rate)
+ return False
+ return True
+
+
class Wget(FetchMethod):
"""Class to fetch urls via 'wget'"""
def supports(self, ud, d):
@@ -70,9 +92,11 @@ class Wget(FetchMethod):
def _runwget(self, ud, d, command, quiet):
+ progresshandler = WgetProgressHandler(d)
+
logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command))
bb.fetch2.check_network_access(d, command)
- runfetchcmd(command, d, quiet)
+ runfetchcmd(command + ' --progress=dot --show-progress', d, quiet, log=progresshandler)
def download(self, ud, d):
"""Fetch urls"""
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
index 93e42df..1365068 100644
--- a/lib/bb/progress.py
+++ b/lib/bb/progress.py
@@ -58,6 +58,37 @@ class ProgressHandler(object):
self._lastevent = ts
self._progress = progress
+class LineFilterProgressHandler(ProgressHandler):
+ """
+ A ProgressHandler variant that provides the ability to filter out
+ the lines if they contain progress information. Additionally, it
+ filters out anything before the last line feed on a line. This can
+ be used to keep the logs clean of output that we've only enabled for
+ getting progress, assuming that that can be done on a per-line
+ basis.
+ """
+ def __init__(self, d, outfile=None):
+ self._linebuffer = ''
+ super(LineFilterProgressHandler, self).__init__(d, outfile)
+
+ def write(self, string):
+ self._linebuffer += string
+ while True:
+ breakpos = self._linebuffer.find('\n') + 1
+ if breakpos == 0:
+ break
+ line = self._linebuffer[:breakpos]
+ self._linebuffer = self._linebuffer[breakpos:]
+ # Drop any line feeds and anything that precedes them
+ lbreakpos = line.rfind('\r') + 1
+ if lbreakpos:
+ line = line[lbreakpos:]
+ if self.writeline(line):
+ super(LineFilterProgressHandler, self).write(line)
+
+ def writeline(self, line):
+ return True
+
class BasicProgressHandler(ProgressHandler):
def __init__(self, d, regex=r'(\d+)%', outfile=None):
super(BasicProgressHandler, self).__init__(d, outfile)
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 06/10] knotty: add code to support showing progress for sstate object querying
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
` (4 preceding siblings ...)
2016-06-23 10:59 ` [PATCH v2 05/10] fetch2: implement progress support Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 07/10] knotty: show task progress bar Paul Eggleton
` (3 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
Add support code on the BitBake side to allow sstate.bbclass in
OpenEmbedded to report progress when it is checking for availability of
artifacts from shared state mirrors.
Part of the implementation for [YOCTO #5853].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/event.py | 27 +++++++++++++++++++++++++++
lib/bb/ui/knotty.py | 15 +++++++++++++--
2 files changed, 40 insertions(+), 2 deletions(-)
diff --git a/lib/bb/event.py b/lib/bb/event.py
index 9b4a4f9..a5f026e 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -647,6 +647,33 @@ class MetadataEvent(Event):
self.type = eventtype
self._localdata = eventdata
+class ProcessStarted(Event):
+ """
+ Generic process started event (usually part of the initial startup)
+ where further progress events will be delivered
+ """
+ def __init__(self, processname, total):
+ Event.__init__(self)
+ self.processname = processname
+ self.total = total
+
+class ProcessProgress(Event):
+ """
+ Generic process progress event (usually part of the initial startup)
+ """
+ def __init__(self, processname, progress):
+ Event.__init__(self)
+ self.processname = processname
+ self.progress = progress
+
+class ProcessFinished(Event):
+ """
+ Generic process finished event (usually part of the initial startup)
+ """
+ def __init__(self, processname):
+ Event.__init__(self)
+ self.processname = processname
+
class SanityCheck(Event):
"""
Event to run sanity checks, either raise errors or generate events as return status.
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 2513501..6fdaafe 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -90,7 +90,7 @@ class NonInteractiveProgress(object):
self.msg = msg
self.maxval = maxval
- def start(self):
+ def start(self, update=True):
self.fobj.write("%s..." % self.msg)
self.fobj.flush()
return self
@@ -304,7 +304,7 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
"bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted",
"bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed",
"bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
- "bb.build.TaskProgress"]
+ "bb.build.TaskProgress", "bb.event.ProcessStarted", "bb.event.ProcessProgress", "bb.event.ProcessFinished"]
def main(server, eventHandler, params, tf = TerminalFilter):
@@ -579,6 +579,17 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if isinstance(event, bb.event.DepTreeGenerated):
continue
+ if isinstance(event, bb.event.ProcessStarted):
+ parseprogress = new_progress(event.processname, event.total)
+ parseprogress.start(False)
+ continue
+ if isinstance(event, bb.event.ProcessProgress):
+ parseprogress.update(event.progress)
+ continue
+ if isinstance(event, bb.event.ProcessFinished):
+ parseprogress.finish()
+ continue
+
# ignore
if isinstance(event, (bb.event.BuildBase,
bb.event.MetadataEvent,
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 07/10] knotty: show task progress bar
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
` (5 preceding siblings ...)
2016-06-23 10:59 ` [PATCH v2 06/10] knotty: add code to support showing progress for sstate object querying Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 08/10] knotty: add quiet output mode Paul Eggleton
` (2 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
In addition to the "currently running n tasks (x of y)" message, show a
progress bar for another view on how much of the build is left. We have
to take care to reset it when moving from the scenequeue to the
runqueue, and explicitly don't include an ETA since not all tasks take
equal time and thus it isn't possible to estimate the time remaining
with the information available.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/ui/knotty.py | 23 +++++++++++++++++++----
1 file changed, 19 insertions(+), 4 deletions(-)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 6fdaafe..c245c22 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -89,6 +89,7 @@ class NonInteractiveProgress(object):
def __init__(self, msg, maxval):
self.msg = msg
self.maxval = maxval
+ self.finished = False
def start(self, update=True):
self.fobj.write("%s..." % self.msg)
@@ -99,8 +100,11 @@ class NonInteractiveProgress(object):
pass
def finish(self):
+ if self.finished:
+ return
self.fobj.write("done.\n")
self.fobj.flush()
+ self.finished = True
def new_progress(msg, maxval):
if interactive:
@@ -204,6 +208,8 @@ class TerminalFilter(object):
console.addFilter(InteractConsoleLogFilter(self, format))
errconsole.addFilter(InteractConsoleLogFilter(self, format))
+ self.main_progress = None
+
def clearFooter(self):
if self.footer_present:
lines = self.footer_present
@@ -246,11 +252,20 @@ class TerminalFilter(object):
if self.main.shutdown:
content = "Waiting for %s running tasks to finish:" % len(activetasks)
- elif not len(activetasks):
- content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+ print(content)
else:
- content = "Currently %s running tasks (%s of %s):" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
- print(content)
+ if not len(activetasks):
+ content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+ else:
+ content = "Currently %2s running tasks (%s of %s)" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
+ maxtask = self.helper.tasknumber_total + 1
+ if not self.main_progress or self.main_progress.maxval != maxtask:
+ widgets = [' ', progressbar.Percentage(), ' ', progressbar.Bar()]
+ self.main_progress = BBProgress("Running tasks", maxtask, widgets=widgets)
+ self.main_progress.start(False)
+ self.main_progress.setmessage(content)
+ self.main_progress.update(self.helper.tasknumber_current)
+ print('')
lines = 1 + int(len(content) / (self.columns + 1))
for tasknum, task in enumerate(tasks[:(self.rows - 2)]):
if isinstance(task, tuple):
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 08/10] knotty: add quiet output mode
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
` (6 preceding siblings ...)
2016-06-23 10:59 ` [PATCH v2 07/10] knotty: show task progress bar Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 09/10] runqueue: add ability to enforce that tasks are setscened Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 10/10] runqueue: report progress for "Preparing RunQueue" step Paul Eggleton
9 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
Quiet output mode disables printing most messages (below warnings) to
the console; however these messages still go to the console log file.
This is primarily for cases where bitbake is being launched
interactively from some other process, but where full console output is
not needed.
Because of the need to keep logging all normal events to the console
log, this functionality was implemented within the knotty UI rather
than in bb.msg (where verbose mode is implemented). We don't currently
have a means of registering command line options from the UI end, thus
the option actually has to be registered in main.py regardless of the
UI, however I didn't feel like it was worth setting up such a mechanism
just for this option.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/main.py | 9 +++++++++
lib/bb/msg.py | 5 ++++-
lib/bb/ui/knotty.py | 57 ++++++++++++++++++++++++++++++-----------------------
3 files changed, 45 insertions(+), 26 deletions(-)
diff --git a/lib/bb/main.py b/lib/bb/main.py
index 283f29b..3fc3ff5 100755
--- a/lib/bb/main.py
+++ b/lib/bb/main.py
@@ -179,6 +179,9 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
parser.add_option("-D", "--debug", action="count", dest="debug", default=0,
help="Increase the debug level. You can specify this more than once.")
+ parser.add_option("-q", "--quiet", action="store_true", dest="quiet", default=False,
+ help="Output less log message data to the terminal.")
+
parser.add_option("-n", "--dry-run", action="store_true", dest="dry_run", default=False,
help="Don't execute, just go through the motions.")
@@ -279,6 +282,12 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
options, targets = parser.parse_args(argv)
+ if options.quiet and options.verbose:
+ parser.error("options --quiet and --verbose are mutually exclusive")
+
+ if options.quiet and options.debug:
+ parser.error("options --quiet and --debug are mutually exclusive")
+
# use configuration files from environment variables
if "BBPRECONF" in os.environ:
options.prefile.append(os.environ["BBPRECONF"])
diff --git a/lib/bb/msg.py b/lib/bb/msg.py
index 8c3ab47..b7c39fa 100644
--- a/lib/bb/msg.py
+++ b/lib/bb/msg.py
@@ -182,9 +182,12 @@ def constructLogOptions():
debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
return level, debug_domains
-def addDefaultlogFilter(handler, cls = BBLogFilter):
+def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
level, debug_domains = constructLogOptions()
+ if forcelevel is not None:
+ level = forcelevel
+
cls(handler, level, debug_domains)
#
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index c245c22..dbcb9c4 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -161,7 +161,7 @@ class TerminalFilter(object):
cr = (25, 80)
return cr
- def __init__(self, main, helper, console, errconsole, format):
+ def __init__(self, main, helper, console, errconsole, format, quiet):
self.main = main
self.helper = helper
self.cuu = None
@@ -169,6 +169,7 @@ class TerminalFilter(object):
self.interactive = sys.stdout.isatty()
self.footer_present = False
self.lastpids = []
+ self.quiet = quiet
if not self.interactive:
return
@@ -267,25 +268,26 @@ class TerminalFilter(object):
self.main_progress.update(self.helper.tasknumber_current)
print('')
lines = 1 + int(len(content) / (self.columns + 1))
- for tasknum, task in enumerate(tasks[:(self.rows - 2)]):
- if isinstance(task, tuple):
- pbar, progress, rate, start_time = task
- if not pbar.start_time:
- pbar.start(False)
- if start_time:
- pbar.start_time = start_time
- pbar.setmessage('%s:%s' % (tasknum, pbar.msg.split(':', 1)[1]))
- if progress > -1:
- pbar.setextra(rate)
- output = pbar.update(progress)
+ if not self.quiet:
+ for tasknum, task in enumerate(tasks[:(self.rows - 2)]):
+ if isinstance(task, tuple):
+ pbar, progress, rate, start_time = task
+ if not pbar.start_time:
+ pbar.start(False)
+ if start_time:
+ pbar.start_time = start_time
+ pbar.setmessage('%s:%s' % (tasknum, pbar.msg.split(':', 1)[1]))
+ if progress > -1:
+ pbar.setextra(rate)
+ output = pbar.update(progress)
+ else:
+ output = pbar.update(1)
+ if not output or (len(output) <= pbar.term_width):
+ print('')
else:
- output = pbar.update(1)
- if not output or (len(output) <= pbar.term_width):
- print('')
- else:
- content = "%s: %s" % (tasknum, task)
- print(content)
- lines = lines + 1 + int(len(content) / (self.columns + 1))
+ content = "%s: %s" % (tasknum, task)
+ print(content)
+ lines = lines + 1 + int(len(content) / (self.columns + 1))
self.footer_present = lines
self.lastpids = runningpids[:]
self.lastcount = self.helper.tasknumber_current
@@ -336,7 +338,10 @@ def main(server, eventHandler, params, tf = TerminalFilter):
errconsole = logging.StreamHandler(sys.stderr)
format_str = "%(levelname)s: %(message)s"
format = bb.msg.BBLogFormatter(format_str)
- bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut)
+ if params.options.quiet:
+ bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut, bb.msg.BBLogFormatter.WARNING)
+ else:
+ bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut)
bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
console.setFormatter(format)
errconsole.setFormatter(format)
@@ -399,7 +404,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
warnings = 0
taskfailures = []
- termfilter = tf(main, helper, console, errconsole, format)
+ termfilter = tf(main, helper, console, errconsole, format, params.options.quiet)
atexit.register(termfilter.finish)
while True:
@@ -498,8 +503,9 @@ def main(server, eventHandler, params, tf = TerminalFilter):
continue
parseprogress.finish()
- print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
- % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
+ if not params.options.quiet:
+ print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
+ % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
continue
if isinstance(event, bb.event.CacheLoadStarted):
@@ -510,7 +516,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
continue
if isinstance(event, bb.event.CacheLoadCompleted):
cacheprogress.finish()
- print("Loaded %d entries from dependency cache." % event.num_entries)
+ if not params.options.quiet:
+ print("Loaded %d entries from dependency cache." % event.num_entries)
continue
if isinstance(event, bb.command.CommandFailed):
@@ -670,7 +677,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if return_value and errors:
summary += pluralise("\nSummary: There was %s ERROR message shown, returning a non-zero exit code.",
"\nSummary: There were %s ERROR messages shown, returning a non-zero exit code.", errors)
- if summary:
+ if summary and not params.options.quiet:
print(summary)
if interrupted:
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 09/10] runqueue: add ability to enforce that tasks are setscened
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
` (7 preceding siblings ...)
2016-06-23 10:59 ` [PATCH v2 08/10] knotty: add quiet output mode Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 10/10] runqueue: report progress for "Preparing RunQueue" step Paul Eggleton
9 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
Add the ability to enter a mode where only a specified whitelist of
tasks can be executed outright; everything else must be successfully
provided in the form of a setscene task (or covered by a setscene task).
Any setscene failure outside of the whitelist will cause the build to
fail immediately instead of running the real task, and any real tasks
that would execute outside of the whitelist cause an immediate build
failure when it comes to executing the runqueue as well.
The mode is enabled by setting BB_SETSCENE_ENFORCE="1", and the
whitelist is specified through BB_SETSCENE_ENFORCE_WHITELIST, consisting
of pn:taskname pairs. A single % character can be substituted for the pn
value to match any target explicitly specified on the bitbake command
line. Wildcards * and ? can also be used as per standard unix file name
matching for both pn and taskname.
Part of the implementation of [YOCTO #9367].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/runqueue.py | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 80 insertions(+), 1 deletion(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index c9b6b84..b62a28a 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -240,6 +240,7 @@ class RunQueueData:
self.stampwhitelist = cfgData.getVar("BB_STAMP_WHITELIST", True) or ""
self.multi_provider_whitelist = (cfgData.getVar("MULTI_PROVIDER_WHITELIST", True) or "").split()
+ self.setscenewhitelist = get_setscene_enforce_whitelist(cfgData)
self.reset()
@@ -1596,8 +1597,51 @@ class RunQueueExecuteTasks(RunQueueExecute):
Run the tasks in a queue prepared by rqdata.prepare()
"""
- self.rq.read_workers()
+ if self.rqdata.setscenewhitelist:
+ # Check tasks that are going to run against the whitelist
+ def check_norun_task(tid, showerror=False):
+ fn = fn_from_tid(tid)
+ taskname = taskname_from_tid(tid)
+ # Ignore covered tasks
+ if tid in self.rq.scenequeue_covered:
+ return False
+ # Ignore stamped tasks
+ if self.rq.check_stamp_task(tid, taskname, cache=self.stampcache):
+ return False
+ # Ignore noexec tasks
+ taskdep = self.rqdata.dataCache.task_deps[fn]
+ if 'noexec' in taskdep and taskname in taskdep['noexec']:
+ return False
+ pn = self.rqdata.dataCache.pkg_fn[fn]
+ if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
+ if showerror:
+ if tid in self.rqdata.runq_setscene_tids:
+ logger.error('Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname))
+ else:
+ logger.error('Task %s.%s attempted to execute unexpectedly' % (pn, taskname))
+ return True
+ return False
+ # Look to see if any tasks that we think shouldn't run are going to
+ unexpected = False
+ for tid in self.rqdata.runtaskentries:
+ if check_norun_task(tid):
+ unexpected = True
+ break
+ if unexpected:
+ # Run through the tasks in the rough order they'd have executed and print errors
+ # (since the order can be useful - usually missing sstate for the last few tasks
+ # is the cause of the problem)
+ task = self.sched.next()
+ while task is not None:
+ check_norun_task(task, showerror=True)
+ self.task_skip(task, 'Setscene enforcement check')
+ task = self.sched.next()
+
+ self.rq.state = runQueueCleanUp
+ return True
+
+ self.rq.read_workers()
if self.stats.total == 0:
# nothing to do
@@ -1940,6 +1984,16 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.scenequeue_covered.add(task)
self.scenequeue_updatecounters(task)
+ def check_taskfail(self, task):
+ if self.rqdata.setscenewhitelist:
+ realtask = task.split('_setscene')[0]
+ fn = fn_from_tid(realtask)
+ taskname = taskname_from_tid(realtask)
+ pn = self.rqdata.dataCache.pkg_fn[fn]
+ if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
+ logger.error('Task %s.%s failed' % (pn, taskname + "_setscene"))
+ self.rq.state = runQueueCleanUp
+
def task_complete(self, task):
self.stats.taskCompleted()
bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
@@ -1950,6 +2004,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
bb.event.fire(sceneQueueTaskFailed(task, self.stats, result, self), self.cfgData)
self.scenequeue_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
+ self.check_taskfail(task)
def task_failoutright(self, task):
self.runq_running.add(task)
@@ -2231,3 +2286,27 @@ class runQueuePipe():
if len(self.queue) > 0:
print("Warning, worker left partial message: %s" % self.queue)
self.input.close()
+
+def get_setscene_enforce_whitelist(d):
+ if d.getVar('BB_SETSCENE_ENFORCE', True) != '1':
+ return None
+ whitelist = (d.getVar("BB_SETSCENE_ENFORCE_WHITELIST", True) or "").split()
+ outlist = []
+ for item in whitelist[:]:
+ if item.startswith('%:'):
+ for target in sys.argv[1:]:
+ if not target.startswith('-'):
+ outlist.append(target.split(':')[0] + ':' + item.split(':')[1])
+ else:
+ outlist.append(item)
+ return outlist
+
+def check_setscene_enforce_whitelist(pn, taskname, whitelist):
+ import fnmatch
+ if whitelist:
+ item = '%s:%s' % (pn, taskname)
+ for whitelist_item in whitelist:
+ if fnmatch.fnmatch(item, whitelist_item):
+ return True
+ return False
+ return True
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2 10/10] runqueue: report progress for "Preparing RunQueue" step
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
` (8 preceding siblings ...)
2016-06-23 10:59 ` [PATCH v2 09/10] runqueue: add ability to enforce that tasks are setscened Paul Eggleton
@ 2016-06-23 10:59 ` Paul Eggleton
9 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-06-23 10:59 UTC (permalink / raw)
To: bitbake-devel
When "Preparing RunQueue" shows up you can expect to wait up to 30
seconds while it works - which is a bit long to leave the user waiting
without any kind of output. Since the work being carried out during this
time is divided into stages such that it's practical to determine
internally how it's progressing, replace the message with a progress
bar.
Actually what happens during this time is two major steps rather than
just one - the runqueue preparation itself, followed by the
initialisation prior to running setscene tasks. I elected to have the
progress bar cover both as one (there doesn't appear to be much point in
doing otherwise from a user perspective). I did however describe it as
"initialising tasks".
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
lib/bb/progress.py | 42 ++++++++++++++++++++++++++++++++
lib/bb/runqueue.py | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 109 insertions(+), 3 deletions(-)
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
index 1365068..f8fa692 100644
--- a/lib/bb/progress.py
+++ b/lib/bb/progress.py
@@ -226,3 +226,45 @@ class MultiStageProgressReporter(object):
else:
out.append('Up to finish: %d' % stage_weight)
bb.warn('Stage times:\n %s' % '\n '.join(out))
+
+class MultiStageProcessProgressReporter(MultiStageProgressReporter):
+ """
+ Version of MultiStageProgressReporter intended for use with
+ standalone processes (such as preparing the runqueue)
+ """
+ def __init__(self, d, processname, stage_weights, debug=False):
+ self._processname = processname
+ MultiStageProgressReporter.__init__(self, d, stage_weights, debug)
+
+ def start(self):
+ bb.event.fire(bb.event.ProcessStarted(self._processname, 100), self._data)
+
+ def _fire_progress(self, taskprogress):
+ bb.event.fire(bb.event.ProcessProgress(self._processname, taskprogress), self._data)
+
+ def finish(self):
+ MultiStageProgressReporter.finish(self)
+ bb.event.fire(bb.event.ProcessFinished(self._processname), self._data)
+
+class DummyMultiStageProcessProgressReporter(MultiStageProgressReporter):
+ """
+ MultiStageProcessProgressReporter that takes the calls and does nothing
+ with them (to avoid a bunch of "if progress_reporter:" checks)
+ """
+ def __init__(self):
+ MultiStageProcessProgressReporter.__init__(self, "", None, [])
+
+ def _fire_progress(self, taskprogress, rate=None):
+ pass
+
+ def start(self):
+ pass
+
+ def next_stage(self, stage_total=None):
+ pass
+
+ def update(self, stage_progress):
+ pass
+
+ def finish(self):
+ pass
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index b62a28a..57be15a 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -241,6 +241,7 @@ class RunQueueData:
self.stampwhitelist = cfgData.getVar("BB_STAMP_WHITELIST", True) or ""
self.multi_provider_whitelist = (cfgData.getVar("MULTI_PROVIDER_WHITELIST", True) or "").split()
self.setscenewhitelist = get_setscene_enforce_whitelist(cfgData)
+ self.init_progress_reporter = bb.progress.DummyMultiStageProcessProgressReporter()
self.reset()
@@ -432,7 +433,8 @@ class RunQueueData:
# Nothing to do
return 0
- logger.info("Preparing RunQueue")
+ self.init_progress_reporter.start()
+ self.init_progress_reporter.next_stage()
# Step A - Work out a list of tasks to run
#
@@ -562,8 +564,9 @@ class RunQueueData:
# e.g. do_sometask[recrdeptask] = "do_someothertask"
# (makes sure sometask runs after someothertask of all DEPENDS, RDEPENDS and intertask dependencies, recursively)
# We need to do this separately since we need all of runtaskentries[*].depends to be complete before this is processed
+ self.init_progress_reporter.next_stage(len(recursivetasks))
extradeps = {}
- for tid in recursivetasks:
+ for taskcounter, tid in enumerate(recursivetasks):
extradeps[tid] = set(self.runtaskentries[tid].depends)
tasknames = recursivetasks[tid]
@@ -585,6 +588,7 @@ class RunQueueData:
if tid in recursiveitasks:
for dep in recursiveitasks[tid]:
generate_recdeps(dep)
+ self.init_progress_reporter.update(taskcounter)
# Remove circular references so that do_a[recrdeptask] = "do_a do_b" can work
for tid in recursivetasks:
@@ -600,6 +604,8 @@ class RunQueueData:
logger.debug(2, "Task %s contains self reference!", tid)
self.runtaskentries[tid].depends.remove(tid)
+ self.init_progress_reporter.next_stage()
+
# Step B - Mark all active tasks
#
# Start with the tasks we were asked to run and mark all dependencies
@@ -664,6 +670,8 @@ class RunQueueData:
else:
mark_active(tid, 1)
+ self.init_progress_reporter.next_stage()
+
# Step C - Prune all inactive tasks
#
# Once all active tasks are marked, prune the ones we don't need.
@@ -674,6 +682,8 @@ class RunQueueData:
del self.runtaskentries[tid]
delcount += 1
+ self.init_progress_reporter.next_stage()
+
#
# Step D - Sanity checks and computation
#
@@ -689,11 +699,15 @@ class RunQueueData:
logger.verbose("Assign Weightings")
+ self.init_progress_reporter.next_stage()
+
# Generate a list of reverse dependencies to ease future calculations
for tid in self.runtaskentries:
for dep in self.runtaskentries[tid].depends:
self.runtaskentries[dep].revdeps.add(tid)
+ self.init_progress_reporter.next_stage()
+
# Identify tasks at the end of dependency chains
# Error on circular dependency loops (length two)
endpoints = []
@@ -709,10 +723,14 @@ class RunQueueData:
logger.verbose("Compute totals (have %s endpoint(s))", len(endpoints))
+ self.init_progress_reporter.next_stage()
+
# Calculate task weights
# Check of higher length circular dependencies
self.runq_weight = self.calculate_task_weights(endpoints)
+ self.init_progress_reporter.next_stage()
+
# Sanity Check - Check for multiple tasks building the same provider
prov_list = {}
seen_fn = []
@@ -804,6 +822,8 @@ class RunQueueData:
else:
logger.error(msg)
+ self.init_progress_reporter.next_stage()
+
# Create a whitelist usable by the stamp checks
stampfnwhitelist = []
for entry in self.stampwhitelist.split():
@@ -813,6 +833,8 @@ class RunQueueData:
stampfnwhitelist.append(fn)
self.stampfnwhitelist = stampfnwhitelist
+ self.init_progress_reporter.next_stage()
+
# Iterate over the task list looking for tasks with a 'setscene' function
self.runq_setscene_tids = []
if not self.cooker.configuration.nosetscene:
@@ -837,6 +859,8 @@ class RunQueueData:
logger.verbose("Invalidate task %s, %s", taskname, fn)
bb.parse.siggen.invalidate_task(taskname, self.dataCache, fn)
+ self.init_progress_reporter.next_stage()
+
# Invalidate task if force mode active
if self.cooker.configuration.force:
for (fn, target) in self.target_pairs:
@@ -850,6 +874,8 @@ class RunQueueData:
st = "do_%s" % st
invalidate_task(fn, st, True)
+ self.init_progress_reporter.next_stage()
+
# Create and print to the logs a virtual/xxxx -> PN (fn) table
virtmap = taskData.get_providermap(prefix="virtual/")
virtpnmap = {}
@@ -859,6 +885,8 @@ class RunQueueData:
if hasattr(bb.parse.siggen, "tasks_resolved"):
bb.parse.siggen.tasks_resolved(virtmap, virtpnmap, self.dataCache)
+ self.init_progress_reporter.next_stage()
+
# Iterate over the task list and call into the siggen code
dealtwith = set()
todeal = set(self.runtaskentries)
@@ -1096,14 +1124,25 @@ class RunQueue:
if self.state is runQueuePrepare:
self.rqexe = RunQueueExecuteDummy(self)
+ # NOTE: if you add, remove or significantly refactor the stages of this
+ # process then you should recalculate the weightings here. This is quite
+ # easy to do - just change the next line temporarily to pass debug=True as
+ # the last parameter and you'll get a printout of the weightings as well
+ # as a map to the lines where next_stage() was called. Of course this isn't
+ # critical, but it helps to keep the progress reporting accurate.
+ self.rqdata.init_progress_reporter = bb.progress.MultiStageProcessProgressReporter(self.cooker.data,
+ "Initialising tasks",
+ [43, 967, 4, 3, 1, 5, 3, 7, 13, 1, 2, 1, 1, 246, 35, 1, 38, 1, 35, 2, 338, 204, 142, 3, 3, 37, 244])
if self.rqdata.prepare() == 0:
self.state = runQueueComplete
else:
self.state = runQueueSceneInit
+ self.rqdata.init_progress_reporter.next_stage()
# we are ready to run, emit dependency info to any UI or class which
# needs it
depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
+ self.rqdata.init_progress_reporter.next_stage()
bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
if self.state is runQueueSceneInit:
@@ -1116,7 +1155,9 @@ class RunQueue:
self.write_diffscenetasks(invalidtasks)
self.state = runQueueComplete
else:
+ self.rqdata.init_progress_reporter.next_stage()
self.start_worker()
+ self.rqdata.init_progress_reporter.next_stage()
self.rqexe = RunQueueExecuteScenequeue(self)
if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp]:
@@ -1129,6 +1170,8 @@ class RunQueue:
if self.cooker.configuration.setsceneonly:
self.state = runQueueComplete
else:
+ # Just in case we didn't setscene
+ self.rqdata.init_progress_reporter.finish()
logger.info("Executing RunQueue Tasks")
self.rqexe = RunQueueExecuteTasks(self)
self.state = runQueueRunning
@@ -1769,6 +1812,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
# therefore aims to collapse the huge runqueue dependency tree into a smaller one
# only containing the setscene functions.
+ self.rqdata.init_progress_reporter.next_stage()
+
# First process the chains up to the first setscene task.
endpoints = {}
for tid in self.rqdata.runtaskentries:
@@ -1778,6 +1823,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
#bb.warn("Added endpoint %s" % (tid))
endpoints[tid] = set()
+ self.rqdata.init_progress_reporter.next_stage()
+
# Secondly process the chains between setscene tasks.
for tid in self.rqdata.runq_setscene_tids:
#bb.warn("Added endpoint 2 %s" % (tid))
@@ -1787,6 +1834,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
#bb.warn(" Added endpoint 3 %s" % (dep))
endpoints[dep].add(tid)
+ self.rqdata.init_progress_reporter.next_stage()
+
def process_endpoints(endpoints):
newendpoints = {}
for point, task in endpoints.items():
@@ -1811,6 +1860,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
process_endpoints(endpoints)
+ self.rqdata.init_progress_reporter.next_stage()
+
# Build a list of setscene tasks which are "unskippable"
# These are direct endpoints referenced by the build
endpoints2 = {}
@@ -1847,7 +1898,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
if sq_revdeps_new2[tid]:
self.unskippable.append(tid)
- for tid in self.rqdata.runtaskentries:
+ self.rqdata.init_progress_reporter.next_stage(len(self.rqdata.runtaskentries))
+
+ for taskcounter, tid in enumerate(self.rqdata.runtaskentries):
if tid in self.rqdata.runq_setscene_tids:
deps = set()
for dep in sq_revdeps_new[tid]:
@@ -1855,6 +1908,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
sq_revdeps_squash[tid] = deps
elif len(sq_revdeps_new[tid]) != 0:
bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
+ self.rqdata.init_progress_reporter.update(taskcounter)
+
+ self.rqdata.init_progress_reporter.next_stage()
# Resolve setscene inter-task dependencies
# e.g. do_sometask_setscene[depends] = "targetname:do_someothertask_setscene"
@@ -1882,10 +1938,14 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
# Have to zero this to avoid circular dependencies
sq_revdeps_squash[deptid] = set()
+ self.rqdata.init_progress_reporter.next_stage()
+
for task in self.sq_harddeps:
for dep in self.sq_harddeps[task]:
sq_revdeps_squash[dep].add(task)
+ self.rqdata.init_progress_reporter.next_stage()
+
#for tid in sq_revdeps_squash:
# for dep in sq_revdeps_squash[tid]:
# data = data + "\n %s" % dep
@@ -1901,6 +1961,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
for dep in self.sq_revdeps[tid]:
self.sq_deps[dep].add(tid)
+ self.rqdata.init_progress_reporter.next_stage()
+
for tid in self.sq_revdeps:
if len(self.sq_revdeps[tid]) == 0:
self.runq_buildable.add(tid)
@@ -1956,6 +2018,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
logger.debug(2, 'No package found, so skipping setscene task %s', tid)
self.outrightfail.append(tid)
+ self.rqdata.init_progress_reporter.finish()
+
logger.info('Executing SetScene Tasks')
self.rq.state = runQueueSceneRun
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v3] fetch2: implement progress support
2016-06-23 10:59 ` [PATCH v2 05/10] fetch2: implement progress support Paul Eggleton
@ 2016-07-06 4:26 ` Paul Eggleton
2016-07-10 22:23 ` Paul Eggleton
0 siblings, 1 reply; 14+ messages in thread
From: Paul Eggleton @ 2016-07-06 4:26 UTC (permalink / raw)
To: bitbake-devel
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.
At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).
Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.
Part of the implementation for [YOCTO #5383].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
Changes since v2:
* Use -v instead of --show-progress in the wget command line since the
latter is only available with wget 1.16 and newer, and we still need
to support distros that have older versions than that. This does
override the default of -nv if present (which it will be via the
FETCHCMD_wget set in bitbake.conf within OE), but the added verbosity
is minimal since the progress information is filtered out of the logs
by the progress handler.
lib/bb/fetch2/__init__.py | 4 ++--
lib/bb/fetch2/git.py | 52 +++++++++++++++++++++++++++++++++++++++++++----
lib/bb/fetch2/wget.py | 28 +++++++++++++++++++++++--
lib/bb/progress.py | 31 ++++++++++++++++++++++++++++
4 files changed, 107 insertions(+), 8 deletions(-)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index b6fcaaa..a27512c 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -779,7 +779,7 @@ def localpath(url, d):
fetcher = bb.fetch2.Fetch([url], d)
return fetcher.localpath(url)
-def runfetchcmd(cmd, d, quiet=False, cleanup=None):
+def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None):
"""
Run cmd returning the command output
Raise an error if interrupted or cmd fails
@@ -821,7 +821,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None):
error_message = ""
try:
- (output, errors) = bb.process.run(cmd, shell=True, stderr=subprocess.PIPE)
+ (output, errors) = bb.process.run(cmd, log=log, shell=True, stderr=subprocess.PIPE)
success = True
except bb.process.NotFoundError as e:
error_message = "Fetch command %s" % (e.command)
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 59827e3..4e2dcec 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -71,11 +71,53 @@ import os
import re
import bb
import errno
+import bb.progress
from bb import data
from bb.fetch2 import FetchMethod
from bb.fetch2 import runfetchcmd
from bb.fetch2 import logger
+
+class GitProgressHandler(bb.progress.LineFilterProgressHandler):
+ """Extract progress information from git output"""
+ def __init__(self, d):
+ self._buffer = ''
+ self._count = 0
+ super(GitProgressHandler, self).__init__(d)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(-1)
+
+ def write(self, string):
+ self._buffer += string
+ stages = ['Counting objects', 'Compressing objects', 'Receiving objects', 'Resolving deltas']
+ stage_weights = [0.2, 0.05, 0.5, 0.25]
+ stagenum = 0
+ for i, stage in reversed(list(enumerate(stages))):
+ if stage in self._buffer:
+ stagenum = i
+ self._buffer = ''
+ break
+ self._status = stages[stagenum]
+ percs = re.findall(r'(\d+)%', string)
+ if percs:
+ progress = int(round((int(percs[-1]) * stage_weights[stagenum]) + (sum(stage_weights[:stagenum]) * 100)))
+ rates = re.findall(r'([\d.]+ [a-zA-Z]*/s+)', string)
+ if rates:
+ rate = rates[-1]
+ else:
+ rate = None
+ self.update(progress, rate)
+ else:
+ if stagenum == 0:
+ percs = re.findall(r': (\d+)', string)
+ if percs:
+ count = int(percs[-1])
+ if count > self._count:
+ self._count = count
+ self._fire_progress(-count)
+ super(GitProgressHandler, self).write(string)
+
+
class Git(FetchMethod):
"""Class to fetch a module or modules from git repositories"""
def init(self, d):
@@ -196,10 +238,11 @@ class Git(FetchMethod):
# We do this since git will use a "-l" option automatically for local urls where possible
if repourl.startswith("file://"):
repourl = repourl[7:]
- clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
+ clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, repourl, ud.clonedir)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, clone_cmd)
- runfetchcmd(clone_cmd, d)
+ progresshandler = GitProgressHandler(d)
+ runfetchcmd(clone_cmd, d, log=progresshandler)
os.chdir(ud.clonedir)
# Update the checkout if needed
@@ -214,10 +257,11 @@ class Git(FetchMethod):
logger.debug(1, "No Origin")
runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, repourl), d)
- fetch_cmd = "%s fetch -f --prune %s refs/*:refs/*" % (ud.basecmd, repourl)
+ fetch_cmd = "LANG=C %s fetch -f --prune --progress %s refs/*:refs/*" % (ud.basecmd, repourl)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
- runfetchcmd(fetch_cmd, d)
+ progresshandler = GitProgressHandler(d)
+ runfetchcmd(fetch_cmd, d, log=progresshandler)
runfetchcmd("%s prune-packed" % ud.basecmd, d)
runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
try:
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index d688fd9..275cd7d 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -31,6 +31,7 @@ import subprocess
import os
import logging
import bb
+import bb.progress
import urllib.request, urllib.parse, urllib.error
from bb import data
from bb.fetch2 import FetchMethod
@@ -41,6 +42,27 @@ from bb.utils import export_proxies
from bs4 import BeautifulSoup
from bs4 import SoupStrainer
+class WgetProgressHandler(bb.progress.LineFilterProgressHandler):
+ """
+ Extract progress information from wget output.
+ Note: relies on --progress=dot (with -v or without -q/-nv) being
+ specified on the wget command line.
+ """
+ def __init__(self, d):
+ super(WgetProgressHandler, self).__init__(d)
+ # Send an initial progress event so the bar gets shown
+ self._fire_progress(0)
+
+ def writeline(self, line):
+ percs = re.findall(r'(\d+)%\s+([\d.]+[A-Z])', line)
+ if percs:
+ progress = int(percs[-1][0])
+ rate = percs[-1][1] + '/s'
+ self.update(progress, rate)
+ return False
+ return True
+
+
class Wget(FetchMethod):
"""Class to fetch urls via 'wget'"""
def supports(self, ud, d):
@@ -66,13 +88,15 @@ class Wget(FetchMethod):
if not ud.localfile:
ud.localfile = data.expand(urllib.parse.unquote(ud.host + ud.path).replace("/", "."), d)
- self.basecmd = d.getVar("FETCHCMD_wget", True) or "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate"
+ self.basecmd = d.getVar("FETCHCMD_wget", True) or "/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate"
def _runwget(self, ud, d, command, quiet):
+ progresshandler = WgetProgressHandler(d)
+
logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command))
bb.fetch2.check_network_access(d, command)
- runfetchcmd(command, d, quiet)
+ runfetchcmd(command + ' --progress=dot -v', d, quiet, log=progresshandler)
def download(self, ud, d):
"""Fetch urls"""
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
index 93e42df..1365068 100644
--- a/lib/bb/progress.py
+++ b/lib/bb/progress.py
@@ -58,6 +58,37 @@ class ProgressHandler(object):
self._lastevent = ts
self._progress = progress
+class LineFilterProgressHandler(ProgressHandler):
+ """
+ A ProgressHandler variant that provides the ability to filter out
+ the lines if they contain progress information. Additionally, it
+ filters out anything before the last line feed on a line. This can
+ be used to keep the logs clean of output that we've only enabled for
+ getting progress, assuming that that can be done on a per-line
+ basis.
+ """
+ def __init__(self, d, outfile=None):
+ self._linebuffer = ''
+ super(LineFilterProgressHandler, self).__init__(d, outfile)
+
+ def write(self, string):
+ self._linebuffer += string
+ while True:
+ breakpos = self._linebuffer.find('\n') + 1
+ if breakpos == 0:
+ break
+ line = self._linebuffer[:breakpos]
+ self._linebuffer = self._linebuffer[breakpos:]
+ # Drop any line feeds and anything that precedes them
+ lbreakpos = line.rfind('\r') + 1
+ if lbreakpos:
+ line = line[lbreakpos:]
+ if self.writeline(line):
+ super(LineFilterProgressHandler, self).write(line)
+
+ def writeline(self, line):
+ return True
+
class BasicProgressHandler(ProgressHandler):
def __init__(self, d, regex=r'(\d+)%', outfile=None):
super(BasicProgressHandler, self).__init__(d, outfile)
--
2.5.5
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH v3] fetch2: implement progress support
2016-07-06 4:26 ` [PATCH v3] " Paul Eggleton
@ 2016-07-10 22:23 ` Paul Eggleton
2016-07-10 22:25 ` Paul Eggleton
0 siblings, 1 reply; 14+ messages in thread
From: Paul Eggleton @ 2016-07-10 22:23 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
On Wed, 06 Jul 2016 16:26:10 Paul Eggleton wrote:
> Implement progress reporting support specifically for the fetchers. For
> fetch tasks we don't necessarily know which fetcher will be used (we
> might initially be fetching a git:// URI, but if we instead download a
> mirror tarball we may fetch that over http using wget). These programs
> also have different abilities as far as reporting progress goes (e.g.
> wget gives us percentage complete and rate, git gives this some of the
> time depending on what stage it's at). Additionally we filter out the
> progress output before it makes it to the logs, in order to prevent the
> logs filling up with junk.
>
> At the moment this is only implemented for the wget and git fetchers
> since they are the most commonly used (and svn doesn't seem to support
> any kind of progress output, at least not without doing a relatively
> expensive remote file listing first).
>
> Line changes such as the ones you get in git's output as it progresses
> don't make it to the log files, you only get the final state of the line
> so the logs aren't filled with progress information that's useless after
> the fact.
>
> Part of the implementation for [YOCTO #5383].
>
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
>
> Changes since v2:
> * Use -v instead of --show-progress in the wget command line since the
> latter is only available with wget 1.16 and newer, and we still need
> to support distros that have older versions than that. This does
> override the default of -nv if present (which it will be via the
> FETCHCMD_wget set in bitbake.conf within OE), but the added verbosity
> is minimal since the progress information is filtered out of the logs
> by the progress handler.
So, you asked me to send this, but unfortunately it looks like you merged the
old version - I guess I should send an additional patch now.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v3] fetch2: implement progress support
2016-07-10 22:23 ` Paul Eggleton
@ 2016-07-10 22:25 ` Paul Eggleton
0 siblings, 0 replies; 14+ messages in thread
From: Paul Eggleton @ 2016-07-10 22:25 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
On Mon, 11 Jul 2016 10:23:30 Paul Eggleton wrote:
> On Wed, 06 Jul 2016 16:26:10 Paul Eggleton wrote:
> > Implement progress reporting support specifically for the fetchers. For
> > fetch tasks we don't necessarily know which fetcher will be used (we
> > might initially be fetching a git:// URI, but if we instead download a
> > mirror tarball we may fetch that over http using wget). These programs
> > also have different abilities as far as reporting progress goes (e.g.
> > wget gives us percentage complete and rate, git gives this some of the
> > time depending on what stage it's at). Additionally we filter out the
> > progress output before it makes it to the logs, in order to prevent the
> > logs filling up with junk.
> >
> > At the moment this is only implemented for the wget and git fetchers
> > since they are the most commonly used (and svn doesn't seem to support
> > any kind of progress output, at least not without doing a relatively
> > expensive remote file listing first).
> >
> > Line changes such as the ones you get in git's output as it progresses
> > don't make it to the log files, you only get the final state of the line
> > so the logs aren't filled with progress information that's useless after
> > the fact.
> >
> > Part of the implementation for [YOCTO #5383].
> >
> > Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> > ---
> >
> > Changes since v2:
> > * Use -v instead of --show-progress in the wget command line since the
> >
> > latter is only available with wget 1.16 and newer, and we still need
> > to support distros that have older versions than that. This does
> > override the default of -nv if present (which it will be via the
> > FETCHCMD_wget set in bitbake.conf within OE), but the added verbosity
> > is minimal since the progress information is filtered out of the logs
> > by the progress handler.
>
> So, you asked me to send this, but unfortunately it looks like you merged
> the old version - I guess I should send an additional patch now.
Wait a second - you didn't merge it at all, sorry my mistake.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2016-07-10 22:25 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-23 10:59 [PATCH v2 00/10] Support progress reporting Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 01/10] knotty: provide a symlink to the latest console log Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 02/10] knotty: import latest python-progressbar Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 03/10] lib: implement basic task progress support Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 04/10] lib/bb/progress: add MultiStageProgressReporter Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 05/10] fetch2: implement progress support Paul Eggleton
2016-07-06 4:26 ` [PATCH v3] " Paul Eggleton
2016-07-10 22:23 ` Paul Eggleton
2016-07-10 22:25 ` Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 06/10] knotty: add code to support showing progress for sstate object querying Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 07/10] knotty: show task progress bar Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 08/10] knotty: add quiet output mode Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 09/10] runqueue: add ability to enforce that tasks are setscened Paul Eggleton
2016-06-23 10:59 ` [PATCH v2 10/10] runqueue: report progress for "Preparing RunQueue" step Paul Eggleton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.