[PATCH 0/4] Progress handling enhancements

[PATCH 0/4] Progress handling enhancements

[Chris Laplante]

This series of patches supersedes a couple of individual patches I had submitted in the past but that were not merged (either because of feedback or because people forgot about them, I guess).

For simplicity and to consolidate my work, I have prepared this patch set.

An additional change to poky is necessary (I will submit it soon) for functionality in the last commit to be fully usable. But no harm will come from merging this patch set before the poky change.

Chris Laplante (4):
  bitbake: knotty: allow progress rate for indeterminate bars
  bitbake: build: extract progress handler creation logic into its own
    method
  bitbake: build/progress: use context managers for progress handlers
  bitbake: build: implement custom progress handlers injected via
    OE_EXTRA_IMPORTS

 lib/bb/build.py     | 79 +++++++++++++++++++++++++++++++++++++++++++----------
 lib/bb/progress.py  | 16 ++++++++++-
 lib/bb/ui/knotty.py |  2 +-
 3 files changed, 80 insertions(+), 17 deletions(-)

--
2.7.4

[PATCH 2/4] bitbake: build: extract progress handler creation logic into its own method

[Chris Laplante]

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
 lib/bb/build.py | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index dae42ac..a0a764a 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -323,6 +323,21 @@ trap 'bb_exit_handler' 0
 set -e
 '''
 
+def create_progress_handler(func, progress, logfile, d):
+    if progress == 'percent':
+        # Use default regex
+        return bb.progress.BasicProgressHandler(d, outfile=logfile)
+    elif progress.startswith('percent:'):
+        # Use specified regex
+        return bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+    elif progress.startswith('outof:'):
+        # Use specified regex
+        return bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+    else:
+        bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
+
+    return logfile
+
 def exec_func_shell(func, d, runfile, cwd=None):
     """Execute a shell function from the metadata
 
@@ -366,17 +381,7 @@ exit $ret
 
     progress = d.getVarFlag(func, 'progress')
     if progress:
-        if progress == 'percent':
-            # Use default regex
-            logfile = bb.progress.BasicProgressHandler(d, outfile=logfile)
-        elif progress.startswith('percent:'):
-            # Use specified regex
-            logfile = bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
-        elif progress.startswith('outof:'):
-            # Use specified regex
-            logfile = bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
-        else:
-            bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
+        logfile = create_progress_handler(func, progress, logfile, d)
 
     fifobuffer = bytearray()
     def readfifo(data):
-- 
2.7.4

[PATCH 3/4] bitbake: build/progress: use context managers for progress handlers

[Chris Laplante]

It seems context management support was half-implemented, but never
finished. For example, LogTee has __enter__ and __exit__ but they
haven't been exercised until now.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
 lib/bb/build.py    | 31 +++++++++++++++++++++++++++----
 lib/bb/progress.py | 16 +++++++++++++++-
 2 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index a0a764a..85ad8ea 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -163,12 +163,35 @@ class LogTee(object):
 
     def __repr__(self):
         return '<LogTee {0}>'.format(self.name)
+
     def flush(self):
         self.outfile.flush()
 
+
+class StdoutNoopContextManager:
+    """
+    This class acts like sys.stdout, but adds noop __enter__ and __exit__ methods.
+    """
+    def __enter__(self):
+        return sys.stdout
+
+    def __exit__(self, *exc_info):
+        pass
+
+    def write(self, string):
+        return sys.stdout.write(string)
+
+    def flush(self):
+        sys.stdout.flush()
+
+    @property
+    def name(self):
+        return sys.stdout.name
+
+
 #
 # pythonexception allows the python exceptions generated to be raised
-# as the real exceptions (not FuncFailed) and without a backtrace at the 
+# as the real exceptions (not FuncFailed) and without a backtrace at the
 # origin of the failure.
 #
 def exec_func(func, d, dirs = None, pythonexception=False):
@@ -375,9 +398,9 @@ exit $ret
             cmd = [fakerootcmd, runfile]
 
     if bb.msg.loggerDefaultVerbose:
-        logfile = LogTee(logger, sys.stdout)
+        logfile = LogTee(logger, StdoutNoopContextManager())
     else:
-        logfile = sys.stdout
+        logfile = StdoutNoopContextManager()
 
     progress = d.getVarFlag(func, 'progress')
     if progress:
@@ -433,7 +456,7 @@ exit $ret
             bb.debug(2, "Executing shell function %s" % func)
 
             try:
-                with open(os.devnull, 'r+') as stdin:
+                with open(os.devnull, 'r+') as stdin, logfile:
                     bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
             except bb.process.CmdError:
                 logfn = d.getVar('BB_LOGFILE')
diff --git a/lib/bb/progress.py b/lib/bb/progress.py
index e9b72e2..4022caa 100644
--- a/lib/bb/progress.py
+++ b/lib/bb/progress.py
@@ -13,6 +13,7 @@ import time
 import inspect
 import bb.event
 import bb.build
+from bb.build import StdoutNoopContextManager
 
 class ProgressHandler(object):
     """
@@ -27,7 +28,14 @@ class ProgressHandler(object):
         if outfile:
             self._outfile = outfile
         else:
-            self._outfile = sys.stdout
+            self._outfile = StdoutNoopContextManager()
+
+    def __enter__(self):
+        self._outfile.__enter__()
+        return self
+
+    def __exit__(self, *excinfo):
+        self._outfile.__exit__(*excinfo)
 
     def _fire_progress(self, taskprogress, rate=None):
         """Internal function to fire the progress event"""
@@ -147,6 +155,12 @@ class MultiStageProgressReporter(object):
             self._stage_total = None
             self._callers = []
 
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *excinfo):
+        pass
+
     def _fire_progress(self, taskprogress):
         bb.event.fire(bb.build.TaskProgress(taskprogress), self._data)
 
-- 
2.7.4

[PATCH 4/4] bitbake: build: implement custom progress handlers injected via OE_EXTRA_IMPORTS

[Chris Laplante]

A separate patch to base.bbclass (in poky) will add the OE_EXTRA_IMPORTS
variable. The contents are appended into OE_IMPORTS. This provides a
mechanism by which layers (in their layer.conf) can make custom progress
handlers available.

As a backup, individual recipes can inject progress handlers into
__builtins__.

Custom handlers are expected to have this __init__ signature:

    def __init__(self, d, outfile=None, otherargs=None):

Recipes can then use the handlers like this:

    do_task[progress] = "custom:mylayer.util.ProgressHandler[:args]"

The last part (everything after and including the second colon) is
optional. If provided, it is passed to HandlerClass's __init__ as
otherargs="args". Otherwise, otherargs=None.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
---
 lib/bb/build.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/lib/bb/build.py b/lib/bb/build.py
index 85ad8ea..e2f91fa 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -356,6 +356,27 @@ def create_progress_handler(func, progress, logfile, d):
     elif progress.startswith('outof:'):
         # Use specified regex
         return bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+    elif progress.startswith("custom:"):
+        # Use a custom progress handler that was injected via OE_EXTRA_IMPORTS or __builtins__
+        import functools
+        from types import ModuleType
+
+        parts = progress.split(":", 2)
+        _, cls, otherargs = parts[0], parts[1], (parts[2] or None) if parts[2:] else None
+        if cls:
+            def resolve(x, y):
+                if not x:
+                    return None
+                if isinstance(x, ModuleType):
+                    return getattr(x, y, None)
+                return x.get(y)
+            cls_obj = functools.reduce(resolve, cls.split("."), bb.utils._context)
+            if not cls_obj:
+                # Fall-back on __builtins__
+                cls_obj = functools.reduce(lambda x, y: x.get(y), cls.split("."), __builtins__)
+            if cls_obj:
+                return cls_obj(d, outfile=logfile, otherargs=otherargs)
+            bb.warn('%s: unknown custom progress handler in task progress varflag value "%s", ignoring' % (func, cls))
     else:
         bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
 
-- 
2.7.4

Re: [PATCH 0/4] Progress handling enhancements

[Richard Purdie]

On Fri, 2019-06-07 at 14:24 -0400, Chris Laplante via bitbake-devel
wrote:
> This series of patches supersedes a couple of individual patches I
> had submitted in the past but that were not merged (either because of
> feedback or because people forgot about them, I guess).
> 
> For simplicity and to consolidate my work, I have prepared this patch
> set.
> 
> An additional change to poky is necessary (I will submit it soon) for
> functionality in the last commit to be fully usable. But no harm will
> come from merging this patch set before the poky change.
> 
> Chris Laplante (4):
>   bitbake: knotty: allow progress rate for indeterminate bars
>   bitbake: build: extract progress handler creation logic into its
> own
>     method
>   bitbake: build/progress: use context managers for progress handlers
>   bitbake: build: implement custom progress handlers injected via
>     OE_EXTRA_IMPORTS

Could you resend at least the first patch please? For some reason I
have the other three in my mail but not the first one.

Alternatively a pointer to a git repo with the patches in would be
good, or cc me on the patches please?

We're about to change the mailing list infrastructure and one reason is
some of these kinds of issues so trying to figure out what happened to
it isn't top of the list of priorities...

Cheers,

Richard