[bitbake][dunfell][1.46][PATCH 0/3] Patch review

[bitbake][dunfell][1.46][PATCH 0/3] Patch review

[Steve Sakoman]

Please review this set of patches for dunfell/1.46 and have comments back by
end of day Tuesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2968

The following changes since commit c0348de8121c3a842bf44906f7e2f79e93f7275b:

  fetch/wget: Add timeout for checkstatus calls (30s) (2021-11-11 10:58:01 +0000)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Richard Purdie (3):
  command: Ensure exceptions inheriting from BBHandledException are
    visible
  tinfoil: When sending commands we need to process events
  process/knotty: Improve early exception handling

 lib/bb/command.py        |  6 +++++-
 lib/bb/server/process.py |  7 ++++++-
 lib/bb/tinfoil.py        | 11 ++++++++++-
 lib/bb/ui/knotty.py      | 21 +++++++++++++++++----
 4 files changed, 38 insertions(+), 7 deletions(-)

-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 1/3] command: Ensure exceptions inheriting from BBHandledException are visible

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Previous changes allowed BBHandledException to be detected but not exceptions
which inherit from it. Fix this. The code really needs totally reworking
to preserve the exceptions.

[YOCTO #14054]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ef762d92df6c2554c6248e80212f984d9ec4c651)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/command.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/bb/command.py b/lib/bb/command.py
index 6abf3866..07128027 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -74,8 +74,12 @@ class Command:
                 result = command_method(self, commandline)
             except CommandError as exc:
                 return None, exc.args[0]
-            except (Exception, SystemExit):
+            except (Exception, SystemExit) as exc:
                 import traceback
+                if isinstance(exc, bb.BBHandledException):
+                    # We need to start returning real exceptions here. Until we do, we can't
+                    # tell if an exception is an instance of bb.BBHandledException
+                    return None, "bb.BBHandledException()\n" + traceback.format_exc()
                 return None, traceback.format_exc()
             else:
                 return result, None
-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 2/3] tinfoil: When sending commands we need to process events

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The server may be displaying useful information for the user through log
messages so we should display anything that has been sent. Its either this
or expecting every UI to implement this code around every command call
which isn't good API.

[YOCTO #14054]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 64ae9d7e2fad804dd9e12706c6d76b4b22f9586b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/tinfoil.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index 8c9b6b8c..ae690389 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -465,7 +465,16 @@ class Tinfoil:
         commandline = [command]
         if params:
             commandline.extend(params)
-        result = self.server_connection.connection.runCommand(commandline)
+        try:
+            result = self.server_connection.connection.runCommand(commandline)
+        finally:
+            while True:
+                event = self.wait_event()
+                if not event:
+                    break
+                if isinstance(event, logging.LogRecord):
+                    if event.taskpid == 0 or event.levelno > logging.INFO:
+                        self.logger.handle(event)
         if result[1]:
             raise TinfoilCommandFailed(result[1])
         return result[0]
-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 3/3] process/knotty: Improve early exception handling

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The new server startup code means exceptions can happen when we aren't
setup to show them to the user correctly, leading to ugly tracebacks.

Add in some special case handling of BBHandledException to at least
ensure that common case doesn't traceback and the user sees meaningful
output.

In the future, the logging setup can likely be improved, as can the way
runCommand handles exceptions, they all should likely become real
exceptions again on the UI side.

[YOCTO #14022]
[YOCTO #14033]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6059d0e77f60ddb679049bd34478f41b1ab7995d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/server/process.py |  7 ++++++-
 lib/bb/ui/knotty.py      | 21 +++++++++++++++++----
 2 files changed, 23 insertions(+), 5 deletions(-)

diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
index 43061eb3..7b135762 100644
--- a/lib/bb/server/process.py
+++ b/lib/bb/server/process.py
@@ -348,7 +348,12 @@ class ServerCommunicator():
             logger.info("No reply from server in 30s")
             if not self.recv.poll(30):
                 raise ProcessTimeout("Timeout while waiting for a reply from the bitbake server (60s)")
-        return self.recv.get()
+        ret, exc = self.recv.get()
+        # Should probably turn all exceptions in exc back into exceptions?
+        # For now, at least handle BBHandledException
+        if exc and "BBHandledException" in exc:
+            raise bb.BBHandledException()
+        return ret, exc
 
     def updateFeatureSet(self, featureset):
         _, error = self.runCommand(["setFeatures", featureset])
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 87e873d6..e70c2464 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -380,14 +380,27 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
               "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
               "bb.build.TaskProgress", "bb.event.ProcessStarted", "bb.event.ProcessProgress", "bb.event.ProcessFinished"]
 
+def drain_events_errorhandling(eventHandler):
+    # We don't have logging setup, we do need to show any events we see before exiting
+    event = True
+    logger = bb.msg.logger_create('bitbake', sys.stdout)
+    while event:
+        event = eventHandler.waitEvent(0)
+        if isinstance(event, logging.LogRecord):
+            logger.handle(event)
+
 def main(server, eventHandler, params, tf = TerminalFilter):
 
-    if not params.observe_only:
-        params.updateToServer(server, os.environ.copy())
+    try:
+        if not params.observe_only:
+            params.updateToServer(server, os.environ.copy())
 
-    includelogs, loglines, consolelogfile, logconfigfile = _log_settings_from_server(server, params.observe_only)
+        includelogs, loglines, consolelogfile, logconfigfile = _log_settings_from_server(server, params.observe_only)
 
-    loglevel, _ = bb.msg.constructLogOptions()
+        loglevel, _ = bb.msg.constructLogOptions()
+    except bb.BBHandledException:
+        drain_events_errorhandling(eventHandler)
+        return 1
 
     if params.options.quiet == 0:
         console_loglevel = loglevel
-- 
2.25.1

[bitbake][dunfell][1.46][PATCH 0/3] Patch review

[Steve Sakoman]

Please review this set of patches for dunfell/1.46 and have comments back by
end of day Thursday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4893

The following changes since commit c16d364dbf68d2a500fecaf8d6e6d62b11475d9f:

  bb/utils: include SSL certificate paths in export_proxies (2023-02-06 23:42:50 +0000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/1.46-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut

Charlie Davies (1):
  bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url

Marek Vasut (2):
  fetch2/git: Prevent git fetcher from fetching gitlab repository
    metadata
  fetch2/git: Clarify the meaning of namespace

 .../bitbake-user-manual-fetching.rst          |  4 ++--
 lib/bb/fetch2/git.py                          | 20 ++++++++++++-------
 2 files changed, 15 insertions(+), 9 deletions(-)

-- 
2.25.1